dorsal/arxiv
View SchemaMechanisms are Transferable: Data-Efficient Low-Resource Adaptation via Circuit-Targeted Supervised Fine-Tuning
| Authors | Khumaisa Nur'aini, Ayu Purwarianti, Alham Fikri Aji, Derry Wijaya |
|---|---|
| Categories | |
| ArXiv ID | 2601.08146vv1 |
| URL | https://arxiv.org/abs/2601.08146 |
| License | http://creativecommons.org/licenses/by-sa/4.0/ |
Abstract
Adapting LLMs to low-resource languages is difficult: labeled data is scarce, full-model fine-tuning is unstable, and continued cross-lingual tuning can cause catastrophic forgetting. We propose Circuit-Targeted Supervised Fine-Tuning (CT-SFT): a counterfactual-free adaptation of CD-T (Contextual Decomposition Transformer) that uses a label-balanced mean baseline and task-directional relevance scoring to identify a sparse set of task-relevant attention heads in a proxy-language checkpoint, then transfer learns to a target language by updating only those heads (plus LayerNorm) via head-level gradient masking. Across NusaX-Senti and XNLI, CT-SFT improves cross-lingual accuracy over continued full fine-tuning while updating only a small subset of model parameters. We find an editing-preserving trade-off: harder transfers favor editing circuit heads, while easier transfers often favor near-zero (i.e., low-relevance heads) updates, preserving the source mechanism. CT-SFT also substantially reduces catastrophic forgetting, preserving proxy/source-language competence during transfer.
{
"annotation_id": "ec4486e5-3ba2-4892-9b1a-8630af5395e5",
"date_created": "2026-02-17T05:53:15.099000Z",
"date_modified": "2026-02-17T05:53:15.099000Z",
"file_hash": "a5dbf7b616c097401ca847a1a43443909bd3cfc7116b329a29749ca14228d244",
"private": false,
"record": {
"abstract": "Adapting LLMs to low-resource languages is difficult: labeled data is scarce, full-model fine-tuning is unstable, and continued cross-lingual tuning can cause catastrophic forgetting. We propose Circuit-Targeted Supervised Fine-Tuning (CT-SFT): a counterfactual-free adaptation of CD-T (Contextual Decomposition Transformer) that uses a label-balanced mean baseline and task-directional relevance scoring to identify a sparse set of task-relevant attention heads in a proxy-language checkpoint, then transfer learns to a target language by updating only those heads (plus LayerNorm) via head-level gradient masking. Across NusaX-Senti and XNLI, CT-SFT improves cross-lingual accuracy over continued full fine-tuning while updating only a small subset of model parameters. We find an editing-preserving trade-off: harder transfers favor editing circuit heads, while easier transfers often favor near-zero (i.e., low-relevance heads) updates, preserving the source mechanism. CT-SFT also substantially reduces catastrophic forgetting, preserving proxy/source-language competence during transfer.",
"arxiv_id": "2601.08146",
"authors": [
"Khumaisa Nur\u0027aini",
"Ayu Purwarianti",
"Alham Fikri Aji",
"Derry Wijaya"
],
"categories": [
"cs.CL",
"cs.AI",
"cs.LG"
],
"license": "http://creativecommons.org/licenses/by-sa/4.0/",
"title": "Mechanisms are Transferable: Data-Efficient Low-Resource Adaptation via Circuit-Targeted Supervised Fine-Tuning",
"url": "https://arxiv.org/abs/2601.08146",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "418b8a19-05af-4e95-b2e2-614285cd6f65",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}