dorsal/arxiv
View SchemaHigh-Rank Structured Modulation for Parameter-Efficient Fine-Tuning
| Authors | Yongkang Liu, Xing Li, Mengjie Zhao, Shanru Zhang, Zijing Wang, Qian Li, Shi Feng, Feiliang Ren, Daling Wang, Hinrich Schütze |
|---|---|
| Categories | |
| ArXiv ID | 2601.07507vv1 |
| URL | https://arxiv.org/abs/2601.07507 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
As the number of model parameters increases, parameter-efficient fine-tuning (PEFT) has become the go-to choice for tailoring pre-trained large language models. Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning, which is widely used to reduce resource requirements. However, decreasing the rank encounters challenges with limited representational capacity when compared to full parameter fine-tuning. We present \textbf{SMoA}, a high-rank \textbf{S}tructured \textbf{MO}dulation \textbf{A}dapter that uses fewer trainable parameters while maintaining a higher rank, thereby improving the model's representational capacity and offering improved performance potential. The core idea is to freeze the original pretrained weights and selectively amplify or suppress important features of the original weights across multiple subspaces. The subspace mechanism provides an efficient way to increase the capacity and complexity of a model. We conduct both theoretical analyses and empirical studies on various tasks. Experiment results show that SMoA outperforms LoRA and its variants on 10 tasks, with extensive ablation studies validating its effectiveness.
{
"annotation_id": "327a811a-ee54-461a-9ac5-7a37f2219bec",
"date_created": "2026-02-17T05:53:12.065000Z",
"date_modified": "2026-02-17T05:53:12.065000Z",
"file_hash": "e7e1bacf53d8844825fe5b3e531e92ddb4a8bf163a39f7eefded565d71f0a160",
"private": false,
"record": {
"abstract": "As the number of model parameters increases, parameter-efficient fine-tuning (PEFT) has become the go-to choice for tailoring pre-trained large language models. Low-rank Adaptation (LoRA) uses a low-rank update method to simulate full parameter fine-tuning, which is widely used to reduce resource requirements. However, decreasing the rank encounters challenges with limited representational capacity when compared to full parameter fine-tuning. We present \\textbf{SMoA}, a high-rank \\textbf{S}tructured \\textbf{MO}dulation \\textbf{A}dapter that uses fewer trainable parameters while maintaining a higher rank, thereby improving the model\u0027s representational capacity and offering improved performance potential. The core idea is to freeze the original pretrained weights and selectively amplify or suppress important features of the original weights across multiple subspaces. The subspace mechanism provides an efficient way to increase the capacity and complexity of a model. We conduct both theoretical analyses and empirical studies on various tasks. Experiment results show that SMoA outperforms LoRA and its variants on 10 tasks, with extensive ablation studies validating its effectiveness.",
"arxiv_id": "2601.07507",
"authors": [
"Yongkang Liu",
"Xing Li",
"Mengjie Zhao",
"Shanru Zhang",
"Zijing Wang",
"Qian Li",
"Shi Feng",
"Feiliang Ren",
"Daling Wang",
"Hinrich Sch\u00fctze"
],
"categories": [
"cs.CL"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "High-Rank Structured Modulation for Parameter-Efficient Fine-Tuning",
"url": "https://arxiv.org/abs/2601.07507",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "acc90798-f213-41fc-a1c4-4fd07b1e24a2",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}