dorsal/arxiv
View SchemaMoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models
| Authors | Xin Ye, Daning Cheng, Boyang Zhang, Yunquan Zhang |
|---|---|
| Categories | |
| ArXiv ID | 2601.06857vv1 |
| URL | https://arxiv.org/abs/2601.06857 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model training. In contrast, affordable hardware is low-cost but constrained by memory capacity and bandwidth, making it unsuitable for direct LLM training. To address this, we propose MoE-DisCo (Mixture-of-Experts with Disentangled Clustering and Coordination), a staged training framework. MoE-DisCo decomposes the MoE model into multiple dense submodels, each consisting of a shared backbone and a single expert, and partitions the training data into subsets using unsupervised clustering. Each submodel is trained independently and in parallel on its assigned data subset using low-cost devices, without any inter-device communication. Subsequently, all experts are integrated into a complete MoE model and fine-tuned globally for a short period on high-memory, high-bandwidth GPUs. Experiments show that our method matches or even surpasses full-parameter training in performance across multiple downstream tasks, loss function, and perplexity (PPL), while reducing training cost by 47.6 percent to 69.5 percent on Qwen1.5-MoE-2.7B and Llama-MoE-3.5B across different datasets.
{
"annotation_id": "d3614be9-1e18-4db9-910e-553292b1a5e8",
"date_created": "2026-02-17T05:53:08.803000Z",
"date_modified": "2026-02-17T05:53:08.803000Z",
"file_hash": "9fb499949be024d8364dccb86c13665d14440d1b4c372782579eb329bd89db22",
"private": false,
"record": {
"abstract": "Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model training. In contrast, affordable hardware is low-cost but constrained by memory capacity and bandwidth, making it unsuitable for direct LLM training. To address this, we propose MoE-DisCo (Mixture-of-Experts with Disentangled Clustering and Coordination), a staged training framework. MoE-DisCo decomposes the MoE model into multiple dense submodels, each consisting of a shared backbone and a single expert, and partitions the training data into subsets using unsupervised clustering. Each submodel is trained independently and in parallel on its assigned data subset using low-cost devices, without any inter-device communication. Subsequently, all experts are integrated into a complete MoE model and fine-tuned globally for a short period on high-memory, high-bandwidth GPUs. Experiments show that our method matches or even surpasses full-parameter training in performance across multiple downstream tasks, loss function, and perplexity (PPL), while reducing training cost by 47.6 percent to 69.5 percent on Qwen1.5-MoE-2.7B and Llama-MoE-3.5B across different datasets.",
"arxiv_id": "2601.06857",
"authors": [
"Xin Ye",
"Daning Cheng",
"Boyang Zhang",
"Yunquan Zhang"
],
"categories": [
"cs.LG",
"cs.AI"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models",
"url": "https://arxiv.org/abs/2601.06857",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "e7bc0ba9-26b4-4cc9-a832-6c1bbadfb47a",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}