dorsal/arxiv
View Schemad3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation
| Authors | Yu-Yang Qian, Junda Su, Lanxiang Hu, Peiyuan Zhang, Zhijie Deng, Peng Zhao, Hao Zhang |
|---|---|
| Categories | |
| ArXiv ID | 2601.07568vv2 |
| URL | https://arxiv.org/abs/2601.07568 |
| License | http://arxiv.org/licenses/nonexclusive-distrib/1.0/ |
Abstract
Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as dLLMs inherently face an accuracy-parallelism trade-off. Despite increasing interest, existing methods typically focus on only one-side of the coin, targeting either efficiency or performance. To address this limitation, we propose d3LLM (Pseudo-Distilled Diffusion Large Language Model), striking a balance between accuracy and parallelism: (i) during training, we introduce pseudo-trajectory distillation to teach the model which tokens can be decoded confidently at early steps, thereby improving parallelism; (ii) during inference, we employ entropy-based multi-block decoding with a KV-cache refresh mechanism to achieve high parallelism while maintaining accuracy. To better evaluate dLLMs, we also introduce AUP (Accuracy Under Parallelism), a new metric that jointly measures accuracy and parallelism. Experiments demonstrate that our d3LLM achieves up to 10$\times$ speedup over vanilla LLaDA/Dream and 5$\times$ speedup over AR models without much accuracy drop. Our code is available at https://github.com/hao-ai-lab/d3LLM.
{
"annotation_id": "a61bd389-43af-4014-a29e-9c802fb11dd1",
"date_created": "2026-02-17T05:53:11.892000Z",
"date_modified": "2026-02-17T05:53:11.892000Z",
"file_hash": "3e3eda6a60384cb5f7e17d0e946228edb128e1c5ff8af20d85120a458b6f0f8c",
"private": false,
"record": {
"abstract": "Diffusion large language models (dLLMs) offer capabilities beyond those of autoregressive (AR) LLMs, such as parallel decoding and random-order generation. However, realizing these benefits in practice is non-trivial, as dLLMs inherently face an accuracy-parallelism trade-off. Despite increasing interest, existing methods typically focus on only one-side of the coin, targeting either efficiency or performance. To address this limitation, we propose d3LLM (Pseudo-Distilled Diffusion Large Language Model), striking a balance between accuracy and parallelism: (i) during training, we introduce pseudo-trajectory distillation to teach the model which tokens can be decoded confidently at early steps, thereby improving parallelism; (ii) during inference, we employ entropy-based multi-block decoding with a KV-cache refresh mechanism to achieve high parallelism while maintaining accuracy. To better evaluate dLLMs, we also introduce AUP (Accuracy Under Parallelism), a new metric that jointly measures accuracy and parallelism. Experiments demonstrate that our d3LLM achieves up to 10$\\times$ speedup over vanilla LLaDA/Dream and 5$\\times$ speedup over AR models without much accuracy drop. Our code is available at https://github.com/hao-ai-lab/d3LLM.",
"arxiv_id": "2601.07568",
"authors": [
"Yu-Yang Qian",
"Junda Su",
"Lanxiang Hu",
"Peiyuan Zhang",
"Zhijie Deng",
"Peng Zhao",
"Hao Zhang"
],
"categories": [
"cs.LG",
"cs.AI"
],
"license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
"title": "d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation",
"url": "https://arxiv.org/abs/2601.07568",
"version": "v2"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "edaf759f-d811-49e6-a928-a80cc7e68f0a",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}