dorsal/arxiv
View SchemaOvercoming Joint Intractability with Lossless Hierarchical Speculative Decoding
| Authors | Yuxuan Zhou, Fei Huang, Heng Li, Fengyi Wu, Tianyu Wang, Jianwei Zhang, Junyang Lin, Zhi-Qi Cheng |
|---|---|
| Categories | |
| ArXiv ID | 2601.05724vv1 |
| URL | https://arxiv.org/abs/2601.05724 |
| License | http://creativecommons.org/licenses/by/4.0/ |
Abstract
Verification is a key bottleneck in improving inference speed while maintaining distribution fidelity in Speculative Decoding. Recent work has shown that sequence-level verification leads to a higher number of accepted tokens compared to token-wise verification. However, existing solutions often rely on surrogate approximations or are constrained by partial information, struggling with joint intractability. In this work, we propose Hierarchical Speculative Decoding (HSD), a provably lossless verification method that significantly boosts the expected number of accepted tokens and overcomes joint intractability by balancing excess and deficient probability mass across accessible branches. Our extensive large-scale experiments demonstrate that HSD yields consistent improvements in acceptance rates across diverse model families and benchmarks. Moreover, its strong explainability and generality make it readily integrable into a wide range of speculative decoding frameworks. Notably, integrating HSD into EAGLE-3 yields over a 12% performance gain, establishing state-of-the-art decoding efficiency without compromising distribution fidelity. Code is available at https://github.com/ZhouYuxuanYX/Hierarchical-Speculative-Decoding.
{
"annotation_id": "cee90977-8cea-46d0-9225-8fcb89978ed3",
"date_created": "2026-02-17T05:53:05.013000Z",
"date_modified": "2026-02-17T05:53:05.013000Z",
"file_hash": "569988c2d6fb5889cb9e7a216b6e5fb27d56b06eb789ad58b01923f0d3040672",
"private": false,
"record": {
"abstract": "Verification is a key bottleneck in improving inference speed while maintaining distribution fidelity in Speculative Decoding. Recent work has shown that sequence-level verification leads to a higher number of accepted tokens compared to token-wise verification. However, existing solutions often rely on surrogate approximations or are constrained by partial information, struggling with joint intractability. In this work, we propose Hierarchical Speculative Decoding (HSD), a provably lossless verification method that significantly boosts the expected number of accepted tokens and overcomes joint intractability by balancing excess and deficient probability mass across accessible branches. Our extensive large-scale experiments demonstrate that HSD yields consistent improvements in acceptance rates across diverse model families and benchmarks. Moreover, its strong explainability and generality make it readily integrable into a wide range of speculative decoding frameworks. Notably, integrating HSD into EAGLE-3 yields over a 12% performance gain, establishing state-of-the-art decoding efficiency without compromising distribution fidelity. Code is available at https://github.com/ZhouYuxuanYX/Hierarchical-Speculative-Decoding.",
"arxiv_id": "2601.05724",
"authors": [
"Yuxuan Zhou",
"Fei Huang",
"Heng Li",
"Fengyi Wu",
"Tianyu Wang",
"Jianwei Zhang",
"Junyang Lin",
"Zhi-Qi Cheng"
],
"categories": [
"cs.AI"
],
"license": "http://creativecommons.org/licenses/by/4.0/",
"title": "Overcoming Joint Intractability with Lossless Hierarchical Speculative Decoding",
"url": "https://arxiv.org/abs/2601.05724",
"version": "v1"
},
"schema_id": "dorsal/arxiv",
"source": {
"execution_id": "bb7a055b-9136-4757-a3e3-46c023de25dd",
"id": "arXiv Dataset",
"type": "Model",
"variant": "snapshot-2026-01-17",
"version": "0.1.0"
},
"user_id": 1000002
}