TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding
Jiaming Fan, Daming Cao, Canchen Huang, Jiale Fu, Jin Zhang, Junjie Gao, Kai Yang, Xiangzhong Luo, and Xu Yang
Abstract
Speculative decoding accelerates large language model (LLM) inference by employing a draft-then-verify paradigm. Building on this approach, tree-structured methods organize draft proposals into multiple candidate paths, thereby increasing the number of tokens that can be accepted in a single verification step. However, existing tree-structured methods rely on a single drafter for all drafting steps, which presents a fundamental trade-off: a smaller drafter operates quickly but produces lower-quality draft trees, while a larger drafter generates higher-quality trees but incurs significant latency. To address this limitation, we introduce TreeGraft, a multi-drafter framework in which drafters of varying costs collaboratively construct a shared draft tree. TreeGraft leverages a stronger drafter to rescore candidates by refining the scores assigned by a weaker drafter, to reselect grafting positions, and to recover promising paths that would otherwise remain unexplored. Additionally, TreeGraft integrates expansions from the stronger drafter in a non-destructive manner, preserving existing branches that may still be accepted by the target model. Collectively, these mechanisms enhance the overall quality of the shared draft tree. To manage drafting costs, TreeGraft incorporates a lightweight scheduler, distilled from an offline value system, that determines when to invoke the stronger drafter. Across 10 model pairs and 6 benchmarks, TreeGraft achieves an average improvement of 15.1% over the better of two fixed single-drafter baselines, with a maximum gain of 26.6%. Our code is publicly available at: [anonymous link provided in the original publication].
Introduction
Large language models have become foundational to numerous natural language processing applications, yet their autoregressive decoding remains a bottleneck due to the sequential nature of token generation. Speculative decoding mitigates this by using a smaller, faster drafter to propose candidate tokens, which are then verified in parallel by the larger target model. Tree-based extensions further improve efficiency by considering multiple candidate sequences simultaneously, increasing the likelihood of accepting longer continuations.
Despite these advances, current tree-based methods typically employ a single drafter throughout the entire drafting process. This creates an inherent dilemma: lightweight drafters are fast but often generate suboptimal draft trees, whereas heavier drafters produce better trees at the cost of higher latency. The choice of a single drafter thus forces a compromise between speed and quality.
In this work, we propose TreeGraft to overcome this limitation by enabling multiple drafters with different capabilities and costs to work together. By dynamically assigning drafting responsibilities, TreeGraft achieves both high-quality draft trees and controlled computational overhead.
Methodology
TreeGraft operates by constructing a shared draft tree where nodes are proposed by a weaker, faster drafter and selectively refined by a stronger, slower drafter. The stronger drafter performs three key functions: rescoring candidate nodes to better reflect their likelihood of acceptance, selecting new grafting positions to insert improved subtrees, and recovering promising branches that the weaker drafter may have missed. These refinements are integrated non-destructively, meaning that existing branches are preserved unless explicitly replaced, ensuring that the target model retains the opportunity to accept them.
A central component of TreeGraft is the adaptive scheduling mechanism. A lightweight scheduler, trained via distillation from an offline value system, decides in real time whether to invoke the stronger drafter. This decision is based on the expected benefit to draft tree quality relative to the additional computational cost, allowing TreeGraft to optimize the trade-off dynamically.
Results
We evaluated TreeGraft across 10 different model pair configurations and 6 standard benchmarks. Compared to the better of two fixed single-drafter endpoints (i.e., using only the weak drafter or only the strong drafter), TreeGraft consistently outperformed both, achieving an average improvement of 15.1% in accepted token length, with a maximum improvement of 26.6%. These results demonstrate the effectiveness of adaptive multi-drafter grafting in enhancing speculative decoding performance.
Conclusion
TreeGraft presents a novel approach to tree-based speculative decoding by introducing a multi-drafter framework that adaptively combines the strengths of different drafters. By rescoring, reselecting grafting positions, and recovering unexplored paths, TreeGraft produces higher-quality draft trees while keeping computational costs in check through a learned scheduler. Our empirical results underscore the potential of this method to significantly accelerate LLM inference without compromising output quality. Future work may explore extending TreeGraft to other model architectures and optimizing scheduling strategies further.
References
[Full reference list as per original publication.]
This article is based on the paper arXiv:2608.26112, submitted on 28 May 2026, and categorized under Computer Science > Computation and Language.
via ArXiv CL+LG