Overview
Title: X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decoding
Authors: Jaeduk Lee, Wan Choi
Subjects: Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC); Machine Learning (cs.LG)
Submitted: 15 July 2026
arXiv: 2609.09166 cs.CL] · [v1
DOI: 10.48550/arXiv.2609.09166
Abstract
This paper investigates collaborative speculative decoding (CoSD), a distributed large language model (LLM) inference framework in which an on-device small language model (SLM) drafts candidate tokens and a server LLM verifies them.
Existing CoSD methods assume a shared vocabulary between the SLM and the LLM and incur substantial communication load, because residual resampling requires the exchange of token distributions between the user device and the edge server. To address these limitations, the authors propose cross-vocabulary CoSD (X-CoSD), a lossless and communication-efficient CoSD framework designed for heterogeneous SLM–LLM vocabularies.
X-CoSD is built on hybrid resampling (HR), which splits residual resampling across two regions: the common-vocabulary region processed on the device, and the LLM-only region processed on the server. As a result, distribution transmission is required only for the common-vocabulary region.
The authors further propose X-CoSD-E, an enhanced variant based on server resampling with device verification (SR-DV). In this scheme, the server sends only replacement candidates sampled from the server LLM, together with their corresponding probabilities, for local verification at the device.
The paper proves that both X-CoSD and X-CoSD-E preserve the server LLM distribution, and experiments show that they significantly improve token generation speed while maintaining generation quality comparable to that of the server LLM.
Key Contributions
- Cross-vocabulary support: Removes the shared-vocabulary assumption that constrains prior CoSD methods, enabling collaboration between heterogeneous SLM and LLM tokenizers.
- Hybrid resampling (HR): Distributes residual resampling between device and server, limiting distribution transmission to the common-vocabulary region only.
- X-CoSD-E with SR-DV: Further reduces communication by having the server transmit only replacement candidates and their probabilities for on-device verification.
- Lossless guarantees: Formal proofs establishing that both X-CoSD and X-CoSD-E preserve the server LLM's output distribution.
- Efficiency gains: Demonstrated improvements in token generation speed without sacrificing generation quality relative to the server LLM.
Context
As on-device LLM deployment expands in 2026, collaborative inference between local SLMs and server-side LLMs has become a practical approach for balancing latency, privacy, and quality. Speculative decoding remains a central technique in this space, but vocabulary mismatch between compact on-device models and large server models has been an enduring obstacle. X-CoSD addresses this gap directly, offering a lossless, communication-efficient path toward heterogeneous collaborative inference.
via ArXiv CL+LG