Overview
Multilingual speech recognition models such as Whisper can transcribe a wide range of languages—including low-resource ones—without language-specific training. However, their computational cost makes deployment challenging, particularly in settings where latency and hardware budgets are constrained. Token merging addresses this inefficiency by dynamically combining redundant features, shortening the input sequence during inference without requiring retraining.
In this paper, the authors systematically evaluate token merging on the Whisper model family across sixteen diverse languages and three model sizes. They also examine how token merging interacts with fine-tuning via DoRA (a parameter-efficient adaptation method) on low-resource languages, an increasingly relevant question as lightweight adaptation techniques become standard in 2026 deployment pipelines.
Key Findings
- Token merging improves computational efficiency with almost no degradation in transcription accuracy across most low-resource languages and model sizes.
- The benefits persist even after the model has been fine-tuned, indicating compatibility with post-hoc adaptation rather than only pre-trained checkpoints.
- Results hold across multiple Whisper scales, suggesting the approach generalizes across model capacity.
Significance
The findings position token merging as a practical method for making multilingual speech recognition faster and cheaper to deploy. By reducing sequence length at inference time without retraining, it offers a straightforward efficiency gain for real-world systems serving linguistically diverse user bases.
Paper Details
- Author: Dylan Luke Holyoak
- Subjects: Computation and Language (cs.CL)
- Comments: 11 pages, 3 figures
- Submitted: 7 July 2026
- arXiv identifier: arXiv:2609.13151 [cs.CL]
- DOI: https://doi.org/10.48550/arXiv.2609.13151
via ArXiv CL+LG