Overview
Automatic Speech Recognition (ASR) models such as OpenAI's Whisper have substantially improved speech recognition quality, yet they remain expensive to fine-tune—especially when large amounts of transcribed audio are available. This cost makes it challenging for many practitioners to adapt these models to new languages or domains.
In this work, Roi Yozevitch and Or Haim Anidjar introduce an embedding-based subset selection approach for choosing smaller training sets for ASR fine-tuning. As of 2026, with Whisper-scale models increasingly deployed across multilingual and domain-specific settings, efficient adaptation strategies of this kind have become a practical priority for teams working under limited compute and annotation budgets.
Method
Using Whisper's pretrained encoder, the authors map each audio sample into a fixed-dimensional embedding space and apply a greedy k-center algorithm to select diverse and representative subsets of the training data. This allows fine-tuning to proceed on a fraction of the available transcribed audio while preserving coverage of the underlying acoustic and linguistic variation.
Evaluation
The approach is evaluated on both Hebrew and Danish. Results show that k-center selection often improves upon random sampling, particularly at smaller subset sizes.
An analysis of lexical diversity reveals that k-center subsets capture a broader range of linguistic content than random subsets of the same size, which may help explain their observed effectiveness.
Key Findings
- Embedding-based subset selection provides a practical strategy for ASR subset selection in the two evaluated languages.
- The clearest recognition benefits are observed at smaller sample budgets.
- k-center subsets exhibit greater lexical diversity than random subsets of equal size, offering a plausible explanation for their effectiveness.
Publication Details
- arXiv: 2610.32997 [cs.CL]
- Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
- Submitted: 5 October 2026
- Comments: 17 pages, 5 figures
- DOI: https://doi.org/10.48550/arXiv.2610.32997
via ArXiv CL+LG