Computer Science > Artificial Intelligence
arXiv:2610.20667 (cs.AI)
Submitted: 14 October 2026
Title
Evaluating Model Capabilities After Benchmark Saturation
Authors
Morio Hamada, Jehyeok Yeon, Matteo Migliarini, Justin Olive, Magda Dubois
Abstract
Rapid advances in AI capabilities are causing benchmarks to saturate. Once several models approach perfect scores, a benchmark loses its power to differentiate between them โ and with it, its utility for informing AI safety cases and frontier policies. Yet benchmark items produce far more information than the scores we typically extract from them: a full transcript of how each model solved the task. We investigate whether these transcripts can be used to assess model capabilities even after benchmark scores have saturated.
We simulate saturation retrospectively using historical benchmark and model releases, and test whether protocols that access only correctly solved transcripts from saturated benchmarks can predict capabilities measured on held-out, unsaturated items. We evaluate representative contemporary approaches for extracting information from model transcripts:
- Pairwise judging
- Single-transcript absolute scoring
- Predictions from deterministic transcript features or embeddings
We find that saturated transcripts retain capability information beyond benchmark accuracy. Amongst our evaluated protocols, pairwise judging performs best, reducing prediction error by 5.3 times compared to a baseline using only saturated benchmark scores. This capability-relevant information is distributed across transcript sections, but concentrated in specific channels of information such as reasoning and tool calls.
However, the predictive power of protocols does not always generalise across distribution shifts โ for example, by underestimating the capabilities of the first reasoning models introduced in early 2025. Saturated benchmarks thus retain capability-predicting signal in their task transcripts, but extending the lifetime of existing benchmarks requires methods designed to extract these signals robustly.
Subjects
- Artificial Intelligence (cs.AI)
- Machine Learning (cs.LG)
Cite as
arXiv:2610.20667 [cs.AI]
(or arXiv:2610.20667v1 [cs.AI] for this version)
DOI: https://doi.org/10.48550/arXiv.2610.20667
via ArXiv AI