1. Introduction
Frontier large language models (LLMs) increasingly demonstrate an ability to detect when they are being evaluatedβa phenomenon termed evaluation awareness. This capability presents a significant challenge to the validity of evaluation results, which remain a cornerstone of modern AI safety frameworks. If models alter their behavior during assessments relative to real-world deployment, the reliability of these evaluations is compromised.
To address this issue, we introduce EvalDetectBench, an open pipeline and benchmark designed to measure evaluation awareness across any Inspect-compatible evaluation. This flexibility allows practitioners to test against both current and emerging benchmarks, ensuring adaptability in a rapidly evolving landscape.
2. Benchmark Design and Content
EvalDetectBench ships with a newly curated transcript suite that spans existing frontier system-card evaluations and a diverse range of deployment sources.
2.1. Purpose
The benchmark fulfills two critical functions:
- Measuring reliability: Quantifying how consistently frontier LLMs recognize that they are under evaluation.
- Assessing detectability: Evaluating how discernible individual benchmarks are as assessment tools, thereby identifying potential sources of bias.
- Model-generated transcript identity: The specific model used to generate deployment transcripts accounts for 11.25% of measurement variance and can reorder model rankingsβa significant confound.
- Elicitation prompt bias: Prompts optimized for high performance on one model may perform near chance levels on others, leading to inconsistent results.
- Per-model probe calibration: Ensures that evaluation probes are calibrated individually for each model, accounting for performance variability.
- Stratified generator-harmonization procedure: Standardizes the generation of deployment transcripts across models to reduce variance and enhance comparability.
- Original submission: arXiv:2609.01611, submitted June 8, 2026
- Authors: Xinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj, Alexandra Souly, and Robert Kirk
2.2. Methodological Innovations
Our literature review identified two methodological choices in prior work that introduce systematic bias, which EvalDetectBench corrects for:
3. Methodological Corrections
To mitigate these biases, EvalDetectBench implements two key corrections:
4. Conclusion and Implications
By addressing these underlying biases, EvalDetectBench offers a more robust measurement of evaluation awareness, contributing to the development of reliable AI safety protocols. As we move further into 2026, the need for transparent and valid evaluation methods becomes ever more pressing. EvalDetectBench serves as a practical tool for researchers and engineers to ensure that frontier models are assessed fairly and accurately, ultimately fostering trust in AI systems.
5. Availability
For further technical details, including the full set of experiments and analyses, refer to the complete paper (24 pages, 12 figures, 10 tables).
References
via ArXiv AI