Overview
Safety benchmarks for large language models (LLMs) commonly rely on multiple-choice question (MCQ) formats. However, prior work has documented that the MCQ format reduces refusal rates—a finding with significant implications for how we interpret model safety behavior. This paper offers the first mechanistic explanation of that effect, examined on the Weapons of Mass Destruction Proxy (WMDP) dataset across two models: Llama-3.1-8B and Qwen3-8B.
Key Findings
Rapid answer commitment. Using the tuned lens technique, the authors show that when processing an MCQ, the model commits to an answer-letter token very quickly. This stands in sharp contrast to the gradual build-up of refusal seen in open-ended formats.
Distance from the refusal direction. Activations on the MCQ format lie roughly 3× farther from the refusal direction than their open-ended counterparts.
Competition between answer selection and refusal. Ablating the attention heads that carry the MCQ-format answer increases refusal. This provides direct evidence that MCQ answer selection competes with refusal.
Implications
These findings indicate that MCQ safety scores primarily measure knowledge capabilities rather than assessing the model's safety mechanisms. In 2026, as MCQ-based safety evaluations remain widespread, this work argues for caution in treating such scores as reliable proxies for refusal behavior—and for complementing them with open-ended and mechanistic assessments.
Publication Details
- Comments: Accepted to BlackboxNLP 2026 (EMNLP 2026 workshop). 17 pages, 15 figures.
- Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
- Cite as: arXiv:2611.08639 [cs.CL]
- DOI: https://doi.org/10.48550/arXiv.2611.08639
- Submitted: 5 Nov 2026
Authors
Stepan Epifantsev, Anna Antipova, Oleg Chernikov, Leon Strelkov, Alexey Dontsov
via ArXiv CL+LG