Abstention Is Not Threshold Compliance: Stated Scoring Rules in Open Language Models
Authors: Dongsun Moon
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
arXiv: 2611.14661 [cs.CL]
Submitted: 10 November 2026
Comments: 34 pages, 3 figures, 29 tables
DOI: https://doi.org/10.48550/arXiv.2611.14661
Abstract
Kalai et al. propose stating the scoring rule in the prompt with an explicit threshold t, so that a model should answer only when its probability of being correct exceeds t. We probe each requirement of compliance on shared items across 12 checkpoints spanning three post-training stages of four fully open 3โ8B recipes, with exploratory models up to 32B.
At t = 0.75, a line limiting the reply to an answer or "I abstain" (the abstention format) raised the mean abstention rate from 20% to 91%. Yet within this format, raising t from 0.5 to 0.9 changed the abstention rate by only โ0.8 percentage points. Without the line, the higher threshold even lowered abstention, and up to 40% of a checkpoint's items received decisions that no compliant policy with threshold-independent beliefs could make. Given the probability that their answer was correct, the checkpoints barely responded to the threshold in a forced-choice probe.
Comparisons with self-assessment signals such as P(True) depended on the grader and on answer drift between prompts. An external threshold on recalibrated P(True) made coverage non-increasing in t by construction. Calibrated on the evaluation distribution (post hoc), it beat the abstention format at t = 0.5; calibrated on TriviaQA and applied to PopQA, however, it fell below that format at 0.75. On OLMo 3 7B, decisions trained on threshold-conditioned targets did not follow unseen thresholds consistently. A post-hoc variant trained at four thresholds answered less as t rose but still failed a pre-specified test, mainly on agreement with its training rule.
A high abstention rate is not evidence of compliance. Evaluations using stated scoring rules should report how often models abstain, whether decisions respond to the stated threshold, and which items they answer.
Key Findings
- Abstention format inflates abstention rates without enforcing threshold sensitivity. Adding a reply-format line raised mean abstention from 20% to 91% at t = 0.75, but varying t from 0.5 to 0.9 shifted abstention by only โ0.8 percentage points.
- Higher thresholds can paradoxically reduce abstention when the format line is absent, undermining the core premise of stated scoring rules.
- Up to 40% of decisions were non-compliant โ inconsistent with any policy that holds threshold-independent beliefs.
- Self-assessment signals such as P(True) are fragile, with comparisons depending on the grader and on answer drift between prompts.
- Threshold training does not generalize reliably to unseen thresholds, as shown on OLMo 3 7B.
Implications for Evaluation
The paper argues that a high abstention rate is not evidence of compliance. Evaluations using stated scoring rules should report:
- How often models abstain;
- Whether decisions respond to the stated threshold;
- Which items models answer.
These reporting requirements are especially relevant as of 2026, when selective prediction and abstention mechanisms are increasingly deployed in open-weight language models and evaluated under stated scoring-rule protocols.
via ArXiv CL+LG