Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation

Author: Ancil Crayton

arXiv: 2610.31678 [cs.CL]

Submitted: 1 October 2026

Venue: Accepted to the Trustworthy AI for Good (AI4GOOD) Workshop at NeurIPS 2026

Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)

Abstract

Many LLM studies compare how models respond to different follow-up prompts. For example, one prompt may neutrally ask a model to reconsider, while another insists that an invented film really exists. The difference is used to measure the effect of conversational pressure. An automatic scorer then labels whether each response makes an unsupported factual claim. Researchers commonly validate the scorer with a single overall accuracy or $F_1$ score.

However, high overall performance can hide prompt-specific mistakes. If the scorer makes more errors after one prompt, the measured difference can be too large, too small, or point in the wrong direction. An audit containing mostly difficult responses can also misrepresent performance in the full study.

We derive how such errors alter a prompt comparison and examine the problem using six models, 180 invented-entity questions, and six prompt conditions. In a balanced audit of 288 responses, overall $F_1$ was 0.916. Yet, depending on the prompt, the scorer flagged 2.6%–33.3% of responses that the primary LLM rater judged not to contain an unsupported claim; these prompt-specific estimates were uncertain. In a second audit, the scorer and two separate LLM raters evaluated the same 300 model–question cases under each prompt. An apparent 9.3-point increase from asserting certainty rather than neutrally asking the model to reconsider fell to 2.3 points under both raters. The primary-rater interval included zero, so it did not establish an increase. Two other prompt comparisons kept their direction under both raters, while an attempted prompt-specific correction remained too uncertain for a firm conclusion.

Researchers should therefore validate the exact difference they plan to report, not just the scorer as a whole: audit both prompts on matching cases and include scorer and rater uncertainty in the final result.

Key Takeaways

• Overall scorer accuracy is not sufficient when the goal is to compare two prompt conditions: prompt-specific error rates can distort the measured difference.
• Validate the comparison itself. Audit both prompts on matched cases rather than validating the scorer in aggregate.
• Report uncertainty from both sources—the automatic scorer and the human or LLM raters—in the final estimate.
• Empirical evidence: a balanced audit yielded overall $F_1$ = 0.916, yet prompt-specific false-flag rates ranged from 2.6% to 33.3%, and one apparent 9.3-point effect dropped to 2.3 points (with an interval including zero) under independent raters.

Citation

@inproceedings{crayton2026validating,
  title     = {Validating the Comparison, Not Just the Scorer: Why Overall Accuracy Can Mislead in Interactive LLM Evaluation},
  author    = {Crayton, Ancil},
  booktitle = {Trustworthy AI for Good (AI4GOOD) Workshop at NeurIPS 2026},
  year      = {2026},
  eprint    = {2610.31678},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi       = {10.48550/arXiv.2610.31678}
}

via ArXiv CL+LG