Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence

Authors: Muhamathu Ameer Ali Aacaas Muhamath

arXiv: 2609.09184 [cs.CV]

Submitted: 29 August 2026

Subjects: Computer Vision and Pattern Recognition (cs.CV)

DOI: 10.48550/arXiv.2609.09184

Code & Supplementary Materials: github.com/aacaas5/evidence-order-calibration-vlm

Length: 8 pages, 6 figures, 4 tables

Abstract

Vision-language model (VLM) confidence may change in aggregate when visual evidence is degraded while remaining structurally inconsistent within individual examples. This work studies answer-level reliability along five-step, question-conditioned evidence-loss trajectories.

Using a frozen Qwen2.5-VL-3B-Instruct model, the authors construct 176 accepted GQA-derived trajectories (880 masking conditions) by progressively masking scene-graph-localized question-critical regions. Native sequence confidence exhibits an evidence monotonicity violation rate (EMVR) of 0.436, and 92.0% of trajectories contain at least one adjacent violation.

A matched non-critical-region control shows that full critical masking reduces accuracy by 28.2 percentage points, versus 0.6 points for equally sized non-critical masks; the paired difference is 27.6 points (95% CI [20.0, 34.7]).

A lightweight post-hoc reliability head is trained on frozen hidden states, sequence confidence, and entropy. Adding evidence-order supervision to binary cross-entropy (BCE) reduces masking EMVR from 0.330 to 0.303 (paired difference −0.027, 95% CI [−0.044, −0.010]). The same mask-trained objective reduces EMVR from 0.449 to 0.402 on held-out question IDs under unseen local Gaussian blur (difference −0.0468, 95% CI [−0.0739, −0.0199]).

AUROC, Brier, and AURC differences between the two learned heads are statistically inconclusive, and native confidence remains stronger for selective-risk ranking. The results separate evidence-order consistency from conventional correctness discrimination rather than establishing generic confidence superiority.

Key Findings at a Glance

| Metric | Value |

|---|---|

| Trajectories (accepted) | 176 |

| Masking conditions | 880 |

| Native EMVR | 0.436 |

| Trajectories with ≥1 adjacent violation | 92.0% |

| Accuracy drop — full critical masking | −28.2 pts |

| Accuracy drop — equally sized non-critical masks | −0.6 pts |

| Paired difference (critical vs. non-critical) | 27.6 pts (95% CI [20.0, 34.7]) |

| EMVR reduction, masking (BCE + evidence-order) | 0.330 → 0.303 (Δ = −0.027) |

| EMVR reduction, unseen Gaussian blur | 0.449 → 0.402 (Δ = −0.0468) |

Why This Matters in 2026

As VLMs are increasingly deployed in safety-critical pipelines—medical imaging triage, autonomous inspection, accessibility tooling, and retrieval-augmented assistants—selective prediction (abstaining when confidence is unreliable) has become a first-class engineering requirement. The 2026 wave of multimodal agents routinely calls VLMs as sub-routines, so miscalibrated confidence propagates silently through downstream decisions.

This paper targets a specific, previously under-instrumented failure mode: within-example structural inconsistency of confidence under progressive evidence loss. A model can appear well-calibrated in aggregate (stable AUROC, Brier, and AURC) while violating the intuitive expectation that removing question-critical evidence should not increase confidence. The introduced EMVR metric makes this failure measurable and separable from conventional ranking quality, and the proposed evidence-order supervision offers a lightweight, post-hoc correction that transfers to an unseen distortion (Gaussian blur) and to held-out question IDs.

Methodology Overview

1. Trajectory construction — Five-step, question-conditioned evidence-loss paths built from GQA scene graphs, localizing regions critical to each question.
2. Masking protocol — Progressive occlusion of question-critical regions, contrasted against matched non-critical masks of equal size.
3. Base model — Frozen Qwen2.5-VL-3B-Instruct, evaluated with native sequence confidence.
4. Post-hoc reliability head — Lightweight head trained on frozen hidden states, sequence confidence, and entropy, with an evidence-order supervision term added to BCE.
5. Evaluation — EMVR on held-out question IDs, plus AUROC, Brier, and AURC for correctness discrimination, and unseen-distortion transfer via local Gaussian blur.



---




Results and Interpretation



• Native confidence is structurally inconsistent. EMVR of 0.436 and 92.0% of trajectories containing at least one adjacent violation indicate that aggregate calibration does not guarantee per-example monotonicity.
• Question-critical regions carry the signal. The 27.6-point paired accuracy gap confirms that the masking procedure isolates genuinely question-relevant evidence.
• Evidence-order supervision works — narrowly and reproducibly. EMVR drops under both the in-distribution masking setting and the unseen Gaussian-blur transfer setting, with non-overlapping confidence intervals supporting the effect.
• The gain is not generic confidence superiority. AUROC, Brier, and AURC differences between the two learned heads are statistically inconclusive, and native confidence remains stronger for selective-risk ranking.



Takeaway: Evidence-order consistency is a distinct reliability property from correctness discrimination. Practitioners building abstention-aware VLM pipelines should measure both.




---




Citation




@article{muhamath2026evidenceorder,
  title  = {Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence},
  author = {Muhamath, Muhamathu Ameer Ali Aacaas},
  year   = {2026},
  eprint = {2609.09184},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi    = {10.48550/arXiv.2609.09184},
  url    = {https://github.com/aacaas5/evidence-order-calibration-vlm}
}

via ArXiv CV