What Entropy-Based View Selection Cannot See: An Audit of Test-Time Adaptation for Vision-Language Models

Authors: Sivasuthan Sukumar, Uthayasanker Thayasivam

arXiv: 2611.14686 [cs.CV] — Submitted 10 Nov 2026

Venue: Accepted at the NeurIPS 2026 Workshops on VLM4RWD

Subjects: Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI)

Abstract

Vision-language models such as CLIP can classify images into categories they were never explicitly trained to recognize, but accuracy degrades when test images differ in style or source from their training data. A common remedy generates randomly cropped and flipped copies of each test image, called views, and combines their predictions. Because some views miss the object entirely, methods must identify which views to trust. The dominant approach uses prediction entropy as a confidence measure and retains the most certain views; other methods rely on signals such as embedding typicality.

Despite repeated refinements of these scoring functions, the information they depend on—the model's own response to each view—remains largely unexamined. This work investigates that choice while holding all other pipeline stages fixed.

Key Findings

View Selection Drives Most of the Gain

The authors first show that view selection, rather than the gradient update, accounts for almost all of the accuracy gains in test-time adaptation pipelines.

Entropy Fails When It Matters Most

The study then examines whether entropy identifies useful views in the difficult case where predictions are wrong, even though at least one view predicts the correct class. In these cases:

Entropy ranks correct views above incorrect ones no better than chance.
It recovers only about one-twentieth of the accuracy recovered by an oracle over the same views.
The finding holds across two image encoders and survives a control that rules out an obvious statistical artifact.

Why Entropy Falls Short

An exact analysis explains the limitation: entropy measures confidence but not which class a view favors. A view can be highly confident and still confidently wrong.

An Unused Signal: Crop Retention

The authors identify an unused signal already available in the augmentation pipeline: the fraction of the original image retained by each crop. Ranking views by this quantity yields measurable improvements:

Better accuracy than entropy on 14 of 15 benchmark datasets.
Better accuracy than the strongest scoring function tested on 11 of 15 datasets.
A pre-method measurement also predicts which datasets will benefit.

Significance in 2026

As vision-language models are increasingly deployed in open-world settings—where distribution shift is the norm rather than the exception—test-time adaptation has become a practical necessity. This audit challenges a widely adopted design choice (entropy-based view selection) and offers a simple, nearly free alternative grounded in the augmentation process itself. The results argue for auditing the informational assumptions behind adaptation heuristics, not just refining their scoring functions.

Citation

@article{sukumar2026entropy,
  title={What Entropy-Based View Selection Cannot See: An Audit of Test-Time Adaptation for Vision-Language Models},
  author={Sukumar, Sivasuthan and Thayasivam, Uthayasanker},
  journal={arXiv preprint arXiv:2611.14686},
  year={2026},
  note={Accepted at NeurIPS 2026 Workshops on VLM4RWD}
}

DOI: https://doi.org/10.48550/arXiv.2611.14686

via ArXiv CV

What Entropy-Based View Selection Cannot See: An Audit of Test