Article
Title: Does Marginal Coverage Guarantee Class-Conditional Safety for Zero-Shot VLMs Under Shift?
Authors: Jai Kumar Sharma, Amartya Dutta
Published: arXiv:2608.19376 [cs.CV], submitted on 19 Aug 2026. Accepted at the ECCV 2026 Workshop on Uncertainty Quantification for Computer Vision (UNCV). 34 pages (16 main + 18 supplementary), 10 figures.
Abstract
Split-conformal prediction provides marginal coverage under exchangeability and is increasingly used as an abstention layer for zero-shot vision-language models (VLMs). This paper audits that practice under deployment shift for CLIP, OpenCLIP, and SigLIP across ImageNet and non-ImageNet settings. We find that marginal coverage can remain relatively high even while class-conditional tail coverage collapses. On ImageNet-Sketch, worst-class coverage falls to approximately zero, and 10–12% of classes remain below a finite-sample null floor—despite a marginal coverage rate of around 0.86. This failure aligns with target-domain class accuracy but is not predicted by the source-domain diagnostics we test. Source-side Mondrian calibration improves the in-distribution tail, but the improvement does not transfer to shifted domains. Clustered conformal and Conf-OT methods improve marginal or average metrics but do not recover the worst-class tail. Target-side class calibration substantially lifts the tail, but requires labels for every class and remains set-size-intensive. We further identify a 2–3× cross-family efficiency gap and show that native SigLIP sigmoid scores remove APS's probability-mass interpretation. These findings persist across the tested model scale, pretraining corpus, prompt, miscoverage level α, and shifted non-ImageNet settings. Marginal conformal coverage should therefore be treated as an average reliability statistic, not as a safety guarantee for the class tail.
Introduction
Conformal prediction has become a standard tool for uncertainty quantification, providing distribution-free guarantees under exchangeability. In the context of zero-shot vision-language models (VLMs)—such as CLIP, OpenCLIP, and SigLIP—split-conformal prediction is frequently used as an abstention mechanism: the model withholds predictions when the conformal set is too large or non-specific. However, the safety of this approach in deployment scenarios with distribution shift remains an open question. This paper investigates whether marginal coverage guarantees—the overall probability that the true class is included—translate into class-conditional safety, particularly for the worst-performing classes.
Our study spans multiple model architectures, pretraining corpora, and dataset shifts, including ImageNet and non-ImageNet settings. We evaluate source-domain diagnostics and several calibration methods to determine their effectiveness in maintaining tail coverage under shift. The results reveal a critical gap: while marginal coverage may appear acceptable (e.g., 0.86 on ImageNet-Sketch), the tail coverage for many classes collapses to near zero, undermining the reliability of the abstention layer.
Methods and Findings
Marginal Coverage vs. Class-Conditional Tail Coverage
Under exchangeability, split-conformal prediction ensures marginal coverage, but this guarantee does not extend to the class-conditional tail. In our experiments, the worst-class coverage on ImageNet-Sketch fell to approximately zero, and 10–12% of classes fell below a finite-sample null floor, despite a marginal coverage of 0.86. This indicates that the average reliability metric masks severe under-coverage for the least frequent or most challenging classes.
Source-Domain Diagnostics and Transfer
We tested several source-domain diagnostics to predict tail failure, but none reliably forecasted the collapse. Mondrian calibration on the source data improved in-distribution tail coverage but did not transfer to shifted domains. Clustered conformal methods and Conf-OT (Confidence-Optimal Transport) improved marginal or average metrics but failed to recover the worst-class tail. Only target-side class calibration—requiring labels for every class—substantially lifted the tail, but at the cost of intensive labeling and larger set sizes.
Cross-Family Efficiency Gap and Score-Space Issues
We identified a 2–3× cross-family efficiency gap: different VLM families (CLIP vs. OpenCLIP vs. SigLIP) exhibit markedly different conformal set sizes for the same coverage level. Furthermore, we show that native SigLIP sigmoid scores invalidate the probability-mass interpretation required by Adaptive Prediction Sets (APS), complicating its use as a conformal score.
Discussion and Implications
Our findings hold across a wide range of model scales, pretraining corpora, prompts, miscoverage levels (α), and shifted non-ImageNet settings. The consistent pattern—marginal coverage remains high while tail coverage collapses—suggests that practitioners should not interpret marginal guarantees as a safety guarantee for the class tail. Instead, they should report additional metrics such as worst-class coverage and class-conditional null floor exceedance.
For deployment in safety-critical applications, we recommend either target-side calibration (when labels are available) or developing new methods that explicitly optimize for class-conditional coverage under shift. The observed efficiency gap also motivates research into more efficient conformal procedures that are robust across VLM families.
Conclusion
Marginal conformal coverage is an average reliability statistic, not a safety guarantee for the class tail. As zero-shot VLMs are increasingly deployed in open-world settings, the reliance on marginal coverage alone can lead to dangerously overconfident abstention layers. We hope this audit encourages the community to adopt stricter evaluation protocols that emphasize worst-case behavior.
Cite as: arXiv:2608.19376 [cs.CV] (or arXiv:2608.19376v1 [cs.CV] for this version). DOI: https://doi.org/10.48550/arXiv.2608.19376
via ArXiv CV