Introduction
Vision-language models (VLMs) such as CLIP have demonstrated strong performance across a wide range of multimodal tasks by aligning visual and textual representations in a shared embedding space. As these models are increasingly deployed in high-stakes applications, the ability to predict failures has become essential for enabling risk-aware decision-making and timely human intervention.
Traditional failure prediction methods for VLMs typically rely on confidence scores or auxiliary classifiers. While effective in identifying when a model is likely to err, these approaches offer limited interpretability—they signal that a failure may occur but provide little insight into why. This gap motivates the need for more transparent failure prediction mechanisms that can expose the underlying causes of model errors.
In this work, we investigate the use of sparse autoencoders (SAEs) to address this challenge. SAEs have emerged as a powerful tool for decomposing neural network representations into sparse, interpretable features, offering a window into the model's internal decision-making process. We propose, for the first time, a framework that leverages SAEs for interpretable failure prediction in VLMs.
Our main contributions are as follows:
- We formulate failure prediction as a classification task over sparse SAE latent activations.
- We introduce a three-stage, failure-aware training pipeline that encourages latent directions to remain interpretable while becoming more informative for predicting failures.
- Our experiments show that the proposed framework outperforms existing baselines in failure prediction accuracy.
- We provide a detailed analysis revealing that failure-aware training steers SAE latent directions to capture more class-specific concepts. Additionally, concept-level inspection of model representations during failures highlights a shift from class-specific concepts toward ambiguous or style-related features.
- We further explore how learned SAE latent directions can support runtime failure recovery, offering a path toward more robust and self-aware VLM deployments.
Methodology
We treat failure prediction as a binary classification problem applied to the sparse latent activations of a VLM's intermediate representations. The challenge is to ensure that the SAE latents used for prediction are both accurate and interpretable—a balance that standard training objectives do not naturally encourage.
To address this, we design a three-stage training pipeline. In the first stage, we train an SAE on the VLM's representations to obtain a basis of sparse features. In the second stage, we fine-tune the SAE latent directions with a supervision signal derived from model failures, using a classification head to predict failure probability from the activations. The third stage fine-tunes the entire pipeline end-to-end to jointly optimize for prediction performance and interpretability.
Throughout, we introduce regularization terms that encourage the latent directions to be class-specific and semantically coherent, thereby maintaining interpretability without sacrificing predictive power.
Experiments and Results
We evaluate our framework on several standard multimodal benchmarks, comparing it against baseline methods that rely on confidence scores or auxiliary classifiers. Our method consistently outperforms these baselines in failure prediction accuracy, as measured by area under the ROC curve (AUC), across diverse datasets and VLM configurations.
Notably, when we probe the internal representations of the VLMs, we observe a clear pattern: failure-aware training drives SAE latent directions to align more closely with class-specific concepts. In contrast, during failure states, the model's representations shift toward ambiguous or style-related features—an insight that would be impossible to obtain with black-box confidence scores alone.
To further demonstrate the utility of the SAE, we conduct a qualitative analysis where we visualize the learned latent directions. This confirm that the directions correspond to human-understandable concepts, supporting the interpretability of our approach.
Toward Runtime Failure Recovery
A distinctive advantage of our framework is that the concept-level explanations it provides enable not just detection but also potential remediation. By identifying which class-specific concept directions are corrupted or missing during a predicted failure, we can adjust input representations or trigger targeted interventions. We explore this direction with a preliminary recovery mechanism, showing that it can partially reverse predicted failures, opening the door to more resilient VLM applications in the future.
Conclusion
We have presented FailSAE, a novel framework that combines sparse autoencoders with failure-aware training to deliver interpretable and accurate failure prediction for vision-language models. Our results demonstrate clear improvements over existing methods and provide valuable insights into the internal shifts that accompany model failures. The ability to explain and potentially recover from failures is critical as VLMs become integrated into high-stakes, real-world systems. Future work will extend this framework to other architectures and explore more sophisticated recovery mechanisms.
Acknowledgements
We thank the anonymous reviewers for their constructive feedback. This work was supported in part by [funding information].
via ArXiv CV