Abstract
Large vision-language models (VLMs) demonstrate impressive in-context learning (ICL) capabilities, yet the precise conditions under which visual context proves beneficial for multimodal ICL remain poorly understood. Empirical findings reveal a puzzling dichotomy: models sometimes effectively exploit visual demonstrations, yet in other instances completely disregard them. We introduce VIB-ICL, an information-theoretic framework that resolves this paradox through the lens of the Information Bottleneck (IB) principle. Specifically, we propose the Cross-Modal Information Gain (CMIG), which quantifies the additional mutual information that visual context contributes about the target beyond what is available from textual context alone. We derive a novel generalization bound showing that the excess risk of multimodal ICL over text-only ICL is governed by CMIG, demonstrating that multimodal ICL provably outperforms its text-only counterpart when visual information is non-redundant. Furthermore, we prove that visual context neglect—previously viewed as a failure mode—is actually the IB-optimal strategy when visual information is redundant. This yields a closed-form Attention Reallocation Principle that prescribes adaptive adjustment of visual attention weights. We instantiate this principle through the VIB-ICL algorithm, which estimates CMIG via variational bounds and dynamically reallocates attention across modalities. Experiments on five benchmarks demonstrate consistent improvements, achieving up to 4.7% accuracy gains and 35% reduction in required demonstrations, thereby validating our theoretical predictions and offering a principled understanding of attention dynamics in multimodal ICL.
1. Introduction
The rapid advancement of large vision-language models (VLMs) has transformed the landscape of multimodal artificial intelligence, enabling unprecedented performance across tasks that require joint understanding of visual and textual information. Among their many capabilities, in-context learning (ICL) stands out as a particularly powerful ability: without any gradient updates, these models can adapt to new tasks by conditioning on a handful of demonstration examples provided in the prompt. This emergent property has made ICL a cornerstone of modern few-shot learning pipelines.
However, the integration of visual demonstrations into ICL presents a peculiar and unresolved challenge. Empirical studies have consistently shown a striking inconsistency in how VLMs utilize visual context: in some scenarios, visual demonstrations dramatically boost task performance, while in others, models either ignore the visual input entirely or, worse, suffer from performance degradation. This phenomenon—commonly termed visual neglect—has puzzled researchers since the advent of multimodal ICL and remains poorly characterized from a theoretical standpoint.
Recent work in 2026 has intensified efforts to understand these attention dynamics, yet a unifying framework that explains both successful utilization and neglect of visual context remains elusive. The Information Bottleneck principle, introduced by Tishby et al. (2000) and subsequently extended for deep learning applications, offers a promising theoretical lens. By formalizing the trade-off between compression and predictive accuracy, the IB framework can characterize how models should optimally allocate representational resources—including attentional capacity—across information sources.
In this paper, we address this gap by proposing VIB-ICL, a variational information bottleneck framework for adaptive attention in multimodal ICL. Our key contributions are as follows:
- The Cross-Modal Information Gain (CMIG) quantifies the additional mutual information that visual context provides about the prediction target beyond textual context.
- A theoretical generalization bound establishes that multimodal ICL's excess risk relative to text-only ICL is directly governed by CMIG, providing formal conditions under which visual demonstrations provably help.
- A counter-intuitive theorem proves that visual neglect can be the IB-optimal strategy—not a failure mode—when visual information is redundant with textual content.
- The Attention Reallocation Principle derives from this theorem, providing a closed-form solution for how visual attention weights should be adaptively modulated.
- The VIB-ICL algorithm implements this principle by estimating CMIG through variational bounds and dynamically reallocating attention weights, yielding consistent improvements across five standard benchmarks.
- Extension to other modalities: The framework naturally generalizes to audio, video, or multimodal mixtures.
- Training-time integration: While VIB-ICL operates at inference time, incorporating CMIG estimates during training may yield even stronger results.
- Theoretical refinement: Tightening the generalization bound and extending it to non-ideal predictors remains an open challenge.
2. Background and Related Work
2.1 In-Context Learning in Multimodal Models
In-context learning refers to a model's ability to perform new tasks by conditioning on demonstration examples without parameter updates. While extensively studied in purely textual domains (Brown et al., 2020; Dong et al., 2023), multimodal ICL extends this paradigm to vision-language settings, where demonstrations typically comprise image-text pairs. Despite empirical successes, theoretical understanding of when multimodal context helps remains fragmented.
Recent works have proposed various perspectives: some view multimodal ICL as implicit gradient descent (Dai et al., 2023), others analyze attention patterns in transformer layers (Tsai et al., 2024), while information-theoretic analyses remain relatively scarce, particularly regarding cross-modal interactions.
2.2 Visual Neglect in VLMs
The tendency of VLMs to ignore visual information has been documented across multiple benchmark studies (Yin et al., 2024; Lin et al., 2025). However, most works treat this as an optimization or alignment failure. Our framework challenges this interpretation, showing that visual neglect can be a rational, optimal response to information redundancy—a perspective with significant implications for model design and prompt engineering.
2.3 Information Bottleneck Principle
The Information Bottleneck method (Tishby et al., 2000) formalizes the trade-off between compressing input representations and preserving predictive information. Its variational formulation (Alemi et al., 2017) has enabled scalable implementation in deep learning. In the context of multimodal models, the IB principle naturally extends to characterizing how information from different modalities should be weighted or discarded.
3. Framework and Theoretical Analysis
3.1 Problem Formulation
We consider a multimodal ICL setting with an input sequence consisting of demonstrations and a query, where each demonstration comprises a visual input xv and textual input xt, and the query requires predicting a target label y. Let V denote the visual random variable, T the textual random variable, and Y the target. Standard ICL predicts Y based on both V and T, while text-only ICL relies solely on T.
3.2 Cross-Modal Information Gain
We define the Cross-Modal Information Gain as:
CMIG = I(V; Y | T)
This quantity measures the reduction in uncertainty about Y that visual input provides given that textual context is already available. When CMIG > 0, visual information is non-redundant; when CMIG ≈ 0, visual information contributes nothing beyond textual context.
3.3 Generalization Bound
Our first principal result establishes a direct connection between CMIG and generalization performance:
Theorem 1 (Excess Risk Bound). Let Rmultimodal and Rtext be the expected risks of the multimodal and text-only ICL predictors, respectively. Then, with high probability,
Rmultimodal - Rtext ≤ 2√(2D_KL(P(V|T)‖Q(V|T)) · CMIG)
where the KL-divergence term captures the information-bottleneck compression trade-off. This bound implies that multimodal ICL provably outperforms text-only ICL whenever CMIG exceeds a threshold that depends on the compression cost, formalizing conditions under which visual demonstrations are beneficial.
3.4 Optimality of Visual Neglect
Our second theoretical contribution challenges conventional wisdom about visual neglect:
Theorem 2 (Neglect Optimality). Under the Information Bottleneck objective, when CMIG ≈ 0—i.e., visual information is redundant given textual context—the optimal solution assigns zero attention to visual features. Visual neglect, in this case, is not a failure but the information-theoretically optimal behavior.
This result provides a principled explanation for the observed dichotomy: models appear inconsistent because they are, in fact, correctly modulating attention based on the redundancy structure of the input, albeit without an explicit control mechanism.
3.5 Attention Reallocation Principle
From Theorem 2, we derive a closed-form guidance for attention adjustment:
Principle 1 (Attention Reallocation). For any layer l in the transformer, the optimal visual attention weight α_v^l satisfies:
αv^l ∝ exp(β · CMIG^l - γ · DKL(P(V^l|T^l)‖Q(V^l|T^l)))
where β and γ are temperature parameters, and CMIG^l measures local cross-modal information gain. This principle prescribes that visual attention should increase proportionally to the unique information that visual features provide, and decrease when such information is compressible or redundant.
4. The VIB-ICL Algorithm
4.1 Overview
VIB-ICL operationalizes the theoretical framework through a two-stage approach: (1) estimation of CMIG through variational bounds, and (2) dynamic reallocation of attention weights according to the Attention Reallocation Principle.
4.2 Variational CMIG Estimation
Direct computation of CMIG = I(V; Y|T) is intractable in high-dimensional spaces. We instead optimize a variational lower bound:
CMIG ≥ E[log q(y|xv, xt)] - E[log q(y|x_t)]
where q represents the model's predictive distribution. This bound can be computed efficiently by comparing model predictions with and without visual input, without requiring explicit density estimation.
4.3 Dynamic Attention Reallocation
Given estimated CMIG values, VIB-ICL adjusts attention weights at each layer:
αv^l ← αv^l · exp(λ · CMIĜ^l) / Z
where λ controls adaptation strength and Z ensures normalization. This adjustment is performed as a post-hoc modification during inference, requiring no additional training.
4.4 Implementation Details
VIB-ICL is implemented as a lightweight wrapper around standard VLM inference, adding minimal computational overhead (approximately 3% latency increase). The algorithm operates in three steps: first, baseline predictions are generated; second, CMIG is estimated by perturbing visual inputs; third, attention is reallocated and final predictions are produced.
5. Experiments
5.1 Setup
We evaluate VIB-ICL across five benchmark tasks spanning visual question answering, image captioning, and visual reasoning: VQAv2, OK-VQA, ScienceQA, VizWiz, and NLVR2. We use three VLMs: LLaVA-NeXT (8B), InstructBLIP (7B), and Qwen-VL (7B). Results are reported for 4-shot and 8-shot ICL settings.
5.2 Results
| Benchmark | Model | Text-only | Multimodal | + VIB-ICL | Δ Accuracy |
|-----------|-------|-----------|------------|-----------|------------|
| VQAv2 | LLaVA-NeXT | 58.2 | 61.4 | 63.1 | +1.7 |
| OK-VQA | LLaVA-NeXT | 44.8 | 47.3 | 50.2 | +2.9 |
| ScienceQA | InstructBLIP | 72.1 | 75.6 | 80.3 | +4.7 |
| VizWiz | Qwen-VL | 51.5 | 53.2 | 55.8 | +2.6 |
| NLVR2 | LLaVA-NeXT | 70.4 | 72.8 | 74.1 | +1.3 |
Across all benchmarks, VIB-ICL delivers consistent accuracy improvements ranging from +1.3 to +4.7 percentage points over standard multimodal ICL. Notably, VIB-ICL also reduces the required number of demonstrations by up to 35%—achieving comparable accuracy to text-only ICL at 8 shots using only 5.2 shots on average.
5.3 Analysis of Attention Dynamics
To validate the Attention Reallocation Principle, we examine attention patterns under VIB-ICL. Consistent with Theorem 2, we observe that VIB-ICL suppresses visual attention when CMIG is low (e.g., when textual descriptions fully capture visual content), and amplifies visual attention when CMIG is high (e.g., for tasks requiring fine-grained visual reasoning).
6. Discussion and Future Directions
The VIB-ICL framework contributes both theoretical understanding and practical methodology to multimodal ICL. Our results suggest that consistent success in multimodal ICL requires not merely better models, but adaptive mechanisms that modulate attention according to the information structure of each input.
Key open questions include:
7. Conclusion
We have presented VIB-ICL, an information-theoretic framework for understanding and improving attention in multimodal in-context learning. By formalizing visual context's contribution through the Cross-Modal Information Gain, we derived the conditions under which visual demonstrations provably help or optimally should be ignored. The resulting Attention Reallocation Principle provides a closed-form adjustment strategy, instantiated in the VIB-ICL algorithm. Experiments demonstrate significant and consistent improvements over baseline approaches, validating our theoretical analysis. This work reframes visual neglect as a feature—not a bug—of optimal information processing, opening new avenues for principled multimodal learning system design.
via ArXiv CL+LG