Regularizing Modality Contribution Drift in Multimodal Continual Learning

## Abstract Multimodal continual learning (MMCL) aims to acquire new knowledge from multimodal data while preserving previously learned information. Existing MMCL methods primarily focus on cross-modal representation alignment or semantic similarity to mitigate catastrophic forgetting. However, they overlook whether the relative contributions of individual modalities and their interactions remain stable across incremental tasks. We formalize this decision-level shift as Modality Contribution Drift (MCD) and introduce the MCD score, which quantifies changes in contribution strength and relative reliance under controlled interventions on modality subsets. Both theoretical and empirical analyses reveal why current MMCL methods fail to reliably control this drift. To address this gap, we propose Continual Modality Contribution Drift Regularization (CMCDR), a framework that preserves the modality contribution structure of previously learned tasks. Recognizing that MMCL settings differ in the availability of old exemplars, CMCDR offers two variants: a replay-based version and a replay-free version. The replay-based variant employs modality-subset interventions as diagnostic probes on stored old samples, comparing their contribution profiles between the current and frozen previous models, thereby constraining changes in modality-specific and interaction contributions. The replay-free variant instead uses current-task samples as probes and distills the frozen model's old-task contribution responses, allowing regularization without exemplar storage. Experiments on multimodal class-incremental learning and continual visual question answering validate the generality and effectiveness of CMCDR across diverse settings.

via ArXiv LG