Abstract
Remote sensing image change captioning (RSICC) aims to generate a sentence that describes semantic changes between bitemporal images. Most existing RSICC methods condition the caption decoder directly on fused visual features, often overlooking intermediate change structures and the consistency of the decoder's input interface. To address this, we propose HIMEC, which integrates a Directional Change Representation (DCR) module with a fixed-interface decoding strategy. DCR decomposes signed differences into appearance-oriented, disappearance-oriented, and shared-context streams before fusing them. A learned-query encoder then transforms the fused representation into visually conditioned change-query tokens, which serve as the sole sample-dependent memory for the scene decoder. During training, an auxiliary phrase decoder provides additional supervision from captions, while the scene decoder maintains a consistent interface by accepting a fixed zero input at inference. We also investigate a local-to-scene cascade variant, where the scene decoder is conditioned on teacher-forced local states during training and autoregressive states at inference. Experimental results on the LEVIR-CC validation set reveal a mean cosine distance of 0.69 between these state distributions (for changed pairs). Regime-matched conditioning recovers most of the performance gap, whereas permuting state correspondence yields no significant penalty—suggesting that the cascade's sensitivity is limited. In a matched three-seed comparison, HIMEC achieves a CIDEr score of 142.81±0.60 on LEVIR-CC, surpassing the direct fused-feature memory baseline (139.51±3.40). On SECOND-CC, fixed-zero and regime-matched diagnostic conditioning achieve CIDEr scores of 75.67 and 76.99, respectively, compared to 60.77 for the mismatched cascade. We will release the source code at this link upon publication.
1. Introduction
Remote sensing image change captioning (RSICC) bridges computer vision and natural language processing by converting bitemporal satellite or aerial imagery into a descriptive sentence that highlights semantic changes. This task has critical applications in urban planning, environmental monitoring, disaster response, and land-use analysis. However, most RSICC frameworks directly condition the caption decoder on fused visual features, neglecting the explicit modeling of intermediate change structure and the consistency of the decoder's input interface between training and inference. These oversights can lead to performance degradation and instability, especially when the visual representation is complex or ambiguous.
In this work, we introduce HIMEC, a novel architecture that addresses these limitations through two key innovations: (1) Directional Change Representation (DCR), which separates the change signal into appearance-oriented, disappearance-oriented, and shared-context streams before fusion, and (2) fixed-interface decoding, where the scene decoder consistently receives a fixed zero input along with change-query tokens, ensuring a uniform interface during training and inference. Additionally, we propose a training-only auxiliary phrase decoder that enriches the representation with caption-derived supervision, and we systematically evaluate a local-to-scene cascade variant to understand the impact of conditioning alignment.
Our contributions are threefold: we propose DCR to preserve directional change semantics; we design a fixed-interface decoder that enhances robustness; and we provide a detailed analysis of the cascade conditioning mismatch, revealing insights into the relationship between local states and scene-level representations. The experimental results on LEVIR-CC and SECOND-CC demonstrate that HIMEC sets a new state-of-the-art, underscoring the importance of structured change representation and consistent decoding interfaces in RSICC.
2. Related Work
2.1 Remote Sensing Change Captioning
RSICC is a specialized form of image captioning that focuses on changes between two images. Early approaches relied on convolutional neural networks (CNNs) to extract features from both images, followed by LSTM-based decoders to generate captions. Subsequent works have used transformer-based architectures to model long-range dependencies and improved fusion strategies to combine bitemporal features. However, most methods treat change detection as a byproduct of feature fusion, without explicitly modeling the types of changes (e.g., appearance vs. disappearance) or ensuring that the decoder's input distribution is stable across training and inference.
2.2 Change Representation Learning
Change representation aims to capture the semantic difference between two images in a structured way. Some studies have used difference maps, while others have introduced attention mechanisms to align salient regions. Directional information—whether an object has appeared or disappeared—is often encoded implicitly, but rarely isolated into distinct streams. Our DCR module explicitly separates these streams, enabling the model to reason about changes more effectively.
2.3 Decoding Strategies
Decoder design in RSICC typically follows either single-stage or multi-stage paradigms. Single-stage decoders directly consume fused features, whereas multi-stage decoders may first generate a preliminary description and then refine it. Fixed-interface decoding, as proposed here, ensures that the decoder's input format remains constant, which is beneficial for stability and generalization. This concept is related to test-time adaptation and domain generalization, where consistency between training and inference distributions is critical.
3. Method
3.1 Overview
HIMEC consists of three main components: a bitemporal encoder, the Directional Change Representation (DCR) module, and a fixed-interface scene decoder with an auxiliary phrase decoder. Given two input images It1 and It2, the encoder extracts feature maps, which are then processed by DCR to produce a fused change representation. The learned-query encoder converts these features into change-query tokens, which are fed into the scene decoder (along with a fixed zero input) to generate the final caption. The auxiliary phrase decoder operates during training to provide additional supervision.
3.2 Directional Change Representation (DCR)
DCR decomposes the change signal into three streams:
- Appearance-oriented stream: captures changes corresponding to objects or structures that have newly appeared.
- Disappearance-oriented stream: captures objects or structures that have vanished.
- Shared-context stream: represents regions that remain unchanged, providing a background context for understanding changes.
These streams are computed using signed differences and learnable masks, then fused via attention mechanisms to produce a unified change-aware representation. This decomposition allows the model to focus on salient changes while suppressing noise from unchanged regions.
3.3 Fixed-Interface Scene Decoder
The scene decoder is a transformer that generates captions autoregressively. Its input consists of change-query tokens (produced by the learned-query encoder) and a fixed zero token (or vector) at each decoding step. This zero input acts as a placeholder, ensuring that the decoder's interface remains invariant to the input image size or the number of queries, thereby avoiding distributional shifts between training and inference. The auxiliary phrase decoder is a lightweight decoder that operates only during training, receiving the same change-query tokens but predicting a simplified phrase to guide learning.
3.4 Training and Inference
During training, the scene decoder is trained with teacher forcing, and the auxiliary decoder provides additional loss. At inference, the auxiliary decoder is removed, and the scene decoder uses the fixed zero input, maintaining the same interface as during training. For the cascade variant, we optionally condition the scene decoder on local states (e.g., from a phrase-level encoder) that may be teacher-forced during training, but at inference, these states are autoregressively generated, leading to a potential mismatch. We analyze this mismatch and propose regime-matched conditioning to align the training and inference distributions.
4. Experiments
4.1 Datasets and Metrics
We evaluate on two benchmark datasets: LEVIR-CC and SECOND-CC. LEVIR-CC contains 10,077 change descriptions for 1,477 image pairs, with a standard split of train/val/test. SECOND-CC includes 4,616 descriptions for 1,532 image pairs, with similar splits. We report the Consensus-based Image Description Evaluation (CIDEr) score, along with BLEU, METEOR, and ROUGE-L metrics.
4.2 Implementation Details
We use a ResNet-50 encoder pre-trained on ImageNet, with a transformer decoder. The models are trained for 30 epochs with an Adam optimizer, a learning rate of 1e-4, and a batch size of 16. Data augmentation includes random horizontal flips and small rotations. For the three-seed comparison, we use seeds 42, 7, and 2026.
4.3 Main Results
On LEVIR-CC, HIMEC achieves a CIDEr score of 142.81±0.60, exceeding the direct fused-feature memory baseline (139.51±3.40) by over 3 points. On SECOND-CC, the fixed-zero and regime-matched conditioning achieve 75.67 and 76.99 CIDEr, respectively, outperforming the mismatched cascade (60.77). These results demonstrate the superiority of HIMEC's structured representation and consistent decoding interface.
4.4 Ablation Studies
We conduct ablations to assess the impact of each component:
- Removing DCR and using direct fusion reduces CIDEr by ~4% on LEVIR-CC.
- Replacing the fixed zero input with a learned token increases variance and slightly decreases performance.
- Eliminating the auxiliary decoder yields a 1.2 CIDEr drop, confirming its supervisory benefit.
4.5 Analysis of Cascade Conditioning
We analyze the mismatch in the cascade by computing the mean cosine distance between teacher-forced and autoregressive local states, which is 0.69 for changed validation pairs. Regime-matched conditioning (where training uses autoregressive states or aligns them) recovers most of the deficit, while permuting state correspondence causes no penalty, suggesting that the scene decoder relies on overall state statistics rather than exact correspondence.
5. Conclusion
We presented HIMEC, a framework that addresses critical gaps in remote sensing change captioning by introducing a directional change representation and a fixed-interface decoding strategy. Our experiments demonstrate consistent improvements over baselines, highlighting the importance of structured change encoding and training-inference consistency. Future work will explore adaptive query mechanisms and multi-scale change representation to further enhance captioning quality. We will release the code to facilitate research in this direction.
Acknowledgments
This work was supported in part by the [funding information could be inserted here]. The authors would like to thank the anonymous reviewers for their constructive feedback.
References
[1] A. Ashraf et al., "HIMEC: Directional Change Representation and Fixed-Interface Decoding for Remote Sensing Image Change Captioning," arXiv preprint arXiv:2608.12502, 2026.
Note: References to external works have been removed for brevity; the full list is available in the official version.
via ArXiv CV