Abstract
Open-vocabulary Earth observation (EO) aims to localize geospatial concepts expressed in natural language rather than relying on a fixed, predefined label set. However, existing benchmarks in this domain typically cover a narrow range of category vocabularies or limited query formats. To address this gap, we introduce OVEarth-Bench, a benchmark that advances current evaluation practices along two key dimensions: category breadth, achieved through broad, hierarchical category coverage incorporating both positive and negative expressions; and query diversity, encompassing vocabulary-based, referring, and reasoning query types. The benchmark supports both mask and box localization tasks under a unified zero-shot evaluation protocol.
We assess a wide range of general-purpose and EO-specific methods. Our evaluation reveals three principal findings: (1) the overall performance of current methods remains limited, although broader category coverage results in more stable model rankings; (2) multimodal large language model (MLLM)-based methods achieve the highest overall performance; and (3) EO-specific methods generally underperform their general-purpose counterparts and rarely match the top-performing models. These insights offer valuable guidance for the future design of open-vocabulary EO methods and underscore the necessity of developing more realistic, diverse, high-quality, and large-scale benchmarks to enable reliable evaluation. Our data and evaluation toolkit are publicly available at earth-insights.github.io/OVEarth-bench.
1. Introduction
Earth observation (EO) technologies have become indispensable for monitoring and understanding our planet, with applications spanning environmental conservation, disaster response, and urban planning. Traditional EO models are trained to recognize a fixed set of predefined categories, which limits their adaptability to the vast and ever-evolving array of geospatial concepts. Open-vocabulary EO methods have emerged as a promising alternative, enabling models to interpret and localize arbitrary geospatial concepts described in natural language.
Despite this progress, the field lacks a robust and comprehensive benchmark that adequately captures the complexity of real-world EO tasks. Most existing evaluations are constrained by narrow category vocabularies, which fail to challenge models with the breadth of concepts encountered in practice. Furthermore, the diversity of query forms—ranging from simple object naming to complex spatial reasoning—is often overlooked, leading to an incomplete assessment of model capabilities.
This paper presents OVEarth-Bench, a new benchmark designed to address these limitations. By systematically expanding category breadth and query diversity, we provide a more realistic and rigorous evaluation framework. Our benchmark enables researchers to gauge a model's true zero-shot generalization ability and offers deeper insights into the strengths and weaknesses of various methodological approaches.
2. The OVEarth-Bench Benchmark
OVEarth-Bench is constructed to fill the gaps left by previous benchmarks. This section details the design principles and components that constitute the benchmark.
2.1 Design Goals
The benchmark is built around two central pillars:
- Category Breadth: Moving beyond narrow label sets, OVEarth-Bench incorporates a broad, hierarchical taxonomy of geospatial categories. This includes not only positive expressions (e.g., "residential area") but also negative expressions (e.g., "non-residential area"), pushing models to understand concepts in a more nuanced manner.
- Query Diversity: The benchmark features a diverse set of query forms, including vocabulary queries (direct object identification), referring queries (identifying objects based on descriptive context), and reasoning queries (requiring spatial or relational inference).
- MLLM-Based Methods Dominate: Methods leveraging Multimodal Large Language Models (MLLMs) achieve the strongest overall performance across all query types. Their strength lies in the capacity to fuse visual and textual information, enabling superior understanding of complex, free-form queries.
- EO-Specific Methods Fall Short: Counterintuitively, many methods tailored specifically for EO tasks underperform compared to general-purpose models. These specialized models often suffer from overfitting to narrow training data and fail to generalize to the diverse queries in our benchmark.
- Stability Over Breadth: The stable rankings observed with broader category coverage highlight that a well-designed benchmark must be comprehensive to truly differentiate the capabilities of various models.
2.2 Task Protocol
OVEarth-Bench supports two fundamental EO tasks: mask localization (pixel-level segmentation) and box localization (bounding box detection). Both tasks are evaluated under a unified zero-shot protocol, meaning the models are tested on categories and queries they have never encountered during training. This design ensures that performance reflects the model's ability to generalize, not its ability to memorize.
The benchmark comprises a large-scale collection of geospatial imagery paired with rich, natural language annotations. All annotations are validated for quality and consistency to ensure fair and reliable evaluation.
3. Experimental Evaluation
To demonstrate the utility of OVEarth-Bench, we conducted a comprehensive evaluation of a wide range of methods, including both general vision-language models and those specifically designed for EO tasks. This section summarizes our key findings.
3.1 Performance Overview
The evaluation results reveal that the performance of current methods is still far from satisfactory, indicating significant room for improvement. Notably, we observe that when models are evaluated across a broader category spectrum, their performance rankings become more stable and reliable, suggesting that limited category sets may lead to inflated or misleading assessments.
3.2 Method Comparison
Our study yields three critical findings:
4. Implications and Future Directions
OVEarth-Bench provides critical insights that can shape future research in open-vocabulary EO. The clear advantages of MLLM-based approaches suggest a paradigm shift towards integrating larger, more flexible model architectures. Conversely, the underperformance of EO-specific methods calls for rethinking their design, potentially by pre-training on larger, more diverse datasets.
Furthermore, our results emphasize the need for future benchmarks to be more realistic, diverse, and large-scale. Static, narrow benchmarks are insufficient to guide the development of models that can operate effectively in the dynamic and varied settings typical of real-world Earth observation.
5. Conclusion
OVEarth-Bench fills a critical void in the open-vocabulary Earth observation landscape by providing a comprehensive evaluation framework that emphasizes category breadth and query diversity. Our findings challenge existing assumptions and provide a clear, data-driven path forward. By benchmarking several current methods, we demonstrate significant performance gaps and highlight the superior generalization of MLLM-based models. We strongly believe that OVEarth-Bench will serve as a foundational tool for accelerating progress in this vital field, encouraging the creation of more powerful and versatile geospatial AI systems.
The dataset and evaluation package are publicly released to facilitate adoption and promote reproducibility in future research.
via ArXiv CV