CMNIE: An Information Extraction Benchmark for Chinese Military News
Authors: Yan Yu, Mengna Zhu, Zhenyu Song, Hao Yang, Haiwen Chen, Mao Wang
Published: Submitted on 9 September 2026 | Accepted at NLPCC 2026
Subjects: Computation and Language (cs.CL) | arXiv:2609.10722 [cs.CL]
Overview
Structured extraction from Chinese military news plays a critical role in intelligence analysis, decision-making, and knowledge base construction. However, existing resources offer limited support for joint information extraction in this domain—particularly when events, event arguments, entities, and relations must be modeled together.
To address this gap, we present CMNIE, an information extraction benchmark for Chinese military news. Extending military-domain resources beyond document-level event annotations, CMNIE jointly annotates event triggers, event arguments, named entities, and entity relations under a unified domain schema.
Dataset Description
CMNIE contains 13,000 instances collected from public Chinese military news sources, with manual annotations covering:
- 7 event types
- 10 argument roles
- 7 entity types
- 8 relation types
The unified domain schema enables joint modeling of event triggers, event arguments, named entities, and their relations—a departure from prior work that typically treats these tasks in isolation.
Experimental Evaluation
We evaluate three categories of methods on a shared test set:
- Supervised IE models
- Zero-shot large language models (LLMs)
- Fine-tuned LLM-based extraction methods
- Relation extraction
- Exact matching of event-argument spans
- Schema adherence — ensuring extracted outputs conform to domain-specific schemas
- Exact span matching — achieving precise boundary alignment with gold annotations
- Joint structured extraction — simultaneously modeling events, entities, arguments, and relations
- Pages: 13
- Figures: 3
- Venue: Accepted at NLPCC 2026
- Cite as: arXiv:2609.10722 [cs.CL]
- DOI: https://doi.org/10.48550/arXiv.2609.10722
Key Findings
Experimental results demonstrate that CMNIE remains a challenging benchmark. Performance is particularly limited in:
Notably, zero-shot LLMs frequently identify relevant semantic units but fail to match gold span boundaries exactly—highlighting a gap between semantic understanding and precise structured extraction that persists even in 2026's most capable models.
Significance
CMNIE provides a standardized benchmark for studying three key challenges in specialized Chinese news processing:
By releasing this benchmark at the intersection of domain specialization and joint extraction, CMNIE enables systematic comparison of extraction approaches as LLM-based methods continue to mature in 2026.
Metadata
via ArXiv CL+LG