ParseBench-2: A Severity-Aware Benchmark for Document Parsing Evaluation
Authors: Pierre-LoΓ―c Doulcet, Boyang Zhang, SebastiΓ‘n G. Acosta, Sacha Bron, Daniel B. Ospina, Antonio Jimeno Yepes, Nutchapol Dendumrongsup, Simon Suo
Subjects: Computer Vision and Pattern Recognition (cs.CV)
Submitted: 12 November 2026 | arXiv: 2611.17400 [cs.CV]
Overview
Document-parsing metrics often give high scores to outputs that alter important facts, while penalizing changes that preserve meaning. People can usually judge these consequences more easily than they can design a metric to measure them. To address this gap, the authors introduce a method for developing parsing metrics from severity policies that specify how each error type should affect the score.
Methodology
Severity policies are expressed through test scenarios that define how a metric should behave. Metric implementations are then optimized against these fixed expectations, using coding agents to refine comparison rules, error detectors, and weights under human review. This approach makes stated human priorities an explicit objective of metric design.
ParseBench-2 Benchmark
The method is applied in ParseBench-2, a comprehensive document-parsing benchmark that combines broad capability coverage with metrics calibrated to error severity. Key characteristics include:
- 4,317 pages spanning six dimensions of parsing quality: text, formatting, tables, charts, visual grounding, and forms.
- Detailed annotations and targeted difficult cases to detect changed facts, broken structural relationships, and missing content.
- Severity-based scoring: the final score is the lower of a parsing score (measuring content recovery) and a severity cap (penalizing detected errors), so recovering most of a page cannot hide a serious error.
Results
Across six severity test suites, the proposed metrics more closely match the expert judgments expressed by the scenario targets than existing metrics, while penalizing at most 3.4% of tested benign variations.
Evaluation Scope
The benchmark provides a comprehensive evaluation spanning frontier vision-language models, specialized document parsers, and open-weight systems, comparing:
- Overall rankings
- Performance across parsing capabilities
- Tradeoffs between accuracy and cost
Cite as: arXiv:2611.17400 [cs.CV] (or arXiv:2611.17400v1 [cs.CV] for this version)
via ArXiv CV
