ParseBench-2: A Severity-Aware Benchmark for Document Parsing

ParseBench-2: A Severity-Aware Benchmark for Document Parsing Evaluation


Authors: Pierre-LoΓ―c Doulcet, Boyang Zhang, SebastiΓ‘n G. Acosta, Sacha Bron, Daniel B. Ospina, Antonio Jimeno Yepes, Nutchapol Dendumrongsup, Simon Suo


Subjects: Computer Vision and Pattern Recognition (cs.CV)


Submitted: 12 November 2026 | arXiv: 2611.17400 [cs.CV]




Overview


Document-parsing metrics often give high scores to outputs that alter important facts, while penalizing changes that preserve meaning. People can usually judge these consequences more easily than they can design a metric to measure them. To address this gap, the authors introduce a method for developing parsing metrics from severity policies that specify how each error type should affect the score.


Methodology


Severity policies are expressed through test scenarios that define how a metric should behave. Metric implementations are then optimized against these fixed expectations, using coding agents to refine comparison rules, error detectors, and weights under human review. This approach makes stated human priorities an explicit objective of metric design.


ParseBench-2 Benchmark


The method is applied in ParseBench-2, a comprehensive document-parsing benchmark that combines broad capability coverage with metrics calibrated to error severity. Key characteristics include:


  • 4,317 pages spanning six dimensions of parsing quality: text, formatting, tables, charts, visual grounding, and forms.
  • Detailed annotations and targeted difficult cases to detect changed facts, broken structural relationships, and missing content.
  • Severity-based scoring: the final score is the lower of a parsing score (measuring content recovery) and a severity cap (penalizing detected errors), so recovering most of a page cannot hide a serious error.

Results


Across six severity test suites, the proposed metrics more closely match the expert judgments expressed by the scenario targets than existing metrics, while penalizing at most 3.4% of tested benign variations.


Evaluation Scope


The benchmark provides a comprehensive evaluation spanning frontier vision-language models, specialized document parsers, and open-weight systems, comparing:


  • Overall rankings
  • Performance across parsing capabilities
  • Tradeoffs between accuracy and cost



Cite as: arXiv:2611.17400 [cs.CV] (or arXiv:2611.17400v1 [cs.CV] for this version)

via ArXiv CV

Related