Datalab Introduces OmniExtractBench to Fix Bias and Opacity in Extraction Benchmarks
Datalab has released OmniExtractBench, an open benchmark for structured document extraction. It tests how accurately a system fills a JSON schema from a PDF. The benchmark pools 620 documents from 4 existing benchmarks. One deterministic scorer grades all of them and explains each decision.
The release lands while extraction vendors publish their own leaderboards. Datalab argues those leaderboards are hard to compare or audit. OmniExtractBench is its attempt at a shared yardstick for the field.
Why a New Benchmark in 2026?
As document AI matures, the market has flooded with extraction tools—from OCR pipelines to vision-language models—each claiming state-of-the-art results on proprietary or selectively chosen datasets. In 2026, enterprises increasingly demand transparent, reproducible evaluation before adopting extraction systems for finance, healthcare, and legal workflows. OmniExtractBench arrives as a response to that demand, offering a vendor-neutral standard that anyone can run and verify.
What OmniExtractBench Measures
The benchmark focuses on a core capability: given a PDF, can a system correctly populate a predefined JSON schema? This mirrors real-world use cases such as invoice processing, form digitization, and contract analysis. By aggregating 620 documents from four existing benchmarks, OmniExtractBench aims to cover diverse layouts, languages, and document types—reducing the risk of overfitting to a single source.
A Deterministic Scorer That Explains Itself
Unlike leaderboards that rely on opaque scoring or subjective judgments, OmniExtractBench uses a single deterministic scorer across all documents. The scorer not only grades outputs but also provides explanations for each decision, enabling practitioners to audit failures and understand systematic biases. This transparency is intended to make comparisons between systems meaningful and actionable.
Toward a Shared Yardstick
Datalab positions OmniExtractBench as a community resource. By making the benchmark open and the scoring mechanism transparent, the company hopes to foster fair competition and accelerate progress in structured document extraction. As extraction vendors continue to publish their own leaderboards, OmniExtractBench offers a neutral alternative—one that prioritizes auditability and reproducibility over marketing claims.
For developers and researchers, the benchmark is available now. Those interested in testing their systems or contributing to the effort can find more details in Datalab's announcement.
via MarkTechPost