A Developer's Guide to OmniParseBench: Leaderboard Benchmarks and OCR Testing

Overview

In this tutorial, we implement OmniParseBench, Datalab's open benchmark for document parsers. The benchmark grades a parser's output using 16,288 small pass-or-fail tests across 2,937 pages, spanning scanned forms and handwriting to arXiv math and dense tables.

Rather than trusting the leaderboard, we rebuild it from the verdicts Datalab released for 27 parsers. We first feed the scorer a hand-made page to see which mistakes flip a verdict. We then:

• Measure ranking certainty
• Slice results by document type
• Test whether the two models the card names as helping build tests behave differently on tests that a model corrected
• Run Tesseract and Indra OCR ourselves on a 50-page sample

Along the way, we find that the top two parsers are statistically tied, the leader changes on most slices, and one launch figure doesn't match the released files.

Getting Started

import os
import re
import sys
import json
import time
import html
import shutil
import platform
import traceback
import subprocess
import collections
import urllib.request
from pathlib import Path

RESULTS = {}  # step -> one-line result, for...

via MarkTechPost