Two document-parsing benchmarks — first on one, mid-table on the other, and why

The pipeline that turns PDFs and images into markdown was evaluated on two public benchmarks. DP-Bench extracts a document into element-level JSON and scores table structure and text sequencing; OmniDocBench converts page images to markdown and scores edit distance, table accuracy and reading order.

The two benchmarks disagreed. That disagreement is what this record is about.

What was evaluated is the production path

This is not a pipeline built for the benchmark. The product’s OCR logic was faithfully reimplemented in Python, with the same model, the same prompt and the same post-processing. The only difference is authentication: production goes through a proxy with a session token while the benchmark calls the API directly. The conversion logic is identical.

That matters because a number measured on a benchmark-only path guarantees nothing about the quality a user actually receives.

DP-Bench — first on table structure

200 documents, the same metrics as the official leaderboard.

Parser	TEDS	TEDS-S	NID
This pipeline	94.63	98.11	92.21
Upstage (enhanced)	95.59	97.62	96.62
Upstage (standard)	96.06	97.25	96.29
AWS	95.48	96.99	95.97
LlamaParse	90.73	93.20	90.53
Microsoft	77.85	85.74	87.03
Google	78.30	80.71	82.17

On TEDS-S, which measures table structure, this is first at 98.11. It beats commercial parsers fine-tuned specifically for document parsing, while being a general-purpose VLM with a prompt on top.

On TEDS, which includes cell contents, and NID, which measures text sequencing, it places high but not first. That means the cell layout is right while the text inside some cells is wrong, and the same distinction reappears in the OmniDocBench results below.

Cost was 6.5 seconds per document and $2.39 for all 200.

OmniDocBench — mid-table, and why

1,651 pages, end-to-end markdown scoring.

Metric	This pipeline	Leaderboard best
Text-Edit (lower is better)	0.094	0.036
Table-TEDS	79.23	93.42
Table-TEDS-S	85.50	95.92
Read-Order-Edit (lower is better)	0.183	0.116

This is a loss. The top of that leaderboard is entirely small VLMs fine-tuned for document parsing. Breaking the loss down by language and document type shows exactly where it comes from.

Slice	Text-Edit (lower is better)
English	0.060
Academic literature	0.043
Exam papers	0.172
Simplified Chinese	0.113
Mixed English and Chinese	0.203
Handwriting	0.365

On English and well-structured documents it is competitive with the top; on Chinese, mixed-script and handwritten pages it collapses. That is a general-purpose VLM losing to specialised models on dense CJK and handwriting OCR, and it is not a gap a prompt can close.

Topping this benchmark therefore requires a document-specific model. The product’s goal is general source extraction feeding a knowledge graph, which makes a general-purpose VLM the reasonable choice, and this result is the price of that choice written down as a number.

Chasing one low score to the bottom

DP-Bench’s Table-F1 came out low at 0.5654. Against 55 ground-truth tables the pipeline predicted 136, and it predicted tables in 49 documents whose ground truth contains none. That reads like a defect that over-produces tables.

All 49 were checked against the source markdown and the ground-truth categories.

Cause	Documents
Chart converted into a data table (labelled Chart in the ground truth)	44
Table of contents rendered as a table (labelled Index)	2
Figure caption wrapped in a one-row table	2
Callout layout expressed as an HTML table	1

44 of them are intended behaviour. The prompt explicitly instructs the model to turn charts into data tables. The purpose is knowledge-graph extraction, so structuring the numbers inside a chart rather than letting them pass through as a picture is correct, and the score is what happens when that collides with the benchmark’s Chart-versus-Table taxonomy.

The adapter did not invent tables either: the number of pipe tables in the source markdown matches the number of tables the adapter emits exactly. Nor is it model hallucination.

Only 3 cases are worth fixing, and their effect is negligible. The production behaviour should stay as it is.

Without that investigation there were two options: turn off chart structuring for the score, or leave a low score standing as an unexamined defect. Both are wrong, and which one applied could only be settled by opening 49 documents.

Limits

Absolute comparisons are approximate. The leaderboard is scored on OmniDocBench v1.6_full while this run scored against a 1,651-page snapshot. The scorer and metric definitions match, but the ground truth is not identical.
No overall score was produced. The overall metric requires the formula score, which was disabled to shorten scoring time.
Reading order depends on the model’s output order, with no layout detection. Post-processing for multi-column documents would help some, but the ceiling is the model’s.

Measurement record

Item	Value
When	July 2026
Model	`gemini-3.5-flash` via OpenRouter, the same as production
Prompt	The OCR prompt verbatim from the product’s `ingest.rs` (merged cells as HTML tables, charts as data tables, formulas as LaTeX)
Post-processing	The product’s `tidy_markdown`, reimplemented as-is
DP-Bench	200 documents; NID, TEDS and TEDS-S; 6.5 s per document; $2.39 total
OmniDocBench	1,651 pages; Text-Edit, Table-TEDS and Read-Order-Edit
Scoring environment	Three separate Python 3.11 virtualenvs (runner, OmniDocBench scorer, DP-Bench scorer)

The reimplementation lives in harness/ocr_common.py, the batch runners in harness/run_omni.py and harness/run_dp.py, and harness/download_data.py fetches the datasets from HuggingFace snapshots. The primary record and the configuration files are under docs/pdf-parse-eval/ in the Consilience repository.