We Benchmarked 7 OCR Models So You Don't Have To

Most OCR benchmark commentary has the same problem: it treats the leaderboard as the story.

That is backwards.

The leaderboard is the output. The interesting part is how fragile that output becomes once you change the inference path, the prompt, the rendering settings, or the post-processing pipeline.

So instead of rephrasing vendor claims, we ran the benchmark ourselves.

This article covers our own olmOCR-bench reproductions across classic OCR, OCR-specific vision-language models, and a few general-purpose VLMs. The benchmark itself is explained in more detail on the olmOCR-Bench page and in the broader Document Processing Benchmarks guide. Here I want to focus on the part that actually matters: what happened when we ran the thing.

The Short Version#

Three findings matter more than the exact ranking:

1. The benchmark pipeline matters a lot. The same model can look broken or competitive depending on whether you use the official pipeline or a "close enough" custom path.
2. Small OCR-specific models are already very good. GLM-OCR at 0.9B and dots.ocr at about 3B are much more serious than the parameter counts suggest.
3. Generic VLMs are not useless, but they fail in predictable ways. They can do math and plain text surprisingly well, then fall apart on layout, tables, or hallucination control.

That is the real takeaway. Not "model X beat model Y by 1.3 points."

What We Actually Ran#

Our current olmOCR-bench result set uses the official upstream evaluator on the Allen AI dataset snapshot: 1,403 PDFs and 8 score buckets, including the easy baseline category. The runs include:

• OCR-specific models such as olmOCR-2-7B, GLM-OCR, PaddleOCR-VL-1.5, LightOnOCR-2-1B, and dots.ocr-1.5
• classic OCR via Tesseract 5.5.2
• several general-purpose Qwen vision models
• a few additional document models such as DeepSeek-OCR-2 and GOT-OCR 2.0

For cross-model comparison, I care about two score views:

• Overall score: includes the easy baseline pages
• Score excluding baseline: a stricter number and usually the more honest one

That distinction matters. Tesseract looks much less embarrassing when the baseline pages are mixed in. Once you remove them, reality returns.

The Current Table#

Here are the runs that are already strong enough to talk about:

Model	Overall	Excluding baseline	Notes
dots.ocr-1.5	82.7%	81.4%	Best score in our current runs, layout-aware parser pipeline
olmOCR-2-7B	82.2%	79.7%	Closest reproduction target because it is the benchmark creator's own model
Qwen3.5-4B	79.1%	n/a	General-purpose VLM, better than expected
DeepSeek-OCR-2	78.6%	75.6%	Promising quality, expensive in wall-clock time
PaddleOCR-VL-1.5	77.3%	74.2%	Respectable only once we switched to the official pipeline
GLM-OCR	75.7%	72.2%	Excellent value for a 0.9B model
LightOnOCR-2-1B	75.5%	72.0%	Strong small model, competitive with GLM-OCR
Qwen3.5-2B	72.5%	n/a	Better than the tiny model, still not an OCR specialist
Qwen3.5-0.8B	57.6%	n/a	Good on some tasks, clearly not enough on full-doc parsing
Tesseract 5.5.2	34.1%	24.8%	Fine for plain text, non-competitive for math/tables
GOT-OCR 2.0	27.6%	17.7%	Poor fit for this benchmark

The top-line ranking is useful, but the more interesting part is why those numbers land where they do.

The First Real Win: We Validated Our Eval Stack#

The most strategically important run was not the highest score. It was olmOCR-2-7B.

Allen AI built both the benchmark and one of the best-performing models on it. That makes it the perfect canary for reproduction fidelity. If our pipeline were sloppy, this is where it would show.

Our reproduced result for olmOCR-2-7B landed at 82.2%, against an official reference of 82.4% +/- 1.1. Seven of eight categories were within one percentage point. That is close enough to treat the pipeline as trustworthy.

That matters more than a flashy chart.

If you cannot reproduce the benchmark creator's own reference model, the rest of your comparison table is theater.

The Most Important Lesson: Methodology Swings Scores Hard#

The cleanest example is PaddleOCR-VL-1.5.

In the early custom Hugging Face and vLLM path, the model looked weak. Using the wrong prompt and the wrong inference path dragged the score down into the mid-40s. That would have led to the obvious but wrong conclusion: the model is overhyped.

Then we switched to the official paddleocr pipeline, which actually matches how the model is supposed to run: layout detection first, per-region task routing after that, then markdown reassembly.

The result jumped to a benchmark-comparable score of 77.3% overall and 74.2% excluding baseline.

That is not a small correction. That is the difference between "this model is broken" and "this model is competitive."

Same weights. Different pipeline. Completely different story.

This is why I do not trust leaderboard snapshots without methodology details. A VLM used as raw page-to-text OCR is not the same system as the same VLM inside its intended parsing stack.

Small Models Are Better Than They Have Any Right To Be#

Two models stood out here: GLM-OCR and LightOnOCR-2-1B.

GLM-OCR is especially interesting because it is only 0.9B parameters and still reached 75.7% overall in our run, slightly above the author-reported score. That is with a relatively simple single-prompt path, not a giant multi-stage enterprise stack.

Its profile is exactly what you would want from a compact OCR model:

• very cheap to run
• small enough to deploy on modest hardware
• strong on math-heavy PDFs and tiny text
• clearly weaker on degraded scans and table structure

That last bullet matters. There is no free lunch. But the tradeoff is honest.

LightOnOCR-2-1B landed in roughly the same band. The interesting point is not which one is 0.2 points ahead on a given day. The interesting point is that the floor has moved. You no longer need a 7B or 30B multimodal model to get respectable document parsing quality on this benchmark.

That has practical consequences for cost, deployment, and experimentation speed.

dots.ocr Shows Why Pipeline Models Keep Winning#

The current top score in our own set is dots.ocr-1.5 at 82.7% overall.

The reason is not mystical. It is a pipeline.

Instead of asking one monolithic model to stare at a page and improvise, it does the less glamorous thing:

• detect layout
• classify regions
• OCR them with the right mode
• reassemble the result into markdown

That is exactly the kind of system that keeps winning in document AI. Not because it is conceptually elegant, but because documents are ugly. Multi-column layouts, formulas, page headers, and tables are different problems pretending to be one problem.

dots.ocr is also a good reminder that the best practical systems often come with annoying operational footnotes:

• gated or mirrored weights
• broken packaging
• custom parser code
• special server flags

That friction is not a side note. It is part of the product reality.

Tesseract Is Still Useful, Just Not For This Job#

Tesseract is the funniest result in the set because it looks decent right until you evaluate it properly.

With our earlier custom evaluator, it scored 78.2%. That sounded respectable.

Then we switched to the official evaluator. The score collapsed to 34.1% overall and 24.8% excluding baseline.

What happened? Nothing dramatic. We just stopped giving it credit for tasks it cannot do.

Tesseract can extract plain text. It cannot produce LaTeX math. It cannot reconstruct markdown tables. That is not a bug. It is just a different tool.

So the honest conclusion is:

• if you only need cheap text extraction, Tesseract still has a place
• if you want PDF linearization that preserves structure, it is not competitive

That is also a useful benchmark lesson. A lot of "OCR accuracy" discourse quietly collapses text recognition, structural parsing, and semantic preservation into one blob. The official olmOCR-bench does not let you cheat like that.

General-Purpose VLMs: Better Than Expected, Worse Than Needed#

The Qwen runs were useful because they answer a question many people have implicitly:

Can a normal multimodal model just do OCR if you prompt it correctly?

The answer is: kind of.

The smaller Qwen models were surprisingly good at:

• dense text
• some mathematical notation
• basic reading order on simpler pages

But they also failed in exactly the places you would expect:

• multi-column layouts
• tables
• output discipline
• hallucination control on complex pages

The 2B model in particular had the classic partial-success pattern: it could read the page, then ruin the result by hallucinating giant repeated table structures. That is worse than just failing cleanly.

This is the pattern I would keep in mind going forward:

• generic VLMs can be decent OCR ingredients
• they are not automatically good document parsers

That may sound obvious, but the market keeps pretending otherwise.

Speed and Cost Matter More Than Leaderboard Culture Admits#

One thing this round of experiments made very obvious: score deltas are often smaller than runtime and cost deltas.

A compact model like GLM-OCR is cheap enough that you can run the full benchmark for pocket change. dots.ocr is also practical. DeepSeek-OCR-2, by contrast, looks promising on quality but is much more annoying in real wall-clock terms.

That changes how I think about "best model."

If one model scores 82 and another scores 76, but the cheaper one:

• runs on commodity hardware
• is easy to serve
• is permissively licensed
• finishes much faster

then the six-point gap is not the whole story. For a lot of teams, it is not even the main story.

This is why I keep coming back to cost-quality framing. The benchmark score alone is not the decision. It is just one axis.

Why This Matters Beyond OCR#

The broader point is not really about OCR. It is about production systems.

Real document workflows are almost never "run one model on one page and trust the output." They are pipelines:

• OCR or document parsing
• layout and structure recovery
• extraction or schema mapping
• validation and arithmetic checks
• fallback logic for ugly cases
• retry and escalation paths when confidence is low

That is why I care so much about reproduction fidelity. If the first stage is mis-measured, the rest of the pipeline conversation starts on bad footing.

So yes, this article is about olmOCR-bench. But the practical use is bigger than that. It helps answer a more relevant question:

Which components are robust enough to deserve a place in a real workflow?

What This Does Not Prove#

It is worth being explicit about the limits.

This is not yet our German invoice benchmark.

olmOCR-bench is useful because it tests full-page PDF linearization across math, tables, tiny text, multi-column layout, old scans, and boilerplate removal. It is a good stress test for document parsing systems.

But it is still a public benchmark with its own biases.

It does not directly answer:

• which model is best for invoice field extraction
• which stack is best for line items
• which vendor handles German business documents best
• which system survives your own weird PDFs

That is the next layer of work. The current runs give us confidence in the infrastructure and in the benchmarking discipline. They are not the final commercial wedge by themselves.

If you need to make that invoice decision today, before our own invoice benchmark exists, the invoice OCR route guide is where we currently help operators choose which route to test first, by volume, privacy, validation needs, and how much evidence actually backs each option.

If You Only Remember Three Things#

• Reproduction fidelity matters. If your eval path differs from the intended pipeline, the score can move enough to reverse the conclusion.
• Small OCR-specific models are already strong. GLM-OCR and LightOnOCR show that document parsing quality no longer requires giant models.
• Benchmark scores are only useful when tied to method. A naked leaderboard number without prompt, rendering, and post-processing context is barely half an evaluation.

What We Do Next#

The next useful step is not another generic leaderboard page. It is to extend this work into datasets and scoring regimes that are closer to actual business document workflows:

• invoice fields
• line-item extraction
• arithmetic validity
• format checks like VAT ID and IBAN
• cost per processed document, not just score per benchmark

That is where the public benchmark story turns into a product story.

For now, though, this round of olmOCR-bench runs already establishes something important: the evaluation infrastructure is real, the methodology is starting to harden, and the gap between "author-reported" and "actually reproducible" is large enough to be worth writing about.

Related Reading#

• olmOCR-Bench: The benchmark page with category breakdowns and source context
• Document Processing Benchmarks: Making Sense of the Chaos: The mental model for evaluating document AI
• Document AI Benchmark Atlas: Which benchmark to use for which task
• Which Invoice OCR Route Should You Test First?: Turning this benchmark discipline into a concrete route decision for invoices