OmniDocBench v1.5

V-token (visual tokens) is the maximum number of image patches the model processes per page. Think of it as the model's "retina resolution" — each token is a learned embedding of a small image region.

Why it matters:

• Cost — More tokens = higher API costs or GPU time
• Latency — Attention is O(n²), so 5× more tokens ≈ 25× more compute
• Context — Multi-page docs exhaust context windows fast

DeepSeek-OCR 2 uses only 1,120 tokens while most competitors use 6,000+, making it ~5× more efficient with comparable quality. This translates to significant cost savings at production scale.

They test different aspects of document processing:

OmniDocBench v1.5 evaluates end-to-end document parsing with granular metrics:

• Text extraction (edit distance)
• Table structure (TEDS)
• Formula recognition (CDM)
• Reading order preservation

olmOCR-Bench uses pass/fail "unit tests" for PDF linearization — more like "did you preserve the important bits?" than a quality score.

When to use each:

• OmniDocBench gives more detailed diagnostics — useful for understanding where your model fails
• olmOCR-Bench is faster to run and covers more document types (arXiv, historical scans, etc.)

Both are valuable; they're complementary, not competing. See our olmOCR-Bench deep dive.

Pipeline systems (marked with *) use separate specialized components — a layout detector, an OCR engine, a table parser, a formula recognizer — each with its own vision encoder.

There's no single "visual token budget" to report because:

• Each component may process images differently
• Some components don't use transformers at all
• The total compute isn't reducible to one number

The V-token metric only makes sense for unified VLM architectures that process the whole page in one forward pass.

OmniDocBench is from Shanghai AI Laboratory and academic collaborators, published at CVPR 2025.

• Dataset & toolkit: Hosted on OpenDataLab, their data platform
• Code: Available on GitHub
• Paper affiliations: Shanghai AI Laboratory, plus collaborators from Abaka AI and 2077AI

OpenDataLab is the data/benchmark platform project under Shanghai AI Lab — think of it as their "data hub" for AI research.

No — v1.5 was a major update with breaking changes:

• More pages in the dataset
• Higher-resolution images for some document types (newspapers, notes)
• Updated "hybrid matching" evaluation logic that merges/splits predictions to find best matches
• Different penalty behavior for minor formatting differences

A model that scored 88 on v1.0 might score 85 or 91 on v1.5. The benchmarks aren't directly comparable.

Always check which version a paper used before comparing results. This page uses v1.5 results only.

The DeepSeek rows are the paper authors' own evaluation runs, overlaid onto the OmniDocBench reference table.

• DeepSeek rows: Evaluated by the DeepSeek team for their paper
• Other rows: From the official OmniDocBench repository leaderboard

Both use the same evaluation protocol and code, so results are comparable. The label just indicates the source of the evaluation, not a different methodology.

The Overall formula explicitly focuses on three core pillars:

Overall = ((1 - Text_Edit) × 100 + Table_TEDS + Formula_CDM) / 3

Reading order is tracked separately because:

• It's a distinct challenge from text/table/formula quality
• It matters a lot for multi-column documents but less for simple single-column pages
• Including it would dilute the signal for the core capabilities

If your use case involves complex layouts (academic papers, newspapers, financial reports), don't rely on Overall alone — check the R-Order Edit column specifically.