LayoutLM

by Microsoft

Open SourceSelf-HostedMIT

Multimodal transformer that jointly models text, layout, and image for document understanding. State-of-the-art on form extraction benchmarks.

Layout AnalysisData ExtractionClassification
🤗 Hugging FaceGitHub📄 arXiv Paper

Overview

LayoutLM (and its successors LayoutLMv2, LayoutLMv3) is Microsoft's groundbreaking approach to document understanding. Unlike traditional OCR that treats documents as images or pure text, LayoutLM jointly learns from text content, 2D positional layout, and visual features.

The key insight is that document layout carries semantic meaning—field labels are typically positioned near their values, headers appear at the top, etc. By pre-training on millions of documents with these joint representations, LayoutLM achieves state-of-the-art results on form understanding, document classification, and key information extraction.

LayoutLMv3 unifies text and image masking with word-patch alignment, achieving 90%+ F1 on FUNSD and leading performance on DocVQA, CORD, and other benchmarks.

Strengths

  • State-of-the-art on FUNSD, CORD, DocVQA benchmarks
  • Joint text + layout + image understanding
  • Pre-trained models available on Hugging Face
  • Strong zero-shot transfer to new document types
  • Active research community and Microsoft backing

Limitations

  • Requires OCR as preprocessing step
  • Higher compute requirements than simple OCR
  • Fine-tuning needed for best results on specific tasks
  • Complex training setup for custom pre-training

Best Use Cases

  • Form field extraction (invoices, receipts)
  • Document classification
  • Key information extraction
  • Document visual question answering