Author: Latha Shree
Editor: Vahe Aslanyan
A practical engineering handbook on document intelligence - layout analysis, structured extraction, and semantic understanding beyond character recognition.
OCR answers one question: “What characters are on this page?” Document intelligence answers harder questions:
This handbook covers the document understanding layer that sits between raw PDF parsing and downstream processing - translation, data extraction, redaction, summarization, or any transformation that must preserve document structure. The patterns described here are drawn from production document-processing systems where computer vision, heuristic classification, and LLM-based extraction work together.
If OCR is the eyes, document intelligence is the brain that knows what it’s looking at.
Consider three documents:
Running a plain OCR engine on each page would produce text, but:
Document intelligence prevents these failures by identifying structure before text is grouped for processing.
A robust understanding layer consists of five components:
1. Layout Analysis (vision model) ← Vision: what regions exist?
2. Table Detection (OCR in table ROI) ← OCR: where are cell boundaries?
3. Paragraph Finding (spatial grouping) ← Structure: what is a paragraph?
4. Styles & Formulas (expression detect) ← Semantics: what must not be altered?
5. Term Extraction (LLM glossary build) ← Meaning: what terms matter?
Each component operates on an Intermediate Language (IL) - a per-character representation with bounding boxes, fonts, and page geometry. Nothing in this stack produces final output; everything enriches the IL so downstream transformation and typesetting can make informed decisions.
Document layout models such as DocLayout-YOLO are object detection models trained on document layout datasets. A common deployment pattern is to export the model to ONNX and run it via ONNX Runtime on CPU:
class OnnxLayoutModel(DocLayoutModel):
@staticmethod
def from_pretrained():
pth = get_layout_onnx_model_path()
return OnnxLayoutModel(pth)
ClassDownstream ImpactparagraphDefines text block boundaries for groupingheadingTriggers heading-specific typesetting (larger font, spacing)tableTriggers the table parser with targeted OCRlistPreserves bullet/number structurefooterIdentifies running footers (often skipped or handled separately)figure_captionGroups caption text with figure referencestitleDocument/chapter title detection
Without layout analysis, text extraction follows PDF content stream order - which often differs from visual reading order. Layout regions define the correct grouping:
Page with two columns:
Without layout: "Introduction Methods Results Discussion" (garbled)
With layout: Col1: "Introduction Methods"
Col2: "Results Discussion"
The paragraph finder respects layout region boundaries and never merges text across region edges.
For GPU-accelerated inference, layout detection can be exposed as a remote RPC endpoint. Some deployments also return an OCR match score for quality feedback. This separates CPU-bound parsing from GPU-bound vision inference - a common production pattern for burst workloads.
Tables are the most layout-sensitive element in documents. A transformed table that loses its grid structure is worse than an untouched one.
Stage 1: Region detection - the layout model identifies table class regions on the page.
Stage 2: Cell OCR - a lightweight OCR model runs only within detected table bounding boxes:
class TableOcrModel:
# Uses a compact text-detection ONNX model
# Detects text lines within the table ROI
# Converts detections to layout-result format for IL integration
This targeted approach avoids running OCR on the entire page. Only table regions - typically 5-15% of page area - get cell-level detection.
Government and institutional forms often use table-like layouts for field grids. The form pipeline adds specialized logic:
def explode_form_table_rows(paragraph, layout_region):
"""Split a form table row into individual field labels."""
def clamp_form_line_paragraph_boxes(paragraph, region):
"""Prevent form labels from overflowing their grid cells."""
def assign_form_horizontal_option_boxes(paragraph, options):
"""Map checkbox/radio options to their label paragraphs."""
Form tables differ from data tables: each cell is a label-value pair, not a data row. The paragraph finder must explode rows into individual processable units while preserving the grid geometry for typesetting.
Form and table regions benefit from dedicated scale floors in typesetting:
FORM_SCALE_FLOOR = 0.65 # Minimum font scale for form labels
FORM_TABLE_SCALE_FLOOR = 0.55 # Even tighter for dense form grids
Without scale floors, rewritten labels (often longer than the source) shrink to illegible sizes to fit original cell dimensions.
Forms are the most challenging document type for transformation systems. They combine fixed layout, dense labels, checkboxes, and legal text that LLMs often refuse to process.
Before pipeline selection, sample the first couple of pages and run heuristic detection:
FORM_MARKERS = (
"application form", "passport", "for official use only",
"date of birth", "surname", "given names",
# ... dozens of markers across multiple languages
)
def looks_like_form_pdf(path: Path) -> bool:
text = extract_pdf_text_sample(path).lower()
marker_hits = sum(1 for m in FORM_MARKERS if m in text)
numbered_fields = len(NUMBERED_FIELD_RE.findall(text))
checkbox_density = text.count("□") + text.count("☐")
if marker_hits >= 3: return True
if marker_hits >= 2 and numbered_fields >= 4: return True
if numbered_fields >= 8 and checkbox_density >= 2: return True
return False
Detection triggers a form pipeline profile with:
ChallengeSolutionLLM refuses to process legal textForm-specific system prompt + commentary rejectionInvented [[WORD]] placeholdersUnwrap hallucinated bracket wrappers in post-processingOverlapping label textForm paragraph box clampingCheckbox symbols alteredToken protection for □, ☐, ☑Numbered field labels mergedForm table row explosionDense layout overflowForm scale floors in typesetting
Form-specific tests should verify:
Not all text should be transformed. Mathematical expressions, chemical formulas, code snippets, and styled inline spans need detection before the LLM call.
The styles-and-formulas stage identifies:
def is_formula_font(font_name: str) -> bool:
"""Detect math-specific fonts."""
def detect_inline_math_boundary(chars) -> tuple[int, int]:
"""Find start/end of inline math expression."""
Detected formulas become formula objects with {vN} placeholders. The LLM receives E=mc{v1} instead of E=mc², and the placeholder is restored during typesetting.
A critical engineering decision: when is a token a math expression vs. a product code?
InputClassificationReasonE=mc²FormulaMath operators, formula fontT15Text (inline)Short alphanumeric product code04XText (inline)SKU fragment1–4Text (inline)Numeric range, not mathLiPo 2SMixed"LiPo" = protected token, "2S" = inline text∫f(x)dxFormulaIntegral notation, formula font
Getting this wrong produces the most visible output bugs. An explicit should-inline-formula-as-text decision function should encode these rules based on production experience with operator manuals and technical documents.
Documents with mixed formatting (bold terms, italic emphasis, size changes) need style capture per character. Rich-text processing preserves inline styles through the transformation:
Rich-text processing should be disabled in OCR workaround mode (scanned documents) because styled span reconstruction is unreliable on image-layer text.
Images and vector graphics are preserved from the original PDF - they are not re-rendered or re-OCR’d. The pipeline:
Text rendered as part of an image (diagram labels, screenshot annotations, watermark text) does not appear in the IL character stream. It bypasses the entire processing pipeline.
Detection signal: text appears in the final PDF but not in the pipeline’s processing log.
Common examples from production:
Fix location: post-render cleanup, not the core pipeline. These require pattern-based replacement in the rendered PDF content streams.
When the vast majority of pages (e.g. >80%) are scanned images:
This produces a readable output document where the original visual layout (photos, diagrams, logos) remains intact.
Before main processing, an LLM pass extracts key terms from the entire document:
[
{"src": "FPV goggles", "tgt": "<consistent target rendering>"},
{"src": "bind procedure", "tgt": "<consistent target rendering>"},
{"src": "LiPo battery", "tgt": "<consistent target rendering>"}
]
This stage can consume a significant share of pipeline time (~30% in some systems) but ensures:
User-provided glossaries are enforced with a compiled pattern-matching engine (e.g. Hyperscan):
# Builds a compiled pattern database from glossary entries
# Matches terms in source paragraphs before the LLM call
# Injects enforced renderings into prompt context
For a legal contract with 200 defined terms, compiled pattern matching adds milliseconds. Linear scanning would add seconds per page.
Technical documents often include symbol-definition lists:
FPV - First Person View
ELRS - ExpressLRS
CRSF - Crossfire
LiPo - Lithium Polymer
These sections need special handling in text flow reconstruction:
DEFINITION_HEADING_RE = re.compile(r"(?i)(nomenclature|symbols|abbreviations|glossary)")
DEFINITION_ROW_RE = re.compile(r"^([A-Z]{2,})\s*[-–—]\s*(.+)$")
Known pitfall: two-column symbol lists can merge into one paragraph when proximity-based grouping crosses column boundaries. The usual fix path is line-level processing or improved table detection for nomenclature sections.
TOCs are deceptively complex. They combine hierarchical numbering, dot leaders, page references, and multi-line titles - all of which break during transformation.
FailureCauseExampleMerged page numbersLong rewritten title joins next entry’s number"372.3" instead of "37 ⋯ 2.3"Dot leader fragmentsLeaders become separate paragraphs". . . ." as a standalone text blockMissing section numbersOCR drops numbers during extraction"8.1" becomes "8 1" or missingDuplicated numbersOCR and the LLM both add numbers"818.1"Unprocessed entriesShort entries skipped by minimum text lengthSource-language titles left in the output TOC
Processing layer: a TOC normalization pass collapses dot-only leader fragments, compresses runaway dotted leaders, and restores known missing section numbers.
Render layer: post-render cleanup detects TOC pages structurally (early-page position, numbered entries, dotted leader patterns) and redraws with measured layout:
This produces visually justified TOCs that match the source document’s appearance while containing correctly processed entries.
The understanding layer should behave differently based on document profile:
ComponentStandardPaperFormScan detectionSkipped (fast mode)EnabledEnabledFormula detectionBasicFullMinimalAuto-glossaryOffOnOffTable parserOnOnOn (form-aware)Min text lengthDefaultDefault1TypesettingAuto-fit fillAcademic spacingForm scale floorsSystem promptAnti-refusalAcademicForm-specific
Profile resolution:
def resolve_pipeline_type(requested, input_path, *, file_type="pdf"):
if requested in {"paper", "form"}:
return requested
if file_type != "pdf" or input_path is None:
return "standard"
if looks_like_form_pdf(input_path):
return "form"
return "standard"
Users can force a profile via API/CLI, or rely on automatic detection.
How do you know document understanding is working? Measure these:
Track these metrics per document profile. Form documents will have different thresholds than academic papers.
Document intelligence transforms OCR from a character recognition problem into a structure understanding problem. The components - layout analysis, table detection, form processing, formula classification, and term extraction - each address a specific class of document complexity that raw OCR cannot handle.
The engineering patterns in this handbook come from production experience processing operator manuals, government forms, academic papers, and legal documents across dozens of languages. The recurring theme: understand document structure before transforming content.
OCR tells you what characters exist. Document intelligence tells you what they mean and how they relate to each other. Build the intelligence layer first - output quality follows from understanding.
%402x.svg){kind=icon}