OCR & Vision Capabilities
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The OCR (Optical Character Recognition) and Vision capabilities within the tldw_server project form a critical part of the Media Ingestion & Processing pipeline. These systems are designed to extract structured information from visual media, including PDFs, images, and videos. By converting visual data into machine-readable text and metadata, these capabilities enable full-text search, AI-driven analysis, and complex RAG (Retrieval-Augmented Generation) workflows across diverse source materials.
The vision subsystem is architected as a set of pluggable adapters and core modules that normalize visual content. This allows the platform to ingest messy source material—such as scanned documents or video frames—and transform them into semantic units that can be indexed in the vector store and queried by LLM providers.
Sources: AGENTS.md:37-41, Docs/Architecture.md:46-50, CLAUDE.md:143-147
Core Architecture and Data Flow
The OCR and Vision systems operate within the app/core/Ingestion_Media_Processing/ directory. The architecture follows a modular pattern where visual media is passed through specialized processors depending on the input type.
Visual Processing Pipeline
When visual media is ingested, it typically follows a flow from raw file extraction to normalized text:
- Media Ingestion: Raw files (PDFs, images) or video streams are received.
- Visual Extraction: Specialized libraries (e.g.,
pymupdf,docling) extract image data or text layers. - OCR Processing: For scanned images or frames without text layers, OCR engines extract the literal text content.
- Normalization: Extracted text and visual metadata are converted into chunks and stored in the Media DB v2.
The diagram above illustrates the decision logic between direct extraction and OCR processing for visual media.
Sources: Docs/Architecture.md:120-130, CLAUDE.md:67-72
Key Components
| Component | Description | Relevant Paths |
|---|---|---|
| OCR Engines | Pluggable adapters for extracting text from images and scanned documents. | app/core/Ingestion_Media_Processing/OCR/ |
| Document Processing | Libraries like pymupdf and docling used for PDF and document analysis. | CLAUDE.md:67-72 |
| Vision Adapters | Interface for connecting to external vision models for image analysis. | Docs/Architecture.md:142-146 |
| Figure Extraction | Specialized logic for identifying and extracting tables/figures from documents. | Docs/Product/UX_REVIEW_PROMPT_MEDIA_PAGES.md:120-125 |
OCR Implementation Details
OCR is specifically triggered during the ingestion of PDFs and images when standard text extraction is insufficient. The project integrates multiple OCR engines and scrapers to ensure high fidelity.
OCR Process Flow
The system manages OCR as part of the background services. For large documents, this processing is chunked to ensure system stability and provide progress updates.
The sequence diagram shows the interaction between the API, processing logic, and OCR engines during document ingestion.
Sources: Docs/Architecture.md:162-170, Docs/Product/UX_REVIEW_PROMPT_MEDIA_PAGES.md:27-35
Document Intelligence and Vision Analysis
Beyond simple text extraction, the vision capabilities extend into "Document Intelligence." This includes the ability to recognize structure and non-textual elements within documents.
Advanced Visual Extractions
The backend exposes specific endpoints (though not always surfaced in the primary UI) that leverage vision models to provide deeper insights:
- Outline/ToC Extraction: Identifying the hierarchy of a document visually.
- Reference Extraction: Identifying citations and references within a visual layout.
- Figure & Table Extraction: Isolating visual data elements for independent analysis.
- Visual Insights: AI-generated insights based on document structure and visual cues.
Sources: Docs/Product/UX_REVIEW_PROMPT_MEDIA_PAGES.md:120-130, Docs/Prompts/UX_MEDIA_MULTI_REVIEW_PROMPT.md:130-140
Multi-Item Comparison and Diffing
The Vision-related UI (specifically in the Media Multi page) supports visual comparisons. While the "Diff" algorithm often operates on text, the vision system provides the underlying data for comparing visual assets side-by-side in "Compare" or "Stack" modes.
Sources: Docs/Prompts/UX_MEDIA_MULTI_REVIEW_PROMPT.md:40-55, Docs/Product/UX_REVIEW_PROMPT_MEDIA_PAGES.md:86-100
Configuration and Management
OCR and Vision settings are typically managed via the centralized configuration file or environment variables.
Relevant Configuration Parameters
| Parameter | Type | Description |
|---|---|---|
OCR_ENGINE | String | Defines the default OCR provider (e.g., Tesseract or cloud-based). |
useMediaRichRendering | Boolean | Feature flag for enabling advanced visual/HTML rendering of content. |
has_original_file | Boolean | Metadata flag indicating if the source image/PDF is available for visual inspection. |
Sources: Docs/Architecture.md:215-225, Docs/Product/UX_REVIEW_PROMPT_MEDIA_PAGES.md:65-75
Conclusion
OCR and Vision capabilities are essential for tldw_server to fulfill its role as a comprehensive research assistant. By bridging the gap between static visual media and searchable, analyzable data, these systems allow the platform to process complex PDFs, scanned documents, and video content with high accuracy. The modular architecture ensures that as new OCR engines and vision models emerge, they can be integrated as adapters into the existing ingestion pipeline.
Sources: AGENTS.md:15-20, Docs/Architecture.md:12-18