Media Ingestion Pipeline

Relevant source files

The following files were used as context for generating this wiki page:

• Docs/Code_Documentation/Ingestion_Media_Processing.md
• tldw_Server_API/app/core/Ingestion_Media_Processing/README.md
• Docs/Product/Media_Ingestion_Pipeline_PRD.md
• Docs/Architecture.md
• Docs/Code_Documentation/Ingestion_Pipeline_MediaWiki.md
• AGENTS.md

The Media Ingestion Pipeline is a core component of the tldw_server architecture, responsible for the safe and pluggable ingestion of heterogeneous media. It facilitates the extraction of text and metadata from various sources—including audio, video, PDF, EPUB, and web content—to provide structured, normalized outputs for downstream modules such as RAG, full-text search, and persistent storage.

Sources: tldw_Server_API/app/core/Ingestion_Media_Processing/README.md:3-6, Docs/Architecture.md:46-48

Architecture and Data Flow

The pipeline follows a DB-agnostic design where processing logic is decoupled from persistence. The flow moves from input validation through normalization, parsing, and optional chunking or analysis before returning a structured dictionary aligned with the system's response schemas.

Core Components and Modules

The pipeline is organized into specialized submodules, each handling specific file formats or processing tasks.

Validation and Security (Upload_Sink.py)

All ingested files pass through a security gate that enforces MIME type checks, extension allowlists, and size limits. It optionally supports YARA scanning and safe extraction of archives to prevent "zip bombs" or path traversal attacks.
Sources: Docs/Code_Documentation/Ingestion_Media_Processing.md:24-40, tldw_Server_API/app/core/Ingestion_Media_Processing/README.md:37-41

Media-Specific Pipelines

Pipeline	Technologies Used	Key Features
Audio/Video	yt-dlp, ffmpeg, faster_whisper, NVIDIA NeMo	STT, Diarization, VAD, timestamp generation.
PDF	PyMuPDF, pymupdf4llm, Docling	Markdown conversion, OCR fallback (Tesseract), VLM table extraction.
Plaintext/Docs	BeautifulSoup, docx2txt, Pandoc	Support for .txt, .md, .html, .xml, .docx, .rtf.
Books	EbookLib	EPUB spine extraction, TOC to Markdown conversion.
MediaWiki	mwxml, mwparserfromhell	Streaming XML dump parsing with checkpointing.

Sources: Docs/Code_Documentation/Ingestion_Media_Processing.md:46-130, tldw_Server_API/app/core/Ingestion_Media_Processing/README.md:8-16

Processing Logic and Transformations

Normalization and Output Shape

Processors are designed to be "pure functions" where feasible. They normalize diverse inputs into a consistent MediaItemProcessResponse format.
Sources: tldw_Server_API/app/core/Ingestion_Media_Processing/README.md:21-22

Field	Description
status	Processing state (Success, Error).
media_type	Detected type (e.g., video, pdf).
metadata	Extracted headers, authors, timestamps.
content	Raw extracted text or transcript.
chunks	Segments generated by the chunking engine.
analysis	Optional AI-generated summary or insights.

Sources: Docs/Code_Documentation/Ingestion_Media_Processing.md:6-8, tldw_Server_API/app/core/Ingestion_Media_Processing/README.md:17-19

Chunking and Analysis

The pipeline integrates with the unified Chunking module to split text into manageable segments for vector embeddings. Following chunking, optional LLM analysis can be triggered to generate summaries, key takeaways, or claims extraction.
Sources: Docs/Code_Documentation/Ingestion_Media_Processing.md:144-150, tldw_Server_API/app/core/Ingestion_Media_Processing/README.md:24-25

API Endpoints

The FastAPI layer provides several endpoints for both ephemeral processing (no database writes) and full ingestion (persistence to Media_DB_v2).

Configuration and Governance

Ingestion behavior is controlled via loaded_config_data['media_processing'].

• Size Limits: Configurable via keys like max_pdf_file_size_mb.
• Resource Governance: Ingestion endpoints enforce require_permissions(MEDIA_CREATE) and utilize rate limiters to prevent resource exhaustion.
• Sanitization: Defaults for HTML/XML sanitization are controlled by sanitize_html_uploads and sanitize_xml_uploads.

Sources: Docs/Code_Documentation/Ingestion_Media_Processing.md:162-180, tldw_Server_API/app/core/Ingestion_Media_Processing/README.md:52-54

Summary

The Media Ingestion Pipeline serves as the gateway for all knowledge entering the tldw_server. By utilizing a modular, DB-agnostic architecture, the system ensures that regardless of the input format, the resulting data is normalized and ready for advanced retrieval and analysis. Current efforts focus on "Stage 3" goals, which include agentic follow-up workflows and structured quality scoring for ingestion runs.
Sources: Docs/Product/Media_Ingestion_Pipeline_PRD.md:13-17, tldw_Server_API/app/core/Ingestion_Media_Processing/README.md:3-6