Project Overview & Vision
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
tldw_server (Too Long; Didn't Watch Server) is an open-source research assistant and media analysis platform designed to help users ingest, transcribe, analyze, and interact with diverse media formats. It operates as a FastAPI-based backend integrated with a Next.js WebUI, providing a comprehensive research workbench that users can self-host to maintain complete control over their source material, transcripts, and analysis artifacts.
Sources: AGENTS.md:5-9, Docs/Website/index.html:150-155
The long-term vision of the project is inspired by "The Young Lady's Illustrated Primer" from Neal Stephenson's The Diamond Age. The project aims to evolve into a personal knowledge assistant that facilitates self-paced learning and research. While currently in a beta state, the system provides practical tools for media ingestion, hybrid Retrieval-Augmented Generation (RAG), and multi-provider LLM orchestration.
Sources: AGENTS.md:11-15, Docs/Website/index.html:157-160
Core Project Vision
The overarching goal is to create a "street-smart primer"—a lightweight service that distills complex streams of information into actionable guidance. By allowing users to keep the entire stack on their own infrastructure, tldw_server ensures that research remains private and persistent, avoiding the data lock-in common with hosted SaaS products.
Sources: Docs/Website/index.html:125-130, Docs/Website/index.html:157-162
High-Level Capabilities
| Capability | Description |
|---|---|
| Media Ingestion | Support for Video, Audio, PDF, EPUB, DOCX, HTML, and MediaWiki dumps. |
| Transcription | STT via faster_whisper, NVIDIA NeMo, and real-time WebSocket streaming. |
| Hybrid Search | Combining SQLite FTS5, vector embeddings (ChromaDB), and re-ranking. |
| Agentic Interface | Production-ready Model Context Protocol (MCP) implementation. |
| Self-Hosting | Designed for local deployment with no telemetry or data collection. |
| Sources: AGENTS.md:32-75, Docs/Website/index.html:168-185 |
System Architecture
The project follows a "thin endpoint, fat core" philosophy, where the FastAPI routers handle request validation while the domain logic is pushed into specialized core modules.
The architecture separates concerns into clients, API routers, domain-specific core packages, and a multi-tenant storage layer.
Sources: Docs/Architecture.md:20-35, Docs/Architecture.md:65-90
Runtime Flow: Media to RAG
A typical data flow involves moving from raw media ingestion to searchable vector context:
- Ingestion:
Ingestion_Media_Processingdownloads or transcodes media usingyt-dlporffmpeg. - Normalization: Content is split into chunks via the
Chunkingmodule. - Embeddings:
Embeddingsmodule computes vectors and writes to ChromaDB. - Retrieval: The
RAGmodule utilizes FTS5 and vector search to serve queries.
Sources: Docs/Architecture.md:143-162
Key Modules and Components
LLM Orchestration
tldw_server supports 16+ LLM backends through a unified interface in LLM_Calls. This allows the system to route requests to commercial providers (OpenAI, Anthropic, Google, etc.) or local inference engines (Ollama, vLLM, llama.cpp) seamlessly.
Sources: AGENTS.md:58-63, Docs/Architecture.md:113-116
Model Context Protocol (MCP)
The project includes a production-grade MCP Unified module. It maps internal modules like media, knowledge, and notes to tools for agentic clients, supporting RBAC and real-time metrics.
Sources: Docs/Architecture.md:126-130, AGENTS.md:65-68
Authentication and Multi-Tenancy
The system supports two primary modes:
- single_user: Simple API key authentication via
X-API-KEY. - multi_user: JWT-based authentication with PostgreSQL support for user management.
User data is logically isolated into per-user SQLite databases stored under<USER_DB_BASE_DIR>/<user_id>/.
Sources: Docs/Architecture.md:197-215, CLAUDE.md:68-75
Technical Stack Summary
| Component | Technology |
|---|---|
| Backend | FastAPI (Python) |
| Frontend | Next.js (TypeScript) |
| Databases | SQLite (default), PostgreSQL (optional) |
| Vector Store | ChromaDB, pgvector (optional) |
| Audio Processing | faster_whisper, NVIDIA NeMo, Kokoro ONNX |
| Document Processing | Docling, PyMuPDF, Pandoc |
| Sources: AGENTS.md:84-93, CLAUDE.md:88-95 |
Conclusion
tldw_server represents a bridge between modern AI research tools and the visionary concept of a personalized digital primer. By focusing on a FastAPI-first architecture and modular core services, the project enables developers and researchers to build complex, self-hosted media analysis workflows while ensuring data sovereignty and extensibility through protocols like MCP and OpenAI-compatible APIs.
Sources: AGENTS.md:11-19, Docs/Architecture.md:235-245