High-Level Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The tldw_server (Too Long; Didn't Watch Server) is a research assistant and media analysis platform designed to ingest, transcribe, analyze, and interact with various media formats. It follows a decoupled architecture consisting of a FastAPI backend and a Next.js frontend, utilizing both local and commercial AI providers for processing tasks.
Sources: Docs/Architecture.md:7-14, AGENTS.md:7-12
The system is structured as a "thin endpoint, fat core" architecture. FastAPI provides the interface layer, while the core business logic is encapsulated in domain-specific modules located within the app/core/ directory. This design ensures that logic is centralized and endpoints remain focused on request validation and response shaping.
Sources: Docs/Architecture.md:27-30, Docs/Architecture.md:144-149
1. Runtime Components
The system architecture consists of four primary tiers: Clients, the FastAPI application layer, Core Domain Modules, and the Storage layer.
1.1 Architectural Overview
The following diagram illustrates the high-level flow from client requests through the API layer to core services and persistent storage.
The flow begins with clients calling REST or WebSocket APIs under /api/v1, which are then processed by core services that interact with various storage backends.
Sources: Docs/Architecture.md:27-30, Docs/Architecture.md:46-77
1.2 Component Responsibilities
| Component | Responsibility | Relevant Path |
|---|---|---|
| FastAPI App | Entry point, routing, and shared dependencies (Auth, DB sessions). | tldw_Server_API/app/main.py |
| Core Modules | Domain logic for RAG, Ingestion, Audio, and LLM calls. | tldw_Server_API/app/core/ |
| Next.js WebUI | Primary interactive client for media management and chat. | apps/tldw-frontend/ |
| External Providers | Integration with OpenAI, Anthropic, Ollama, and local engines. | app/core/LLM_Calls/ |
Sources: Docs/Architecture.md:49-77, AGENTS.md:104-128
2. Core Backend Modules
The backend is organized into specialized domain modules that handle the heavy lifting of the application.
2.1 Media Ingestion & RAG Pipeline
The primary workflow involves processing raw media into searchable chunks.
- Ingestion: Downloads or extracts content from videos, PDFs, or web pages.
- Chunking: Splits content into manageable segments based on templates.
- Embeddings: Generates vectors using local or commercial models.
- Retrieval: Combines Full-Text Search (SQLite FTS5) with vector search (ChromaDB).
Sources: Docs/Architecture.md:100-128, AGENTS.md:47-65
2.2 LLM and Chat Orchestration
The Chat module manages conversation history and character cards, delegating the actual model interaction to the LLM_Calls provider abstraction layer. This layer supports over 16 backend providers, including commercial APIs and local services like Ollama and vLLM.
Sources: Docs/Architecture.md:121-128, AGENTS.md:79-84
2.3 Audio Processing
Audio features include Speech-to-Text (STT) via faster_whisper or NVIDIA NeMo, and Text-to-Speech (TTS) using OpenAI-compatible or local Kokoro engines. The system supports real-time streaming transcription over WebSockets.
Sources: Docs/Architecture.md:129-134, AGENTS.md:52-58
3. Data Flow: Media Processing
The following sequence diagram represents the lifecycle of a media ingestion request.
This multi-stage process ensures that data is normalized and indexed for both relational and semantic retrieval.
Sources: Docs/Architecture.md:155-171, AGENTS.md:106-108
4. Storage and Multi-Tenancy
The project supports both single_user (API Key) and multi_user (JWT) authentication modes.
4.1 Database Architecture
Storage is partitioned to support logical isolation between users.
- AuthNZ DB: Centralized database (
users.db) for user credentials. - Per-User Databases: Each user has a dedicated directory under
<USER_DB_BASE_DIR>/<user_id>/containing:Media_DB_v2.db: Content, chunks, and metadata.ChaChaNotes.db: Chat history and notes.chroma_storage/: Vector indices.
Sources: Docs/Architecture.md:195-225, CLAUDE.md:76-88
4.2 Storage Locations
| Database Type | Default Location | Purpose |
|---|---|---|
| AuthNZ | Databases/users.db | User accounts and permissions. |
| Media | user_databases/<user_id>/Media_DB_v2.db | Media items and FTS5 indices. |
| Notes | user_databases/<user_id>/ChaChaNotes.db | User notes and character chat sessions. |
| Vector | user_databases/<user_id>/chroma_storage/ | ChromaDB vector storage. |
Sources: Docs/Architecture.md:200-220, AGENTS.md:131-141
5. Frontend and Client Integration
The Next.js WebUI serves as the primary interface, communicating with the FastAPI backend via standardized Pydantic-validated REST and WebSocket APIs.
5.1 Communication Patterns
- REST APIs: Used for standard CRUD operations and state updates.
- WebSockets: Utilized for real-time STT streaming and MCP tool execution.
- Server-Sent Events (SSE): Primarily used for streaming LLM chat completions.
Sources: Docs/Architecture.md:244-249, apps/FRONTEND_AUDIT.md:52-60
Conclusion
The high-level architecture of tldw_server prioritizes modularity and provider flexibility. By abstracting storage access through DB_Management and external service interactions through specialized core modules, the system maintains scalability across different deployment modes (local vs. hosted) and authentication environments.