Search & Retrieval (RAG)
Relevant source files
The following files were used as context for generating this wiki page:
Search & Retrieval (RAG) within tldw_server is a unified pipeline designed to provide contextual, high-accuracy information retrieval and generation from various media sources. The system combines traditional full-text search (FTS5) with vector embeddings to enable hybrid search, re-ranking, and answer assembly powered by Large Language Models (LLMs).
The primary goal of the RAG module is to provide a modular and generalized setup that is easy to decompose, exposing various "knobs and dials" to the user while providing verifiable citations for generated results. It serves as the core of the information retrieval capabilities for the research assistant platform.
Sources: Docs/Design/RAG_Plan.md:20-27, AGENTS.md:52-56, CLAUDE.md:52-56
Architecture and Components
The RAG system is structured as a multi-stage pipeline that processes data from ingestion through to final response generation. It leverages a combination of relational databases for metadata and keyword search, and vector stores for semantic retrieval.
High-Level Component Overview
| Component | Description |
|---|---|
| Unified Pipeline | The central orchestration layer and single entry point for all RAG features. |
| Database Retrievers | Multi-source retrieval handles interfaces with Media DB and vector stores. |
| Query Expansion | Enhances user queries using synonyms, acronyms, and entities. |
| Reranking | Applies Cross-encoders or semantic strategies to prioritize relevant results. |
| Semantic Cache | Stores previous search results to improve performance and reduce latency. |
| Generation | Integrates with LLMs to produce answers based on retrieved context. |
| Citations | Generates academic or inline citations for the retrieved information. |
Sources: tldw_Server_API/tests/RAG_NEW/TEST_STRATEGY.md:10-18, Docs/Architecture.md:214-230
Data Flow Diagram
The following diagram illustrates the flow of a user query through the RAG pipeline, from input to the final generated response with citations.
The diagram shows the sequence of operations starting with a cache check, moving through query expansion and hybrid retrieval (FTS5 + Vector), and concluding with LLM generation and citation enrichment.
Sources: tldw_Server_API/tests/RAG_NEW/TEST_STRATEGY.md:10-18, Docs/Design/RAG_Plan.md:43-55
Pipeline Stages
The RAG implementation follows a structured sequence of operations defined in the "RAG Plan" and implemented via the unified pipeline.
1. Pre-Query: Ingestion and Processing
Before retrieval can occur, data must be prepared. This involves:
- Parsing: Initial data extraction from various formats (PDF, Ebook, Video).
- Chunking: Splitting data according to strategy (Websites vs. PDFs). Chunks include summaries of the parent document and their relative position.
- Embedding: Converting text chunks into vector embeddings stored with metadata for semantic search.
Sources: Docs/Design/RAG_Plan.md:43-55, AGENTS.md:38-42
2. Retrieval and Search
The retrieval layer performs a hybrid search to find relevant information:
- SQLite FTS5: Used for full-text keyword searches within the media database.
- Vector Search: Uses ChromaDB or pgvector to find semantically similar chunks based on embeddings.
- BM25 + Vector + Re-ranking: A combined pipeline that merges keyword and semantic results to increase accuracy.
Sources: AGENTS.md:52-56, CLAUDE.md:52-56, Docs/Architecture.md:214-230
3. Post-Processing and Generation
Once chunks are retrieved, the system performs:
- Fusion and Ranking: Merging results from different sources and applying score fusion.
- Selection: Identifying the most relevant chunks for the prompt context.
- Answer Assembly: Submitting the prompt to the LLM (using providers like OpenAI, Anthropic, or local backends).
- Citation Generation: Providing citations for generated results to ensure technical accuracy and verifiability.
Sources: Docs/Design/RAG_Plan.md:56-62, tldw_Server_API/tests/RAG_NEW/TEST_STRATEGY.md:10-18
Technical Implementation Details
The RAG module is primarily located in tldw_Server_API/app/core/RAG/. Key implementation files include:
unified_pipeline.py: Serves as the central orchestration layer, managing parameters for query expansion, caching, and reranking.database_retrievers.py: Manages retrieval from multiple data sources.profiles.py: Handles RAG profiles and request-mapping boundaries.semantic_cache.py: Implements the semantic similarity matching for performance optimization.
Sources: Docs/superpowers/specs/2026-04-07-rag-module-review-design.md:123-145, tldw_Server_API/tests/RAG_NEW/TEST_STRATEGY.md:10-18
API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/v1/rag/search | POST | Unified RAG search endpoint for retrieval and generation. |
/api/v1/rag/health | GET | Status and health monitoring for RAG services. |
Sources: AGENTS.md:330-335, CLAUDE.md:330-335, Docs/superpowers/specs/2026-04-07-rag-module-review-design.md:14-16
Testing and Quality Assurance
The RAG module utilizes a three-tier testing strategy consisting of unit, integration, and property-based tests.
Testing Tiers
- Unit Tests: Test individual components in isolation (e.g.,
test_retrieval.py), mocking only external services like LLMs. - Integration Tests: Verify end-to-end workflows using real MediaDatabase and cache components (e.g.,
test_rag_integration.py). - Property-Based Tests: Use Hypothesis to verify invariants, such as original query preservation and document uniqueness.
Sources: tldw_Server_API/tests/RAG_NEW/TEST_STRATEGY.md:21-42
Performance Metrics
The system monitors several metrics to ensure efficiency:
- Execution Time: Goal for standard pipeline with cache hit is < 500ms.
- Latency: Retrieval and reranking latency.
- Memory Usage: Target is < 100MB growth over 50 requests.
Sources: tldw_Server_API/tests/RAG_NEW/TEST_STRATEGY.md:122-132
Summary
The Search & Retrieval (RAG) system is a critical component of tldw_server, providing the bridge between stored media and interactive LLM responses. By integrating FTS5, vector search, and a flexible unified pipeline, it enables researchers to query their personal library with high precision and verifiable citations. Its modular design allows for the independent configuration of query expansion, reranking, and caching to balance performance and quality.
Sources: Docs/Design/RAG_Plan.md:20-27, AGENTS.md:52-56