Vector Storage (ChromaDB / pgvector)
Relevant source files
The following files were used as context for generating this wiki page:
Vector Storage in the tldw_server project provides the infrastructure for storing, indexing, and querying high-dimensional embeddings generated from media content. This system is a critical component of the Retrieval-Augmented Generation (RAG) pipeline, enabling semantic search capabilities that go beyond traditional keyword matching.
The project supports a pluggable architecture for vector storage, primarily utilizing ChromaDB as the default local provider, while offering support for PostgreSQL with pgvector for more robust or multi-user deployments. These stores are used to manage embeddings for various media formats, including video transcripts, audio, PDFs, and web content, ensuring that semantically relevant context can be retrieved for chat and research tasks.
Sources: Docs/Architecture.md:46-51, AGENTS.md:54-58
System Architecture
The vector storage system operates as a per-user service within the tldw_server ecosystem. Each user’s embeddings and metadata are logically isolated to maintain privacy and data integrity.
Integration within Core Modules
Vector storage is integrated into the following data flow:
- Embeddings Module: Computes embeddings for chunks and writes vectors/metadata to the storage.
- RAG Module: Queries the vector store to fetch semantically similar context for LLM generation.
- Media DB v2: Maintains references to vector IDs to link relational metadata with vector indices.
Sources: Docs/Architecture.md:129-146, CLAUDE.md:112-114
Component Diagram
The following diagram illustrates the relationship between the FastAPI application, core modules, and the underlying vector databases.
The diagram shows the modular abstraction where both Embeddings and RAG services interact with vector stores through a factory/adapter pattern.
Storage Providers
The system provides flexibility by supporting two distinct vector storage backends, configurable via environment variables or configuration files.
1. ChromaDB (Default)
ChromaDB serves as the default vector database, typically used in local and single_user deployments.
- Storage Location: By default, it is stored per-user under
<USER_DB_BASE_DIR>/<user_id>/chroma_storage/. - Primary Use: Lightweight, file-based indexing for rapid local semantic retrieval.
2. pgvector (PostgreSQL)
For production environments or multi_user deployments, the system supports PostgreSQL with the pgvector extension.
- Configuration: Enabled via
config.txtor environment variables when high concurrency and centralized relational/vector data are required. - Integration: Works alongside the AuthNZ PostgreSQL instance if configured.
Sources: Docs/Architecture.md:189-192, AGENTS.md:104-106, CLAUDE.md:214-220
Comparison Table
| Feature | ChromaDB | pgvector (PostgreSQL) |
|---|---|---|
| Primary Deployment | Local / Single-user | Multi-user / Production |
| Storage Model | File-based (per-user dir) | Relational + Vector Extension |
| Default Config | SQLite-like local persistence | Centralized DB Instance |
| Key Advantage | Zero-config, easy setup | High concurrency, integrated metadata |
Sources: Docs/Architecture.md:189-192, AGENTS.md:100-111, CLAUDE.md:237-241
Data Flow: Ingestion and Retrieval
The vector store acts as the persistence layer for the semantic processing pipeline.
Vector Ingestion Sequence
When media is processed, the system follows a structured pipeline to populate the vector store:
This sequence ensures that every vector in the store can be traced back to its specific media chunk in the relational database.
Retrieval Pipeline
During a RAG request, the vector store is queried as part of a hybrid pipeline:
- Query Encoding: The incoming user query is converted into a vector.
- Semantic Search: The vector store performs a similarity search (e.g., Cosine Similarity).
- Hybrid Step: Results are often combined with SQLite FTS5 (Full-Text Search) scores.
- Re-ranking: Top results are re-ranked before being sent to the LLM.
Sources: Docs/Architecture.md:114-118, AGENTS.md:65-70
Configuration and Management
Vector storage settings are primarily handled in config.txt or .env.
Key Configuration Variables
| Variable | Description |
|---|---|
DATABASE_URL | Used for AuthNZ and potentially pgvector connection. |
USER_DB_BASE_DIR | Root directory for per-user ChromaDB storage. |
AUTH_MODE | Determines if storage is handled in single_user or multi_user context. |
Sources: Docs/Architecture.md:167-172, AGENTS.md:290-295, CLAUDE.md:323-328
Management via DB Abstractions
All vector store operations should be accessed through the abstractions in core/DB_Management/ or the vector store wrappers in core/RAG/. Raw access to ChromaDB or pgvector APIs is discouraged within endpoints to maintain modularity.
Sources: Docs/Architecture.md:194-196, CLAUDE.md:112-117
Conclusion
Vector storage in tldw_server provides a scalable and flexible semantic index. By supporting both local-first (ChromaDB) and production-grade (pgvector) backends, the system ensures that semantically indexed media is accessible across different deployment scales. This infrastructure directly enables the project's core research capabilities, allowing the "Primer" vision to be realized through high-accuracy contextual retrieval.
Sources: AGENTS.md:18-25, Docs/Architecture.md:46-51