LanceDB Vector Store
Relevant source files
The following files were used as context for generating this wiki page:
The LanceDB Vector Store serves as the primary engine for semantic search and retrieval within the KnowledgeVault ecosystem. It stores embedded document chunks as high-dimensional vectors, enabling the RAG (Retrieval-Augmented Generation) pipeline to find relevant technical context based on user queries. Unlike traditional relational databases, LanceDB is optimized for Approximate Nearest Neighbor (ANN) search, which is critical for scaling to thousands of technical documents.
Sources: README.md:1-10, Plan.md:4-10
Architecture and Data Integration
LanceDB operates as an embedded vector database, co-existing with SQLite. While SQLite handles relational metadata and FTS (Full-Text Search), LanceDB manages the high-density vector data and chunk content. The system utilizes a "tri-vector" mode or hybrid search approach to combine semantic similarity with traditional keyword matching.
System Integration Flow
This diagram illustrates how the Vector Store interacts with the LLM and relational components during a RAG cycle.
The RAG Engine orchestrates the flow by first embedding the query and then searching LanceDB for the most similar document fragments.
Sources: README.md:35-55, Plan.md:145-155
Core Features and Implementation
Multi-Scale Indexing
To improve recall across various levels of technical detail, the system implements multi-scale indexing. Documents are indexed at multiple chunk sizes (e.g., 768 and 1536 characters) simultaneously. This allows the retriever to capture both granular details and broader conceptual context within a single search operation.
Sources: README.md:164-168, Plan.md:110-120
Retrieval Windowing and Expansion
To maintain coherence in technical documentation (such as preserving the integrity of code blocks or multi-step procedures), the store supports adjacent-chunk windowing. When a specific chunk is retrieved, the system can automatically include N adjacent chunks from the same document to provide necessary context to the LLM.
Sources: Plan.md:143-160
Distance Metrics and Relevance
The Vector Store primarily utilizes Cosine Similarity as the metric for vector search. Relevance is determined by the _distance field returned by LanceDB, which is then often converted to a similarity score for user-facing badges.
Sources: Plan.md:88-105, README.md:200-205
Configuration and Schema
The Vector Store behavior is highly configurable through environment variables and system settings, impacting how documents are segmented and retrieved.
Key Configuration Parameters
| Variable | Default Value | Description |
|---|---|---|
EMBEDDING_MODEL | harrier-oss-v1-0.6b | The model used to generate vectors (1024-dim). |
CHUNK_SIZE_CHARS | 2000 | Target size for document fragments in characters. |
CHUNK_OVERLAP_CHARS | 200 | Overlap between consecutive chunks to prevent context loss. |
RETRIEVAL_TOP_K | 12 | Number of chunks retrieved for the RAG context. |
MAX_DISTANCE_THRESHOLD | 0.5 | Filters out results with a distance greater than this value. |
Sources: README.md:140-160, Plan.md:215-225, frontend/src/lib/api.ts:250-275
Relational Metadata Sync
The system maintains a vector_delete_pending table in SQLite to track orphaned chunks. If a document is deleted but its LanceDB chunks cannot be immediately removed, they are queued for a background sweep to ensure data consistency.
Sources: backend/app/models/database.py:655-665, qa-report.md:45-50
Search Logic and Deduplication
The retrieval pipeline employs a group-aware deduplication policy to prevent a single document from dominating the context window.
Search and Deduplication Flow
The PER_DOC_CHUNK_CAP (default: 5) and UNIQUE_DOCS_IN_TOP_K (default: 5) settings ensure a diverse range of sources are provided to the assistant.
Sources: README.md:170-175, Plan.md:162-175
Maintenance and Migration
Re-indexing and Dimension Validation
LanceDB is sensitive to embedding dimension changes. If the EMBEDDING_MODEL is changed (e.g., from 768-dim BGE-M3 to 1024-dim Harrier), the vector store must be wiped and re-indexed. The system includes a migrate_embeddings.py script to handle this transition by resetting file statuses to pending in SQLite, triggering the background processor to re-embed all documents.
Sources: README.md:235-265, Plan.md:195-205
Index Rebuild Threshold
To maintain performance during heavy document churn, the system monitors the "delete churn fraction." Once the fraction of deleted records exceeds the INDEX_REBUILD_DELTA (default: 0.2), an ANN index rebuild is triggered to optimize search latency.
Sources: README.md:176-178
Conclusion
The LanceDB Vector Store is a critical infrastructure component that enables high-performance semantic retrieval for technical documentation. Through features like multi-scale indexing, adjacent-chunk windowing, and robust metadata synchronization with SQLite, it provides a coherent and relevant context for the KnowledgeVault RAG pipeline. Its reliance on character-based chunking and specific distance metrics ensures that the generated AI responses are grounded in accurate and diverse source evidence.