Vector Search (Qdrant)
Relevant source files
The following files were used as context for generating this wiki page:
Vector search in Archivum is powered by Qdrant, a high-performance vector database that enables semantic information retrieval. It serves as a derived index, meaning all data within Qdrant is generated from the canonical Markdown files and can be fully rebuilt if necessary. The system uses vector embeddings to allow users to search the wiki using natural language rather than exact keyword matching.
The implementation relies on an asynchronous Qdrant client to manage collections, upsert document chunks, and perform similarity searches. It supports multiple embedding providers, including local processing via fastembed and external OpenAI-compatible APIs (e.g., OpenRouter, Ollama).
Sources: archivum-prd-v1.0.md:278-283, apps/backend/archivum/db/qdrant_client.py:1-20, README.md:237-245
Architecture and Data Flow
The vector search architecture follows a pipeline where text content is processed into high-dimensional vectors and stored in Qdrant points.
Ingestion and Upsert Flow
When a page is created or updated, the system triggers an upsert to Qdrant. The content is first split into overlapping chunks to ensure context is preserved across vector boundaries. Each chunk is then converted into an embedding vector using the configured provider.
The diagram shows the transformation of raw Markdown into vectorized points stored in the Qdrant collection.
Sources: apps/backend/archivum/db/qdrant_client.py:180-200, apps/backend/archivum/db/qdrant_client.py:273-315
Search and Query Flow
Search requests involve embedding the user's natural language query and performing a cosine similarity search against the Qdrant collection.
The sequence diagram illustrates the retrieval process from a user query to the final list of semantic hits.
Sources: apps/backend/archivum/db/qdrant_client.py:326-360, apps/backend/archivum/mcp/server.py:65-71
Key Components and Logic
Text Chunking
To handle large documents, Archivum approximates tokens as whitespace-separated words. The default _chunk_text function uses a chunk_size of 512 words and an overlap of 64 words. This overlap ensures that semantic meaning isn't lost at the boundaries of chunks.
Sources: apps/backend/archivum/db/qdrant_client.py:180-198
Embedding Resolution
Archivum dynamically resolves the embedding vector dimension based on the provider and model. If EMBED_DIM is not explicitly set in the configuration, the system probes the provider with a "dimension probe" vector and caches the result.
Sources: apps/backend/archivum/db/qdrant_client.py:80-112
Point Metadata
Every point stored in Qdrant includes a payload that allows the system to link vector hits back to the original wiki pages:
slug: The unique identifier for the wiki page.title: The display title of the page.chunk_index: The sequence number of the chunk.wiki_id: The ID for multi-tenant isolation.text: The raw text content of the chunk (used for excerpts).
Sources: apps/backend/archivum/db/qdrant_client.py:301-310
Configuration and Integration
Embedding Providers
Archivum supports various backends for generating vectors. The selection is controlled by the EMBED_PROVIDER environment variable.
| Provider | Description | Required Settings |
|---|---|---|
local | Uses fastembed for CPU-based local embeddings. | EMBED_MODEL |
openai_compat | Connects to any OpenAI-compatible embedding API. | EMBED_BASE_URL, EMBED_API_KEY |
openrouter | Uses OpenRouter's unified API. | OPENROUTER_API_KEY |
ollama | Uses a local Ollama instance. | OLLAMA_BASE_URL |
Sources: apps/backend/archivum/db/qdrant_client.py:38-60, README.md:144-159
Qdrant Client Functions
Relevant implementation functions in apps/backend/archivum/db/qdrant_client.py:
upsert_page(...): Deletes stale vectors, chunks the new content, generates embeddings, and inserts new points. (Lines 273-315)search(...): Performs a semantic search and deduplicates hits so that multiple chunks from the same page only appear once in the results. (Lines 326-360)search_raw(...): Returns all individual chunk hits, primarily used for building context in the AIquerytool. (Lines 363-388)init_collection(...): Ensures the Qdrant collection exists and handles recreation if there is a dimension mismatch between the model and the DB. (Lines 216-270)
MCP Tool Integration
The search_wiki and query tools in the MCP server directly consume the Qdrant client to provide AI agents with semantic access to the knowledge base.
Sources: apps/backend/archivum/mcp/server.py:65-71, apps/backend/archivum/mcp/server.py:175-190
Index Maintenance
Because Qdrant is a derived index, it can become out of sync if the underlying Markdown files are modified externally. The system provides a maintenance endpoint to force a full rebuild:
- Endpoint:
POST /api/rebuild-indexes - Action: Iterates through all Markdown pages in the
wiki_datavolume and re-runs theupsert_pagelogic for each.
Sources: README.md:241-245, packages/archivum-cli/src/api.js:33
Vector search is critical for the "Compound Knowledge" goal of Archivum, ensuring that as the wiki grows, relevant information remains retrievable through natural language interaction.