Text Embeddings Integration
Relevant source files
The following files were used as context for generating this wiki page:
Text embeddings are a core component of the Archivum system, enabling semantic search and context-aware querying across the knowledge base. By converting markdown wiki pages into high-dimensional vector representations, the system can perform similarity searches that go beyond keyword matching, allowing users and AI agents to retrieve relevant information based on conceptual meaning.
The integration facilitates a "knowledge compounds" model where every ingested document is vectorized and stored in a specialized vector database. This derived index is fully rebuildable from the canonical markdown files, ensuring data resilience while providing high-performance retrieval for the Query Engine and MCP Tools.
Sources: archivum-prd-v1.0.md:14-25, README.md:12-20, Progress.md:25-30
Architecture and Data Flow
The text embedding system is architected around a pipeline that transforms raw text into searchable vectors. The primary storage for these vectors is Qdrant, which serves as the vector database for the entire stack.
Ingestion Pipeline Integration
During the ingestion process, once a document is parsed and processed into wiki pages by the AI agent, the content is passed to the embedding module. The pipeline coordinates the conversion of page content into embeddings and their subsequent storage in Qdrant.
The diagram shows the sequential flow of data from ingestion to various storage backends, highlighting the role of the embeddings module in populating Qdrant.
Sources: apps/backend/archivum/ingest/pipeline.py:157-166, README.md:280-295
Core Components
| Component | Responsibility |
|---|---|
| Qdrant | High-performance vector database storing embeddings for semantic search. |
| Ingest Pipeline | Orchestrates the extraction of text and triggers the qdrant.upsert_page function. |
| Embedding Provider | The service or library responsible for generating the vectors (e.g., local FastEmbed, OpenAI, or Ollama). |
| Search Engine | Executes similarity queries against Qdrant to find relevant context for user questions. |
Sources: README.md:175-185, apps/backend/archivum/ingest/pipeline.py:11-20
Embedding Generation and Storage
The system supports multiple embedding providers, ranging from fully local execution to cloud-based APIs. The default configuration utilizes BAAI/bge-small-en-v1.5 for local embeddings, ensuring that the knowledge base remains functional without external dependencies if desired.
Vector Upsert Logic
When a page is persisted, the upsert_page method in the Qdrant client performs the following:
- Chunking: The markdown content is divided into manageable segments (chunks) to improve retrieval granularity.
- Vectorization: Each chunk is converted into a vector using the configured
EMBED_MODEL. - Indexing: Vectors are stored in Qdrant with associated metadata, including the page slug and title, allowing the system to map search hits back to specific wiki pages.
# From apps/backend/archivum/ingest/pipeline.py:157-166
# Upsert into Qdrant
with span("qdrant.upsert_page", slug=final_slug) as sp_q:
chunk_count = await qdrant.upsert_page(
slug=final_slug,
title=page.title,
content=page.content,
wiki_id=wiki_id,
settings=s,
)
sp_q["chunks"] = chunk_count
Sources: apps/backend/archivum/ingest/pipeline.py:157-166, README.md:175-180
Retrieval and Semantic Search
The embeddings integration is primarily consumed by search and query interfaces. This allows the system to find relevant information even when the user's phrasing does not exactly match the stored text.
Search Flow
When a semantic search is initiated (via the Web UI or MCP), the system vectorizes the input query and performs a nearest-neighbor search in Qdrant.
Query Synthesis (RAG)
The query tool uses embeddings to perform Retrieval-Augmented Generation. It retrieves the top semantic hits from Qdrant and injects them as context into the LLM prompt to generate an answer with citations.
| Feature | Implementation |
|---|---|
| Top-K Retrieval | Default top_k=5 for standard search; limit=6 for query context. |
| Citation Mapping | Maps Qdrant results back to SQLite page records to provide accurate source links. |
| Fallback | If vector results are sparse, the system can fall back to keyword search (SQLite FTS). |
Sources: apps/backend/archivum/mcp/server.py:207-220, archivum-prd-v1.0.md:275-285
Configuration
Embeddings are configured via environment variables in the .env file. These settings determine the provider, model, and connectivity details.
| Variable | Default | Description |
|---|---|---|
EMBED_PROVIDER | local | Options: local (fastembed), openai_compat, openrouter, ollama. |
EMBED_MODEL | BAAI/bge-small-en-v1.5 | The specific model used for vectorization. |
QDRANT_URL | http://localhost:6333 | Connection string for the Qdrant vector database service. |
EMBED_API_KEY | — | Required if using a cloud-based provider. |
Sources: README.md:175-182, archivum-prd-v1.0.md:415-425
Summary
The text embeddings integration provides the semantic foundation for Archivum’s AI capabilities. By delegating vector storage to Qdrant and supporting both local and cloud-based embedding providers, the system balances privacy with performance. This integration enables the "self-organizing" nature of the wiki, as it allows the AI agent and the query engine to navigate and synthesize knowledge based on conceptual relationships stored within the vector space.
Sources: README.md:12-25, archivum-prd-v1.0.md:14-20