Document Indexing Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Document Indexing Pipeline is the core system responsible for transforming raw data from various external connectors into searchable, indexed information within the Onyx platform. It bridges the gap between external data sources (such as Google Drive, Slack, Confluence, or GitHub) and the internal vector database (Vespa), ensuring that documents are accurately processed, metadata is preserved, and content is ready for Retrieval-Augmented Generation (RAG).
The pipeline operates primarily within specialized background workers. It involves fetching content, upserting it to a relational database (PostgreSQL), breaking content down into manageable chunks, generating semantic embeddings, and finally writing those chunks to a vector store.
Sources: AGENTS.md:28-40, CLAUDE.md:28-40, web/tests/e2e/chat/chat_message_rendering.spec.ts:71-80
Pipeline Architecture and Workflow
The indexing process is decoupled into multiple stages to handle concurrency and scale. It relies on a combination of primary coordinators and specialized workers to manage the data flow from source to index.
High-Level Data Flow
The flow begins with a connector fetching data and ends with chunks stored in Vespa.
The flow represents the progression of a document from its origin to the final vector store.
Sources: AGENTS.md:32-40, web/tests/e2e/chat/chat_message_rendering.spec.ts:114-124
Core Workers
The pipeline is distributed across several worker types managed by Celery:
| Worker Type | Responsibilities |
|---|---|
| Docfetching | Connects to external APIs, retrieves raw documents, and spawns processing tasks. |
| Docprocessing | Performs the heavy lifting: upserting to DB, chunking, embedding, and writing to Vespa. |
| Primary Worker | Coordinates high-level management and Vespa synchronization. |
| KG Processing | Handles Knowledge Graph building and relationship extraction between documents. |
Sources: AGENTS.md:28-56, CLAUDE.md:28-56
Detailed Processing Stages
1. Document Fetching and Storage
Documents are retrieved from external sources via various connector types (Load, Poll, or Slim). Once fetched, the Docprocessing Worker performs an initial upsert to PostgreSQL to maintain the document's state and metadata.
- Load Connector: Bulk indexes documents to reflect a specific point in time.
- Poll Connector: Incrementally updates the index based on time ranges.
- Slim Connector: A lightweight check of document IDs for pruning purposes.
Sources: backend/onyx/connectors/README.md:10-23, AGENTS.md:35-40
2. Chunking and Contextualization
To enable efficient semantic search, large documents are broken into smaller "chunks." This stage does more than just splitting text; it adds contextual information to each chunk to ensure that semantic meaning is preserved even when the chunk is viewed in isolation.
This diagram shows the interaction during the chunking phase where raw text is transformed into context-aware segments.
Sources: AGENTS.md:37-40, web/tests/e2e/chat/chat_message_rendering.spec.ts:115-116
3. Embedding Generation
Once chunks are created, they are sent to a Model Server. This server uses NLP models to generate vector representations (embeddings) of the text. These embeddings represent the semantic meaning of the chunk in a high-dimensional space.
- Technology: Python 3.11, FastAPI (Model Server), LangChain, and LiteLLM.
- Port: The model server typically runs on port 9000 during development.
Sources: CONTRIBUTING.md:154-157, AGENTS.md:112-118
4. Vector Database Indexing
The final stage of the pipeline involves writing the embedded chunks and their associated metadata to Vespa, the project's vector database. Vespa handles both full-text search and semantic vector search, allowing the AI chat system to retrieve relevant documents using a Retrieval-Augmented Generation (RAG) pipeline.
Sources: AGENTS.md:38-40, web/tests/e2e/chat/chat_message_rendering.spec.ts:117-124
System Monitoring and Health
The pipeline includes built-in mechanisms to ensure reliability and performance:
- Watchdog Monitoring: The
Docfetching Workerimplements watchdog monitoring to identify and handle stuck connectors. - Heartbeat/Liveness: Background jobs have automated heartbeat checks to ensure workers remain active.
- Task Prioritization: Celery queues are categorized into High, Medium, and Low priorities to manage resource-intensive operations like pruning.
Sources: AGENTS.md:33-65, CLAUDE.md:33-65
Conclusion
The Document Indexing Pipeline is a sophisticated, multi-stage system that transforms unstructured data into a structured semantic index. By utilizing a distributed worker architecture and context-aware chunking, it enables the Onyx platform to provide accurate, grounded AI responses through its RAG-based chat interface.
Sources: web/tests/e2e/chat/chat_message_rendering.spec.ts:114-124, AGENTS.md:28-40