Document Ingestion & Processing
Relevant source files
The following files were used as context for generating this wiki page:
The Document Ingestion & Processing system is a core component of the KnowledgeVault (RAGAPPv3) platform. It is responsible for transforming raw technical documents into a structured, searchable format that can be utilized by Retrieval-Augmented Generation (RAG) pipelines. The system supports multi-format ingestion, including PDF, DOCX, XLSX, PPTX, CSV, SQL, and various code files, using semantic chunking to preserve logical structures like tables and code blocks.
Sources: README.md:14-25, backend/app/config.py:340-360
Ingestion is triggered through multiple channels: manual web uploads, directory monitoring via a file watcher, or automated email ingestion through IMAP polling. Once triggered, documents undergo a multi-phase pipeline—parsing, text extraction, chunking, and embedding—before being indexed into both a LanceDB vector store for semantic search and an SQLite FTS5 index for full-text search.
Sources: README.md:158-164, specs/kms-content-search-rag/SPEC.md:46-55
Ingestion Architecture & Data Flow
The ingestion process follows an asynchronous pipeline managed by background workers. Documents move through specific states in the files table, beginning with a pending status and progressing through processing until they are indexed or encounter an error.
Sources: backend/app/models/database.py:34-45, backend/app/config.py:100-110
The diagram above illustrates how documents from various sources are funneled into a centralized background processor that manages the transformation into vector and text indices.
Sources: README.md:75-95, specs/kms-content-search-rag/SPEC.md:46-60
Pipeline Components
Document Parsing & Extraction
The system utilizes Unstructured.io for high-quality parsing of complex document types. It supports multiple strategies including fast, hi_res, and auto. During this phase, the system extracts parsed_text, which is then stored in the files table to enable downstream full-text search and knowledge compilation.
Sources: backend/app/config.py:76-80, backend/app/models/database.py:645-660
Semantic Chunking
Extracted text is split into overlapping segments based on character count rather than tokens to ensure precision. The default configuration uses a chunk size of 2000 characters with a 200-character overlap. This process is "structure-aware," attempting to keep code blocks and tables intact within a single chunk to maintain coherence for the LLM.
Sources: Plan.md:40-55, backend/app/config.py:72-76
Vector Embedding
Each chunk is transformed into a high-dimensional vector. The system is optimized for the microsoft/harrier-oss-v1-0.6b model, producing 1024-dimensional dense embeddings. To improve retrieval relevance, the system supports instruction-aware prefixes that are prepended to the text before embedding.
Sources: README.md:255-265, backend/app/config.py:86-90, Plan.md:140-155
| Configuration Option | Type | Default | Description |
|---|---|---|---|
chunk_size_chars | int | 2000 | Character-based chunk size (~500 tokens) |
chunk_overlap_chars | int | 200 | Overlap between adjacent chunks |
embedding_model | str | harrier-oss-v1-0.6b | Primary model for vector generation |
vector_metric | str | cosine | Distance metric for similarity search |
embedding_batch_size | int | 64 | Number of texts sent per API request |
| Sources: backend/app/config.py:72-98, Plan.md:255-260 |
Database Schema & Indexing
The files table serves as the primary registry for ingested documents. It tracks metadata, processing progress, and state.
Sources: backend/app/models/database.py:27-65
This diagram shows the relationships between uploaded files and the broader knowledge management ecosystem, including vaults and wiki pages.
Sources: backend/app/models/database.py:30-70, backend/app/models/database.py:320-335
Full-Text Search (FTS)
The system maintains two distinct SQLite FTS5 indices to support different search use cases:
- Metadata Search (
files_search_fts): Indexes filenames, types, status, and email-specific fields for fast document listing. - Content Search (
files_content_fts): Indexes theparsed_textof every document, allowing users to search the body text of files independently of the RAG chat pipeline.
Sources: specs/kms-content-search-rag/SPEC.md:46-60, backend/app/models/database.py:650-675
Knowledge Compilation Post-Ingestion
Upon successful indexing, the ingestion pipeline can trigger secondary "Knowledge Compilation" jobs. These jobs extract structured entities, claims, and relations from the newly indexed chunks to populate the system's Wiki and Knowledge Management System (KMS).
Sources: backend/app/api/routes/wiki.py:465-480, backend/app/services/wiki_store.py:640-655
Compilation Triggers
wiki_compile_on_ingest: Automatically enqueues a wiki job when a document moves to theindexedstate.kms_compile_on_ingest: Creates or refreshes a KMS document entry immediately after indexing.- Manual Recompile: Users can manually trigger a recompilation for a specific document if the underlying knowledge extraction fails or needs updating.
Sources: backend/app/config.py:345-365, specs/kms-content-search-rag/SPEC.md:70-80
Conclusion
The document ingestion and processing system provides a robust foundation for RAG by combining high-fidelity parsing with sophisticated semantic chunking and dual-mode indexing. By automating the pipeline from raw file upload to structured knowledge extraction, the system ensures that technical documentation is not only stored but actively transformed into an accessible and context-aware knowledge base.
Sources: README.md:14-40, Plan.md:270-285