Ingestion Pipeline & Parsers
Relevant source files
The following files were used as context for generating this wiki page:
The Ingestion Pipeline is the core mechanism of Archivum responsible for transforming raw data—such as files, URLs, and code—into a structured, semantic knowledge base. It orchestrates the flow from raw source acquisition to text extraction, AI-driven entity and relationship parsing, and finally persistence into multiple storage layers including Markdown files, SQLite, Qdrant (vector search), and Kuzu (knowledge graph).
This system ensures that knowledge compounds over time by automatically building and maintaining a persistent wiki. It supports a wide range of formats, including documents, office files, data structures, and media, using a modular parser architecture and an LLM-powered Wiki Agent to synthesize content into interconnected wiki pages.
Sources: archivum-prd-v1.0.md:19-25, apps/backend/archivum/ingest/pipeline.py:1-15
Pipeline Architecture
The pipeline follows a strictly sequential, multi-stage process to ensure data integrity and avoid LLM rate limits during batch operations.
High-Level Data Flow
The following diagram illustrates the lifecycle of a source as it moves through the ingestion pipeline:
The pipeline begins by logging the ingest attempt in SQLite to track progress and state. Sources: apps/backend/archivum/ingest/pipeline.py:104-118
Pipeline Components
| Component | Responsibility |
|---|---|
| Pipeline Orchestrator | Coordinates the sequence of parsing, extraction, and persistence. |
| Parsers | Extract clean text and metadata from various file formats and URLs. |
| Wiki Agent | An LLM-driven agent that identifies topics, entities, and relationships. |
| Persistence Layer | Writes to Markdown, SQLite, Qdrant, and Kuzu Graph databases. |
Sources: apps/backend/archivum/ingest/pipeline.py:11-13, archivum-prd-v1.0.md:89-94
Source Parsers
Archivum utilizes a modular parsing system to handle diverse file types. The parse_source function automatically detects the correct parser based on file extensions or MIME types.
Supported Formats and Logic
| Category | Formats | Parsing Logic / Library |
|---|---|---|
| Documents | .md, .txt, .pdf, .epub | Native parsing for text; PyMuPDF for PDF; ebooklib for EPUB. |
| Web | http://, https://, .html | Playwright for JS-heavy pages; BeautifulSoup for HTML cleaning. |
| Office | .docx, .pptx, .xlsx | python-docx, python-pptx, openpyxl. |
| Data | .csv, .json, .jsonl | Pandas for schema summarization and sample extraction. |
| Code | .py, .js, .ts, etc. | AST-aware extraction of functions, classes, and docstrings. |
| Media | .png, .mp3, .mp4 | Claude Vision for images; Whisper for audio/video transcription. |
Sources: archivum-prd-v1.0.md:96-120, README.md:315-325
Extraction Data Structure
All parsers return a unified ParsedDoc structure:
@dataclass
class ParsedDoc:
text: str
source: str
metadata: dict[str, Any] = field(default_factory=dict)
Sources: apps/backend/archivum/ingest/parsers.py, tests/test_ingest_pipeline.py:16-24
The Wiki Agent
The WikiAgent uses LLMs (primarily claude-haiku) to transform extracted text into structured knowledge. It is configured via the WIKI_SYSTEM_PROMPT to act as an encyclopedic writer.
Extraction Logic
The agent performs three primary tasks:
- Wiki Page Synthesis: Creates comprehensive Markdown pages with YAML frontmatter.
- Entity Identification: Extracts named entities (people, concepts, orgs, tech, places).
- Relationship Mapping: Identifies how entities relate (e.g.,
related_to,mentions).
Sources: apps/backend/archivum/ingest/agent.py:24-52
LLM Providers
The system supports multiple extraction providers, allowing for local or cloud-based operation:
- Anthropic: Uses prompt caching on the system prompt for cost efficiency.
- OpenRouter: Alternative cloud provider for various models.
- OpenAI Compatible / Ollama: Supports local models for fully air-gapped ingestion.
Sources: apps/backend/archivum/ingest/agent.py:81-95, README.md:144-152
Fallback Mechanism
If the LLM extraction fails, the agent employs a _fallback_extraction method. This uses regex-based proper noun extraction to identify potential entities and generates a basic wiki page from the source snippet to ensure no data is lost. Sources: apps/backend/archivum/ingest/agent.py:265-288
Persistence and Indexing
Once knowledge is extracted, the pipeline synchronizes it across four distinct storage systems to enable different types of retrieval.
Storage Synchronization Sequence
Indexing Details
- Unique Slugs: The pipeline ensures slug uniqueness in SQLite by appending counters (e.g.,
slug-2) if a conflict exists. Sources: apps/backend/archivum/ingest/pipeline.py:87-97 - Wikilink Wiring: The pipeline extracts
[[wikilink]]targets from the generated content and createsREFERENCESedges in the Kuzu graph database. Sources: apps/backend/archivum/ingest/pipeline.py:228-245 - Mentions: It automatically scans page content for known entity names and creates
MENTIONSedges in the graph. Sources: apps/backend/archivum/ingest/pipeline.py:240-244
Batch Ingestion
Archivum supports batch ingestion for bulk-importing existing notes or research collections.
- Sequential Processing: To prevent LLM rate limit errors, files are processed one after another rather than in parallel.
- Progress Streaming: The
ingest_batchfunction accepts aProgressCallbackto provide real-time updates to the UI per file and per stage (parsing, extracting, persisting).
Sources: archivum-prd-v1.0.md:140-150, apps/backend/archivum/ingest/pipeline.py:284-301
Conclusion
The Ingestion Pipeline and Parsers form the foundational intelligence of Archivum. By combining robust multi-format parsing with LLM-driven synthesis and a multi-model storage approach, the system transforms static files into a dynamic, interconnected knowledge graph that matures automatically as new information is added. Sources: README.md:10-18, archivum-prd-v1.0.md:40-45