Background Task Processing
Relevant source files
The following files were used as context for generating this wiki page:
Background Task Processing in this project primarily centers around the asynchronous compilation and management of knowledge through the Wiki and Knowledge Management System (KMS) subsystems. These tasks handle long-running operations such as document indexing, knowledge extraction, and automated metadata generation without blocking the main application flow or user interactions.
The system utilizes a job-based architecture where tasks are enqueued via specific triggers—such as document ingestion or user queries—and tracked through various states from pending to completion. This ensures high availability and responsiveness for the chat and document management interfaces.
Sources: specs/wire-wiki-chat/SPEC.md, specs/kms-content-search-rag/SPEC.md
Job Lifecycle and Management
The background processing system tracks the lifecycle of asynchronous tasks using a specific set of statuses and triggers. Jobs are stored in the database and can be monitored, retried, or cancelled via the API and frontend panels.
Job Statuses
The system defines the following statuses for background jobs:
| Status | Description |
|---|---|
pending | The job is enqueued and waiting for an available worker. |
running | The job is currently being processed. |
completed | The job finished successfully. |
failed | The job encountered an error during execution. |
cancelled | The job was manually terminated by a user. |
Sources: frontend/src/lib/api.ts:1330, frontend/src/pages/WikiJobsPanel.tsx:20-26
Job Triggers
Jobs are initiated by various system events, categorized as triggers:
- Ingest: Triggered when a new document is uploaded and indexed.
- Query: Triggered post-answer during chat interactions to extract or compile knowledge.
- Memory: Triggered when promoting user memories to the wiki.
- Manual: Manually enqueued by a user via the UI.
- Settings Reindex: Triggered by changes to system settings that require re-processing of data.
Sources: frontend/src/lib/api.ts:1322, frontend/src/pages/WikiJobsPanel.tsx:28-34
Architecture and Data Flow
The background task architecture is integrated into the backend services and uses an event bus to communicate state changes to the frontend in real-time.
The diagram above illustrates the flow from a system trigger to the final state update broadcasted via SSE.
Sources: backend/app/api/routes/wiki.py:461-487, specs/wire-wiki-chat/SPEC.md
Event Streaming (SSE)
The backend provides a Server-Sent Events (SSE) stream at /wiki/events that emits terminal-state job events for a specific vault. This allows the frontend to update UI components like the WikiJobsPanel automatically when a job completes or fails without constant polling.
Sources: backend/app/api/routes/wiki.py:461-487, frontend/src/pages/WikiPage.tsx:64-70
Wiki Compilation Tasks
Wiki compilation is a critical background process that transforms raw document data or chat interactions into structured wiki claims and entities.
Key API Endpoints
POST /wiki/recompile: Enqueues asettings_reindexjob to re-derive all stale claims in a vault.POST /wiki/documents/{file_id}/status: Returns semantic wiki status and latest job details for a document.POST /wiki/jobs/{job_id}/retry: Re-queues a failed job.POST /wiki/jobs/{job_id}/cancel: Terminates a pending or running job.
Sources: backend/app/api/routes/wiki.py:539-650, frontend/src/lib/api.ts:1400-1430
Automated Extraction Flow
When a document finishes indexing, the system can automatically enqueue a wiki ingestion job if wiki_compile_on_ingest is enabled.
Knowledge Management (KMS) Integration
The KMS subsystem parallels the Wiki's background processing. It includes a KMSRetrievalService and tasks for compiling document content into KMS entries.
- KMS Compilation: Triggered by document ingestion (
KMS_COMPILE_ON_INGEST). - FTS Indexing: Content-level search is supported via a separate
files_content_ftstable, maintained through SQL triggers on thefilestable. - KMS Jobs: Monitored similarly to wiki jobs with fields for
trigger_type,status, andresult_json.
Sources: specs/kms-content-search-rag/SPEC.md, frontend/src/lib/api.ts:1532-1544
Summary
Background task processing provides the foundation for KnowledgeVault's asynchronous intelligence. By decoupling heavy document processing and knowledge extraction from the user-facing API, the system maintains high performance. The use of a centralized job store, database triggers for FTS, and an SSE-based event bus ensures that users receive real-time updates on the state of their knowledge base.