Background Indexing Jobs
Relevant source files
The following files were used as context for generating this wiki page:
Background Indexing Jobs represent the asynchronous operations responsible for processing source code repositories into searchable vector embeddings within the MCB system. These jobs handle the full pipeline of file discovery, content chunking, and metadata extraction, allowing the system to maintain up-to-date semantic context without blocking the primary MCP (Model Context Protocol) server interface.
The indexing system is designed as a state-aware workflow where operations move from initialization through active processing to completion or failure. This lifecycle is visible through both the MCP index_status tool and the internal web-based admin dashboard.
Sources: README.md, crates/mcb-server/src/controllers/web/jobs.rs:1-15, crates/mcb-domain/tests/integration/semantic_search_workflow.rs:18-35
Job Lifecycle and State Management
Indexing operations are tracked using a set of defined statuses that represent the current progress of a background job. Each job is assigned a unique OperationId to allow for tracking and reporting across different system layers.
Indexing Operation Statuses
| Status | Description |
|---|---|
Starting | The job has been initialized and is preparing to scan the target path. |
InProgress | The system is currently reading files, creating chunks, or generating embeddings. |
Completed | The indexing process finished successfully for all discovered files. |
Error | The job failed due to an unrecoverable issue (e.g., path not found, storage quota). |
Sources: crates/mcb-server/src/controllers/web/jobs.rs:72-85, crates/mcb-domain/src/ports/mod.rs
Lifecycle Flow
The following diagram illustrates the transition of an indexing job from initiation via an MCP command to its final state in the system storage.
The indexing workflow involves multiple phases: parsing, chunking, and embedding generation.
Sources: crates/mcb-domain/tests/integration/semantic_search_workflow.rs:18-50, crates/mcb-server/src/controllers/web/jobs.rs:72-80
Job Execution Components
The execution of a background indexing job involves the coordination of several infrastructure and domain components.
1. Request Handling
Jobs are typically triggered through the IndexHandler, which processes IndexArgs. These arguments allow for fine-grained control over the job, including directory exclusions and file size limits.
- Action types:
Start,Status,Clear. - Filters:
extensions,exclude_dirs,ignore_patterns.
Sources: crates/mcb-server/tests/e2e/golden_tools.rs:25-50, crates/mcb-server/src/args/mod.rs
2. Processing Pipeline
The core logic follows a structured workflow to transform raw files into a searchable codebase snapshot.
The system utilizes Tree-sitter parsers for AST-aware chunking across 14 supported languages.
Sources: crates/mcb-domain/tests/integration/semantic_search_workflow.rs:18-55, README.md, crates/mcb-server/src/formatter/mod.rs
Monitoring and Reporting
MCB provides multiple interfaces for monitoring the progress and outcome of background indexing jobs.
Admin Dashboard
The web-based jobs controller renders a real-time view of active and historical operations. It aggregates metrics such as the total count of running jobs and completion statistics.
- Metric Cards: Display "Status" (Idle/Running), "Total Jobs", and "Complete".
- Operations Table: Lists ID, Type (Indexing/Validation), Status, and File count.
Sources: crates/mcb-server/src/controllers/web/jobs.rs:20-45
Indexing Result Schema
When a job completes (or fails), an IndexingResult is generated containing the following metrics:
| Field | Type | Description |
|---|---|---|
files_processed | usize | Total number of files successfully read. |
chunks_created | usize | Total granular code blocks generated. |
files_skipped | usize | Files ignored due to size or pattern filters. |
errors | Vec<String> | List of specific file-level processing errors. |
operation_id | Option<Id> | The unique identifier for the job. |
Sources: crates/mcb-server/tests/unit/formatter/formatter_tests.rs:15-30, crates/mcb-server/tests/unit/mcp_server/mcp_error_handling_tests.rs:25-35
Error Handling and Troubleshooting
Background jobs are designed to "fail loud" if the source of truth is absent or if a critical error occurs.
- Validation: Jobs validate the target path before starting. A nonexistent path results in immediate rejection.
- Troubleshooting Guidance: Error responses formatted for MCP clients include actionable advice, such as verifying the directory or checking supported languages (e.g., Rust, Python).
- Performance Warnings: The system monitors processing time; slow search or indexing operations may trigger performance warnings in the response metadata.
Sources: AGENTS.md:10-15, crates/mcb-server/tests/unit/mcp_server/mcp_error_handling_tests.rs:40-55, crates/mcb-server/tests/unit/mcp_server/mcp_error_handling_tests.rs:115-135
Summary
Background Indexing Jobs are the backbone of MCB's semantic capabilities. By offloading the resource-intensive tasks of AST parsing and embedding generation to asynchronous operations, MCB ensures that the MCP server remains responsive. These jobs are fully observable through specialized status tools and a dedicated web dashboard, providing developers with detailed statistics on codebase coverage and processing health.