Incremental Indexing Pipeline

Relevant source files

The following files were used as context for generating this wiki page:

• packages/core/src/sync/synchronizer.ts
• packages/core/src/context.ts
• packages/core/README.md
• packages/mcp/README.md
• packages/mcp/src/handlers.ts

The Incremental Indexing Pipeline is a sophisticated system within Claude Context designed to efficiently synchronize a local codebase with its corresponding vector database representation. Instead of re-indexing an entire project after every modification, the pipeline identifies specific changes—additions, removals, and modifications—and updates only the affected segments. This approach significantly reduces computational overhead, API costs for embeddings, and synchronization time, especially for large-scale codebases.

Sources: packages/core/README.md, packages/core/src/context.ts:241-244

System Architecture

The pipeline relies on a combination of file-content hashing and Merkle Tree structures to detect state changes. The primary orchestration occurs in the Context class, which utilizes the FileSynchronizer to manage state snapshots and delta detection.

Core Components

Component	Responsibility	Source
FileSynchronizer	Manages SHA-256 file hashing, Merkle Tree construction, and snapshot persistence.	packages/core/src/sync/synchronizer.ts
MerkleDAG	Provides a structural representation of the codebase state for rapid comparison.	packages/core/src/sync/synchronizer.ts:182-200
Context	Coordinates the workflow between change detection and vector database updates.	packages/core/src/context.ts:241-314
Snapshot Manager	Persists codebase states to ~/.context/merkle/ for cross-session tracking.	packages/core/README.md, packages/core/src/sync/synchronizer.ts:25-31

High-Level Data Flow

The following diagram illustrates how the system transitions from detecting local file changes to updating the vector index.

Change Detection Logic

Change detection is performed by the FileSynchronizer using a multi-step verification process.

1. File Hashing and Metadata

Each file is hashed using the SHA-256 algorithm based on its content, not its metadata (like timestamps), ensuring that only actual code changes trigger a re-index. The system automatically ignores files matching patterns in .gitignore or hidden directories starting with a dot.

Sources: packages/core/src/sync/synchronizer.ts:33-40, packages/core/src/sync/synchronizer.ts:114-118

2. Merkle Tree Construction

File hashes are organized into a Merkle Directed Acyclic Graph (DAG). A root hash is generated representing the entire state of the directory.

• Initial Check: The system compares the current root hash against the one stored in the local snapshot (~/.context/merkle/[hash].json).
• Detailed Analysis: If the root hashes differ, the pipeline performs a file-by-file comparison of the fileHashes map.

Sources: packages/core/README.md, packages/core/src/sync/synchronizer.ts:182-200

3. Change Classification

The comparison results in three distinct categories of changes:

• Added: Paths present in the current scan but missing from the snapshot.
• Removed: Paths present in the snapshot but missing from the current scan.
• Modified: Paths present in both, but with differing SHA-256 hashes.

Sources: packages/core/src/sync/synchronizer.ts:236-258

Synchronization Workflow

Once changes are identified, the reindexByChange method in the Context class executes the update logic.

Deletion Phase

For every file categorized as removed or modified, the pipeline must purge existing vectors from the database. It constructs a query using the relativePath to find all associated chunk IDs and issues a delete command to the Milvus vector database.

// Example of the deletion logic for modified/removed files
const escapedPath = relativePath.replace(/\\/g, '\\\\');
const results = await this.vectorDatabase.query(
    collectionName,
    `relativePath == "${escapedPath}"`,
    ['id']
);
if (results.length > 0) {
    const ids = results.map(r => r.id as string).filter(id => id);
    await this.vectorDatabase.delete(collectionName, ids);
}

Sources: packages/core/src/context.ts:316-332

Indexing Phase

For added and modified files, the pipeline treats them as new content. The files are:

1. Read from the file system.
2. Split into chunks using the configured Splitter (AST-based or LangChain).
3. Converted into embeddings via the Embedding provider.
4. Inserted into the vector collection.

Sources: packages/core/src/context.ts:297-308

Configuration and Environment

The incremental indexing behavior can be customized through the ContextConfig and environment variables.

Variable / Option	Purpose	Source
forceReindex	Overrides incremental logic to recreate the entire collection.	packages/core/src/context.ts:133
ignorePatterns	Prevents specific files/folders from being hashed or tracked.	packages/core/src/context.ts:40-101
HYBRID_MODE	Determines if the pipeline processes dense vectors or hybrid (BM25 + dense) vectors.	packages/core/src/context.ts:119-125
MERKLE_DIR	The directory where state snapshots are persisted (~/.context/merkle).	packages/core/src/sync/synchronizer.ts:25-27

Sequence Diagram: Incremental Update

The following sequence shows the interaction between the Context, FileSynchronizer, and the VectorDatabase.

Conclusion

The Incremental Indexing Pipeline is central to the project's performance, ensuring that vector search results stay synchronized with the evolving codebase without requiring redundant full-project scans. By leveraging Merkle Trees and content-based hashing, it provides a robust mechanism for maintaining up-to-date semantic context for AI agents.

Sources: packages/core/README.md, packages/core/src/context.ts:310-314