Incremental File Indexing
Relevant source files
The following files were used as context for generating this wiki page:
Incremental File Indexing is a core performance optimization in chkpt that enables fast, efficient workspace snapshots. By utilizing a persistent index (stored as index.bin), the system can detect changes in the workspace without re-hashing every file. This mechanism significantly reduces I/O and CPU overhead during "warm" saves, where only a subset of files has changed.
The indexing system tracks file metadata such as modification timestamps (mtime), file size, and inodes to determine if a file's content needs to be re-evaluated. If the metadata matches the indexed state, chkpt reuses the existing content hash, skipping the expensive reading and hashing phases.
Sources: README.md:14-16, CLAUDE.md:7-11, crates/chkpt-core/src/ops/save.rs:114-135
Architecture and Data Structures
The indexing system revolves around the FileIndex and FileEntry structures. These components manage the lifecycle of file metadata from initial discovery during a workspace scan to persistence in the project's store.
The Index File Entry
The FileEntry represents the indexed state of a single file. It contains the necessary entropy to detect modifications without reading the file content.
| Field | Type | Description |
|---|---|---|
path | String | Relative path of the file from the workspace root. |
blob_hash | [u8; 16] | XXH3-128 content hash of the file. |
size | u64 | File size in bytes. |
mtime_secs | i64 | Last modification time (seconds). |
mtime_nanos | i64 | Last modification time (nanoseconds). |
inode | Option<u64> | Filesystem inode number (if available). |
mode | u32 | File permissions/mode. |
Sources: crates/chkpt-core/src/ops/save.rs:142-150, crates/chkpt-core/src/ops/restore.rs:420-428
Storage Layout
The index is stored within the project-specific directory inside the .chkpt home. The StoreLayout provides the path to index.bin, which is typically serialized using the bitcode format for high-speed I/O.
The diagram shows the location of the indexing file relative to other core storage components.
Sources: crates/chkpt-core/src/config.rs:49-51, crates/chkpt-core/src/ops/save.rs:360-365
Save Operation Flow
During a save operation, the index acts as a filter. The process compares ScannedFile objects (from a fresh filesystem walk) against the FileEntry objects loaded from index.bin.
Index Filtering Logic
- Load: The system opens
FileIndexfromindex.bin. - Scan: The workspace is scanned to produce a list of
ScannedFileobjects. - Compare: For each scanned file, the system checks if a cached entry exists.
- Decision:
- Match: If
mtime,size,inode, andmodeall match, the file is considered "clean." The cachedblob_hashis reused. - Mismatch/New: If any metadata differs or the file is new, it is marked for preparation (reading, hashing, and potential compression).
- Match: If
This flowchart illustrates how the index determines which files require expensive processing.
Sources: crates/chkpt-core/src/ops/save.rs:90-135, crates/chkpt-core/src/ops/save.rs:271-294
Restore and Synchronization
The index must remain synchronized with the actual state of the workspace. This is handled during both save and restore operations.
Index Updates during Save
At the end of a save operation, the system identifies "stale" paths (files present in the index but missing from the current scan) and "updated" paths. The index.apply_changes function is called to prune deleted files and persist the metadata of newly processed or changed files.
Sources: crates/chkpt-core/src/ops/save.rs:136-140, crates/chkpt-core/src/ops/save.rs:248-251
Index Reconstruction during Restore
When a snapshot is restored, the workspace changes to match a past state. To ensure the next save is incremental, the index is updated with the metadata of the restored files.
- Calculate Diff: The system compares the target snapshot state against the current workspace.
- Apply Files: Files are added, changed, or removed.
- Update Index: For every file that was written to disk during restore, the system fetches fresh metadata (like the new
mtimeandinode) and updates the index accordingly.
Performance Impact
The use of the SQLite-based catalog and the binary index.bin allows chkpt to achieve high performance even on large repositories like the Linux kernel.
| Project | Files | Cold Save | Incremental Save |
|---|---|---|---|
| React | 6,879 | 0.26s | 0.05s |
| Rust | 58,760 | 2.04s | 0.33s |
| Linux | 92,923 | 2.95s | 0.47s |
Note: Cold Save includes the initial index creation. Incremental Save reflects the time when 5 files are modified.
Sources: README.md:21-30
Conclusion
Incremental File Indexing is the mechanism that transforms chkpt from a simple backup tool into a high-performance checkpointing system. By maintaining a lean binary index of file metadata, the system avoids redundant I/O and hashing, enabling near-instant snapshots for large development workspaces. This index is kept strictly in sync during both save and restore operations to maintain the integrity of the content-addressable store.