Operation: Save Checkpoint
Relevant source files
The following files were used as context for generating this wiki page:
The Save Checkpoint operation is the primary mechanism in chkpt for capturing a point-in-time snapshot of a workspace. It creates a content-addressed, deduplicated, and compressed backup of files without utilizing Git, allowing for rapid rollbacks and incremental storage.
The operation follows a structured pipeline that includes scanning the filesystem, identifying changed files via a SQLite-backed index, hashing content with XXH3-128, and persisting data in a sharded pack-file structure.
Sources: README.md:5-13, crates/chkpt-core/src/ops/save.rs:77-111
Orchestration Logic
The save operation is orchestrated by the save function in the chkpt-core crate. It coordinates multiple subsystems including the filesystem scanner, the index manager, and the content-addressable storage (CAS).
Save Execution Flow
The following diagram illustrates the high-level sequence of steps taken during a save operation.
Explanation: This flowchart represents the linear progression of the save pipeline, from initial project identification to final metadata persistence.
Sources: crates/chkpt-core/src/ops/save.rs:77-241, README.md:120-128
Core Components and Data Structures
The save operation relies on specific structures to manage file metadata and processing states.
Configuration and Options
| Structure | Field | Type | Description |
|---|---|---|---|
SaveOptions | message | Option<String> | Optional label for the checkpoint. |
SaveOptions | include_deps | bool | Whether to include directories like node_modules. |
SaveResult | snapshot_id | String | The unique UUID of the created checkpoint. |
SaveResult | stats | SnapshotStats | Metrics including file count and bytes saved. |
Sources: crates/chkpt-core/src/ops/save.rs:24-34, crates/chkpt-cli/src/main.rs:107-114
Internal Pipeline Structures
PreparedFile: A temporary structure used during the pipeline containing the relative path, blob hash, compressed content (if new), and filesystem metadata (mtime, inode).ShardedSeenHashes: A thread-safe, shardedHashSetused to perform in-memory deduplication during parallel processing to reduce lock contention.ProcessedFile: The final representation of a file after its content has been hashed and stored, used for building the directory tree.
Sources: crates/chkpt-core/src/ops/save.rs:43-69
Processing Pipeline
The pipeline is optimized for performance using parallel processing and incremental change detection.
Incremental Change Detection
chkpt uses a FileIndex (stored as index.bin) to skip processing for unchanged files. A file is considered "cached" if its path, size, mtime (seconds and nanos), inode, and mode match the entries in the index.
Sources: crates/chkpt-core/src/ops/save.rs:114-135, crates/chkpt-core/benches/save_pipeline.rs:170-188
Content Processing (Read, Hash, Compress)
For files not found in the cache, the system initiates a parallel pipeline:
- Reading: Files are read using standard I/O or
mmapfor larger files. - Hashing: XXH3-128 generates a 16-byte content hash.
- Deduplication: The hash is checked against
known_blob_hashes(from the catalog) and theShardedSeenHashesset. - Compression: New content is compressed using LZ4.
Explanation: This sequence diagram shows the deduplication logic used to avoid redundant compression and storage of identical file contents.
Sources: crates/chkpt-core/src/ops/save.rs:155-189, crates/chkpt-core/benches/save_pipeline.rs:260-285
Metadata Persistence
After files are processed, the operation persists the workspace structure.
Merkle Tree Construction
chkpt builds a tree structure bottom-up. It groups ProcessedFile entries by directory, hashes the directory manifest, and recursively builds parent trees until the root hash is generated. This allows for fast comparison between snapshots.
Sources: crates/chkpt-core/src/ops/save.rs:470-550
Catalog and Index Update
- Catalog: The
catalog.sqlitedatabase records the newSnapshotmetadata, itsSnapshotStats, and theManifestEntrylist (mapping paths to blob hashes). - Index: The
index.binfile is updated to remove stale paths and add the newly processedFileEntrydata, ensuring the next save is incremental.
Sources: crates/chkpt-core/src/ops/save.rs:222-241, crates/chkpt-napi/test/e2e.spec.ts:33-42
Conclusion
Operation: Save Checkpoint provides a high-performance, incremental backup solution by combining parallelized content processing with a robust indexing system. By utilizing content-addressing and LZ4 compression, it ensures that workspace snapshots are both storage-efficient and computationally fast, enabling developers to create frequent recovery points during risky operations.