Core Library Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The chkpt-core library serves as the engine for the chkpt system, providing high-performance, content-addressable snapshot capabilities for workspaces. It is designed to capture the entire state of a directory, including untracked files, while minimizing storage footprints through deduplication and compression.
The architecture centers on a content-addressed storage (CAS) model using XXH3-128 hashing for deduplication and LZ4 for data compression. The library manages workspace scanning, incremental change detection via an SQLite-backed catalog, and efficient restoration of snapshots. It provides the underlying logic used by the CLI, Node.js bindings, and MCP tools.
Sources: README.md:5-13, CLAUDE.md:7-10
Storage and Directory Layout
The core library organizes data into specific project stores located within a hidden directory (defaulting to ~/.chkpt/). Each project is assigned a deterministic 16-character hex ID derived from its canonical workspace path.
Store Components
The StoreLayout structure defines the hierarchy of the storage engine:
| Component | Path | Description |
|---|---|---|
| Catalog | catalog.sqlite | SQLite database containing snapshot metadata, manifests, and blob indices. |
| Index | index.bin | Binary cache of current workspace metadata for incremental saves. |
| Packs | packs/ | Compressed blob data stored in .dat files. |
| Trees | trees/ | Content-addressed tree nodes representing directory structures. |
| Locks | locks/ | Process synchronization markers (e.g., project.lock). |
Sources: crates/chkpt-core/src/config.rs:13-58, crates/chkpt-cli/skills/chkpt/references/store-layout.md:3-13
Logical Storage Relationship
The following diagram illustrates how workspace files are transformed into the internal store structure.
This diagram shows the flow from physical workspace files to the deduplicated, compressed storage backend.
Sources: README.md:95-103, crates/chkpt-core/src/ops/save.rs:88-150
Save Pipeline
The save operation is an orchestrated multi-stage pipeline designed for performance, utilizing parallel processing for hashing and compression.
Stages of a Save Operation
- Project Identification: Computes a unique project ID from the workspace path using XXH3-128.
- Scanning: Recursively walks the workspace. It respects
.chkptignoreand ignores.git,node_modules, and.chkptdirectories by default. - Incremental Detection: Compares scanned file metadata (mtime, size, inode) against the
index.bincache to identify changed files. - Content Hashing & Compression: New or modified files are hashed (XXH3-128). Data is compressed using LZ4 in a bounded producer/consumer pipeline to manage memory.
- Deduplication: Hashes are checked against
catalog.sqlite. Identical content is skipped even if the file paths differ. - Pack Writing: New compressed blobs are appended to pack files in the
packs/directory. - Tree Building: A bottom-up reconstruction of the directory hierarchy creates a Merkle tree, resulting in a single root tree hash.
- Catalog Update: Persists snapshot metadata and file manifests into the SQLite database.
Sources: crates/chkpt-core/src/ops/save.rs:88-164, crates/chkpt-core/tests/scanner_test.rs:18-50
Parallel Pipeline Flow
The save pipeline utilizes a sharded deduplication set to reduce lock contention during multi-threaded processing.
This diagram details the concurrent execution of file processing during the save operation.
Sources: crates/chkpt-core/src/ops/save.rs:59-86, crates/chkpt-core/src/ops/save.rs:446-500
Metadata Catalog Schema
The catalog.sqlite database is the authoritative source for snapshot history and blob locations.
| Table | Primary Fields | Description |
|---|---|---|
| snapshots | id, created_at, root_tree_hash | High-level snapshot records and parent/child relationships. |
| snapshot_files | snapshot_id, path, blob_hash | Flattened manifests mapping paths to specific content hashes. |
| blob_index | blob_hash, pack_hash, size | Global index mapping content hashes to their physical pack locations. |
Sources: crates/chkpt-cli/skills/chkpt/references/store-layout.md:31-52
Data Structures
ScannedFile
Represents file metadata retrieved during the workspace scan.
pub struct ScannedFile {
pub relative_path: String,
pub absolute_path: PathBuf,
pub size: u64,
pub mtime_secs: i64,
pub mtime_nanos: i64,
pub inode: Option<u64>,
pub mode: u32,
pub is_symlink: bool,
pub device: Option<u64>,
}
Sources: crates/chkpt-core/src/ops/save.rs:12-22, crates/chkpt-core/tests/scanner_test.rs:136-141
SnapshotStats
Aggregated metrics recorded for every checkpoint.
pub struct SnapshotStats {
pub total_files: u64,
pub total_bytes: u64,
pub new_objects: u64,
}
Sources: crates/chkpt-core/src/ops/save.rs:166-170
Architecture Principles
Content-Addressability
Every file is identified by the XXH3-128 hash of its content. This allows the system to store a file only once, even if it appears multiple times in the workspace or across different snapshots.
Sources: README.md:16, CLAUDE.md:7-8
Performance Optimizations
- Locality Sorting: During save, files are sorted by inode/on-disk locality to reduce random I/O during hashing.
- Mmap Hybrid: Large files are mapped into memory (
memmap2) for hashing, while small files are read via standard I/O. - Sharded Deduplication: Prevents global lock contention during parallel processing by splitting the "seen hashes" set into 64 shards.
- Incremental Index: A binary cache (
index.bin) avoids re-hashing unchanged files by comparing filesystem metadata.
Sources: crates/chkpt-core/src/ops/save.rs:64-83, crates/chkpt-core/benches/save_pipeline.rs:41-48, crates/chkpt-core/benches/save_pipeline.rs:134-145
Conclusion
The chkpt-core library provides a robust foundation for workspace versioning by combining high-speed hashing, parallel compression, and an efficient metadata management system. Its reliance on a content-addressed storage model ensures significant disk space savings through deduplication, while the incremental save logic provides near-instant snapshots for large projects.