Error Handling & Concurrency
Relevant source files
The following files were used as context for generating this wiki page:
Error handling and concurrency in the chkpt project are designed to ensure data integrity and high performance during workspace snapshots. The system employs a "libraries use thiserror, binaries use anyhow" strategy for error management and utilizes multi-threaded pipelines with file-system locking to manage concurrent access and heavy I/O workloads.
Concurrency is primarily managed through sharded deduplication sets, thread-scoped parallel processing, and cross-platform locking mechanisms to prevent data corruption during simultaneous operations.
Sources: CLAUDE.md:38-38, crates/chkpt-core/src/ops/save.rs:88-100
Error Handling Architecture
The project follows a standard Rust error handling pattern tailored for a workspace-oriented library. Errors are categorized based on their impact on the store (e.g., corruption, locking, or I/O).
Error Strategy
- Library Crates: Use the
thiserrorcrate to define explicit, actionable error types. This allows calling code to programmatically handle specific failure modes likeLock heldorSnapshot not found. - Binary Crates: Use the
anyhowcrate for flexible error reporting in the CLI and MCP server, providing context to the end-user.
Sources: CLAUDE.md:38-38, crates/chkpt-plugin/skills/chkpt/SKILL.md:65-70
Common Error Types
| Error Condition | Internal Meaning | Typical Action |
|---|---|---|
Lock held by another process | A ProjectLock is currently active for the workspace. | Inform user and suggest waiting or retrying. |
Snapshot not found | The requested UUID does not exist in the catalog.sqlite. | Suggest running chkpt list. |
Store corrupted | Integrity check failed (e.g., missing manifest or root tree). | Suggest manual inspection of the store. |
IO error | Permission issues or disk full during blob writing. | Report raw system error. |
Sources: crates/chkpt-cli/skills/chkpt/references/cli-commands.md:111-116, crates/chkpt-core/src/ops/save.rs:240-244
Concurrency & Locking
To prevent multiple processes from corrupting the content-addressable store (especially the SQLite catalog and pack files), chkpt implements a mandatory locking mechanism.
Project-Level Locking
Before any mutation (save, restore, delete) begins, the system must acquire a ProjectLock. This lock is stored in the .chkpt/stores/<project_id>/locks directory.
The lock is automatically released when the ProjectLock guard is dropped at the end of the function scope.
Sources: crates/chkpt-core/src/ops/save.rs:88-90, crates/chkpt-core/src/config.rs:59-61
Parallel Save Pipeline
The save operation utilizes a sophisticated parallel pipeline to maximize CPU and I/O throughput.
Threaded Processing Flow
The process_prepared_files_streaming function splits scanned files into chunks and processes them using a thread pool. This pipeline handles hashing (XXH3-128) and compression (LZ4) concurrently.
Sharded Deduplication
To allow high-concurrency hashing without a single global bottleneck, the system uses ShardedSeenHashes. It distributes hashes across 64 mutex-protected shards based on the first two bytes of the hash.
struct ShardedSeenHashes {
shards: Vec<Mutex<HashSet<[u8; 16]>>>,
}
impl ShardedSeenHashes {
fn insert(&self, hash: [u8; 16]) -> bool {
let shard_index = ((hash[0] as usize) << 8 | hash[1] as usize) % self.shards.len();
let mut shard = self.shards[shard_index].lock().unwrap();
shard.insert(hash)
}
}
Sources: crates/chkpt-core/src/ops/save.rs:65-81
I/O Locality and Performance
- Inode Sorting: Scanned files are sorted by Inode to reduce random I/O seek times during parallel reads.
- Hybrid Reads: Small files are read via standard FS calls, while larger files (≥64KB) can be memory-mapped (
mmap) for efficiency. - Hardlink Preservation: The pipeline ensures that hardlinked files are processed in a way that avoids redundant hashing and compression.
Sources: crates/chkpt-core/src/ops/save.rs:136-137, crates/chkpt-core/benches/save_pipeline.rs:14-49, crates/chkpt-core/src/ops/save.rs:432-435
Technical Summary
| Component | Responsibility | Technical Basis |
|---|---|---|
| Error Handling | Programmatic failure recovery | thiserror / anyhow |
| Concurrency | Parallel hash/compress | std::thread::scope / mpsc |
| Deduplication | Lock-free-ish hash set | ShardedSeenHashes (64 shards) |
| Mutual Exclusion | Single-writer protection | ProjectLock (file-system based) |
| I/O Optimization | Maximize disk throughput | Inode sorting & mmap hybrid |
Concurrency and error handling in chkpt ensure that even large-scale workspaces (like the Linux kernel) can be snapshotted rapidly without risking data corruption from concurrent processes or unexpected I/O failures.
Sources: crates/chkpt-core/src/ops/save.rs:88-150, crates/chkpt-core/benches/save_pipeline.rs:105-125