Content-Addressable Storage (Blobs & Packs)
Relevant source files
The following files were used as context for generating this wiki page:
Content-Addressable Storage (CAS) is the fundamental storage paradigm of the chkpt system. Instead of referencing files by their path or name, the system identifies and stores data based on a cryptographic hash of its contents. This approach enables native deduplication—identical files across different snapshots or even within the same workspace are only stored once.
The storage layer is divided into two primary concepts: Blobs, which represent the raw, compressed data of individual files, and Packs, which aggregate multiple blobs into larger contiguous files to improve filesystem efficiency and reduce metadata overhead.
Sources: README.md:9-10, crates/chkpt-core/src/ops/save.rs:88-95
Core Components and Architecture
The CAS architecture is designed for high-performance ingestion and retrieval. It utilizes the XXH3-128 hashing algorithm for speed and LZ4 compression to minimize disk footprint.
Data Flow for Blob Storage
When a file is saved, it passes through a pipeline of scanning, hashing, and compression before being written to a pack.
The system uses a sharded deduplication set to reduce lock contention during parallel processing of multiple files.
Sources: crates/chkpt-core/src/ops/save.rs:135-155, crates/chkpt-core/src/store/blob.rs:36-39
Storage Structures
| Component | Description | Format / Implementation |
|---|---|---|
| Blob | The unit of content. Identified by a 16-byte (128-bit) hash. | XXH3-128 |
| Pack | A collection of compressed blobs stored in .dat files. | LZ4 Encapsulated |
| Tree | A content-addressed manifest of a directory's state. | Bitcode Encoded |
| Catalog | SQLite database mapping hashes to specific pack locations. | catalog.sqlite |
Sources: crates/chkpt-cli/skills/chkpt/references/store-layout.md:7-15, crates/chkpt-core/src/store/tree.rs:19-26
Blob Processing and Hashing
Blobs are the raw data units of the system. The processing logic optimizes for both small and large files by using different memory strategies.
Memory Strategies
To handle diverse file sizes efficiently, chkpt employs a hybrid approach:
- Small Files (< 256 KB): Read directly into a heap-allocated
Vec<u8>. - Large Files (>= 256 KB): Memory-mapped (
Mmap) to avoid unnecessary copying and reduce memory pressure.
Sources: crates/chkpt-core/src/store/blob.rs:7-34
Hashing Mechanism
The system uses XXH3-128 to generate a 16-byte hash. This hash serves as the "address" for the content.
- Raw Bytes: Used internally for storage efficiency ([u8; 16]).
- Hex Representation: A 32-character string used for filenames and human-readable metadata.
Sources: crates/chkpt-core/src/store/blob.rs:41-49
Pack Management
Packs prevent the "small file problem" by aggregating many blobs into a single file. This improves I/O performance during both backup and restore.
Catalog Schema for Blobs
The relationship between a blob and its containing pack is managed via the blob_index table in the SQLite catalog.
Tree Packs
Directories are also stored as content-addressed objects called "Trees." These are encoded using the bitcode crate and stored in a specialized tree pack (trees.dat).
Directory Layout
The physical storage is organized to prevent performance degradation on filesystems that struggle with large numbers of files in a single directory.
| Path | Purpose |
|---|---|
packs/ | Contains .dat files holding compressed file blobs. |
trees/ | Stores directory structure manifests. |
catalog.sqlite | Authoritative index of all snapshots and blob locations. |
index.bin | Local metadata cache for incremental save detection. |
Sources: crates/chkpt-core/src/config.rs:45-65, crates/chkpt-cli/skills/chkpt/references/store-layout.md:7-15
Implementation Details: Tree Building
The system builds trees "bottom-up." It groups processed files by their parent directory, computes the hash for the deepest directories first, and then includes those directory hashes as entries in their respective parent trees.
// Logic from crates/chkpt-core/src/ops/save.rs:500-515
// Sort directories bottom-up (deepest first)
dir_list.sort_unstable_by(|a, b| {
let depth_a = if a.is_empty() { 0 } else { a.matches('/').count() + 1 };
let depth_b = if b.is_empty() { 0 } else { b.matches('/').count() + 1 };
depth_b.cmp(&depth_a).then_with(|| a.cmp(b))
});
Sources: crates/chkpt-core/src/ops/save.rs:500-515
Tree Entry Structure
Each entry in a tree contains the name, type (File, Dir, or Symlink), the content hash, size, and filesystem mode.
Sources: crates/chkpt-core/src/store/tree.rs:19-26
Summary
The Content-Addressable Storage system in chkpt provides an efficient, deduplicated, and compressed foundation for workspace snapshots. By utilizing XXH3-128 hashing, LZ4 compression, and a hybrid memory-mapping strategy, it achieves high throughput for both cold and incremental saves while maintaining a compact on-disk representation through the use of Packs and Trees.