SQLite Metadata Catalog
Relevant source files
The following files were used as context for generating this wiki page:
The SQLite Metadata Catalog is the central authority for tracking workspace snapshots, file manifests, and content-addressed blob locations within the chkpt system. It replaces legacy JSON-based metadata storage with a high-performance relational database located at catalog.sqlite within each project's store directory.
The catalog enables fast incremental saves, efficient restore planning, and unified management of the content-addressable storage (CAS) layer. It ensures that metadata operations—such as listing snapshots or resolving content hashes to physical pack files—remain performant even as the number of checkpoints and files grows.
Database Schema and Data Models
The catalog utilizes three primary tables to organize metadata. Relationships between these entities allow for deduplication of both file content and entire snapshot manifests.
Entity Relationship Diagram
The following diagram illustrates the relationships between snapshots, their file manifests, and the underlying blob storage index.
Core Tables
| Table | Description |
|---|---|
snapshots | Stores top-level metadata for each checkpoint, including timestamps, optional messages, and statistical summaries of the save operation. |
snapshot_files | A flattened manifest containing every file path in a snapshot, mapped to its specific content hash (blob_hash), file size, and filesystem mode. |
blob_index | The authority for content location. It maps a blob_hash (the XXH3-128 content hash) to a pack_hash (identifying the .dat file in the packs/ directory). |
Sources: crates/chkpt-core/src/store/catalog.rs:11-38, crates/chkpt-plugin/skills/chkpt/references/store-layout.md:38-66
Architecture and Implementation
Persistence and Optimization
The MetadataCatalog implementation in chkpt-core uses rusqlite for database interactions. Upon opening the database, several PRAGMA settings are applied to optimize for speed and concurrency, specifically leveraging Write-Ahead Logging (WAL).
conn.execute_batch(
"PRAGMA journal_mode=WAL;
PRAGMA synchronous=NORMAL;
PRAGMA cache_size=-64000;
PRAGMA temp_store=MEMORY;
PRAGMA mmap_size=268435456;",
)?;
Sources: crates/chkpt-core/src/store/catalog.rs:124-130
Snapshot Lifecycle Management
1. Save and Insertion
During a save operation, the system identifies changed files. If a snapshot's manifest is identical to its parent, the catalog avoids duplicating the snapshot_files entries. It uses the manifest_snapshot_id column to point to the snapshot that actually owns the file list.
- Logic: If
new_objects == 0and no paths were removed or updated,insert_snapshot_metadata_onlyis called to create a lightweight snapshot entry. - Tree Construction: Even if the manifest is reused, a unique
root_tree_hashis calculated or retrieved to represent the directory structure.
Sources: crates/chkpt-core/src/ops/save.rs:218-250, crates/chkpt-core/src/store/catalog.rs:149-183
2. Resolution and Retrieval
The catalog provides flexible resolution for snapshot references. Users can request latest or provide a partial ID (prefix).
3. Deletion and Cleanup
Deleting a snapshot involves a multi-step process to ensure data integrity. If a snapshot being deleted is the "owner" of a manifest reused by other snapshots, the catalog reassigns ownership of the manifest to one of the aliased snapshots before performing the deletion.
- Blob Garbage Collection: After a snapshot is deleted, the system determines which blobs are no longer reachable by any remaining snapshots.
- Pack Management: If all blobs within a specific pack file are unreferenced, the catalog facilitates the physical deletion of the
.datand.idxfiles from the filesystem.
Sources: crates/chkpt-core/src/ops/delete.rs:80-130, crates/chkpt-core/src/store/catalog.rs:261-303
Blob Indexing and Content Addressability
The blob_index table is critical for the system's deduplication capabilities. It tracks the mapping of 128-bit content hashes to physical storage locations.
- Bulk Upserts: During saving, new blob locations are recorded in batches using
bulk_upsert_blob_locationswith anON CONFLICT DO UPDATEclause to handle potential collisions or updates. - Reference Counting: The method
pack_reference_countis used during cleanup to check how many active blobs still reside within a specific pack file.
Sources: crates/chkpt-core/src/store/catalog.rs:305-345, crates/chkpt-core/src/store/catalog.rs:378-382
Location and Configuration
The catalog file is always named catalog.sqlite and is located within the project's unique store directory. This path is determined by hashing the canonical workspace path.
- Default Path:
~/.chkpt/stores/<project_id>/catalog.sqlite - Override: The
CHKPT_HOMEenvironment variable can change the root.chkptdirectory.
Sources: crates/chkpt-core/src/config.rs:5-45, crates/chkpt-plugin/skills/chkpt/references/store-layout.md:5-15
Conclusion
The SQLite Metadata Catalog provides the robust transactional foundation required for chkpt's performance goals. By centralizing snapshot metadata, manifest records, and the blob index into a single relational database, the system achieves fast lookups and reliable management of deduplicated content. Its support for manifest aliasing and automated pack cleanup ensures that storage remains efficient over the long-term lifecycle of a project.