Stores and Collections

Relevant source files

The following files were used as context for generating this wiki page:

• crates/sentinel/src/lib.rs
• README.md
• IMPLEMENTATION_PLAN.md
• crates/sentinel/Cargo.toml
• Cargo.toml

Stores and Collections are the fundamental architectural components for data organization in Cyberpath Sentinel. The system is built on the principle that the "Filesystem is the Database," where data is stored as files on disk. A Store represents the root of the database, corresponding to a main directory. Within a Store, data is organized into Collections, which are represented by sub-folders and are analogous to tables in a traditional DBMS.

This model provides native auditability and transparency, as every document is an inspectable file, and every collection is a directory. Standard filesystem tools can be used for operations like backups (tar), replication (rsync), and simple queries (grep). The entire structure is designed for simplicity, security, and compliance.

Sources: README.md:10-12, README.md:40-43, IMPLEMENTATION_PLAN.md:20-25

Architecture Overview

The core architecture of Sentinel revolves around a hierarchical structure managed by the Store and Collection managers. A Store is the top-level container, representing a directory path on the filesystem. Each Store can contain multiple Collection instances, which are themselves directories that hold the actual data documents as individual files (e.g., JSON).

Sources: README.md:10-12, IMPLEMENTATION_PLAN.md:27-41

This diagram shows the logical hierarchy of data organization in Sentinel.

Physical Directory Structure

On the filesystem, a store and its collections are laid out as a nested directory structure. Special subdirectories like .deleted, .index, and .wal are planned for handling soft deletes, query indices, and write-ahead logging, respectively.

data/
├── users/
│   ├── .metadata.json
│   ├── .deleted/
│   │   └── user-789.json
│   ├── .index/
│   │   └── email.idx
│   ├── user-123.json
│   └── user-456.json
└── audit_logs/
    ├── .wal/
    └── audit-2026-01-01.json

Sources: README.md:112-123, IMPLEMENTATION_PLAN.md:44-59

Store Manager

The Store Manager is the entry point for all database operations. It is responsible for managing the root directory, handling store-level configurations, and creating or accessing collections. The implementation is planned as a Store struct.

Sources: IMPLEMENTATION_PLAN.md:63-69

Store and StoreConfig Data Structures

The Store struct holds the root path and configuration, while StoreConfig defines operational parameters like caching, WAL, and encryption settings.

StoreConfig Options

The behavior of a Store is controlled by the StoreConfig struct.

Parameter	Type	Description	Default Value
max_collection_size	usize	Maximum number of files in a collection.	4,000,000
enable_wal	bool	Enables the Write-Ahead Log for durability.	true
enable_encryption	bool	Enables file-level encryption.	false
enable_caching	bool	Enables the in-memory LRU cache.	true
cache_size_mb	usize	Size of the in-memory cache in megabytes.	256 MB
checkpoint_interval	Duration	Interval for performing WAL checkpoints.	5 minutes

Sources: IMPLEMENTATION_PLAN.md:70-77

Store Methods

Key methods planned for the Store struct for managing collections.

Method	Description
new()	Creates a new Store instance at the given path.
collection()	Gets an existing Collection by name.
create_collection()	Creates a new Collection with a specific config.
delete_collection()	Deletes a Collection.
list_collections()	Lists the names of all collections in the store.
backup()	Backs up the entire store to a destination.
restore()	Restores a store from a backup source.

Sources: IMPLEMENTATION_PLAN.md:79-86

Collection Manager

The Collection Manager handles all document-level operations within a specific collection (folder). It is responsible for CRUD operations, queries, and managing collection-specific features like indexing and transactions.

Sources: IMPLEMENTATION_PLAN.md:92-97

Collection and Document Data Structures

The Collection struct manages documents, which are represented by the Document struct. A Document contains the user data (serde_json::Value) and internal metadata.

Document Structure

Each document contains user-defined data as well as system-managed metadata fields, which are crucial for auditing and integrity verification.

Field	Type	Description
_id	String	The document's primary key (and filename).
_ts	String (ISO 8601)	Timestamp of the last update.
_v	integer	Version number of the document.
_hash	String (e.g., sha256:abc...)	Cryptographic hash of the document content.
_sig	String (e.g., rsa2048:def...)	Digital signature for tamper detection.
encrypted_fields	Array<String>	A list of fields that are encrypted at rest.

Sources: IMPLEMENTATION_PLAN.md:60-62

Basic Usage Flow

The typical workflow involves creating a Store, getting or creating a Collection from it, and then performing CRUD operations on documents within that collection.

This sequence diagram illustrates creating a collection and inserting a new document.

Example Code

The following Rust code demonstrates the basic API for interacting with stores and collections.

use cyberpath_sentinel::Store;

// Create a store (creates data/ folder)
let store = Store::new("./data").expect("Failed to create store");

// Create a collection (creates data/users/ folder)
let users = store.collection("users").expect("Failed to create collection");

// Insert a document (creates data/users/user-123.json)
users.insert("user-123", json!({
    "name": "Alice",
    "email": "alice@example.com"
})).expect("Failed to insert");

// Query documents
let user = users.get("user-123").expect("Failed to get");

// List all documents
let all_users = users.list().expect("Failed to list");

// Update a document
users.update("user-123", json!({
    "name": "Alice",
    "email": "alice.smith@example.com"
})).expect("Failed to update");

// Delete a document (moves to data/users/.deleted/user-123.json)
users.delete("user-123").expect("Failed to delete");

Sources: README.md:86-110, crates/sentinel/src/lib.rs