Creating Documents (Insert)

The core "create" operation in Cyberpath Sentinel is the insertion of a new document into a collection. This action fundamentally translates to creating a new file on the disk. The document's primary key becomes the filename, and the collection is represented by a folder. This design ensures that every piece of data is inspectable and auditable by design, leveraging the filesystem as the database.

The insert operation is designed to be atomic and durable, with plans for concurrency control and write-ahead logging to handle multi-writer scenarios and ensure crash recovery. From a performance perspective, a document insertion is an O(1) operation, as it involves a single file write.

Sources: README.md:11-12, README.md:172-173, IMPLEMENTATION_PLAN.md:20-26

Core API

The primary method for creating a new document is Collection::insert. This asynchronous function is part of the Collection manager, which handles all document operations within a specific collection (folder).

Sources: IMPLEMENTATION_PLAN.md:100-119

Function Signature

The proposed implementation defines the insert function as follows:

// From IMPLEMENTATION_PLAN.md
impl Collection {
    pub async fn insert(&self, id: &str, data: Value) -> io::Result<()>
    // ...
}

Sources: IMPLEMENTATION_PLAN.md:114-117

Parameters

Sources: IMPLEMENTATION_PLAN.md:114-117, IMPLEMENTATION_PLAN.md:73

Basic Usage Example

// From README.md
use cyberpath_sentinel::Store;

// Create a store (creates data/ folder)
let store = Store::new("./data").expect("Failed to create store");

// Create a collection (creates data/users/ folder)
let users = store.collection("users").expect("Failed to create collection");

// Insert a document (creates data/users/user-123.json)
users.insert("user-123", json!({
    "name": "Alice",
    "email": "alice@example.com",
    "role": "admin",
    "created_at": "2026-01-11T12:00:00Z"
})).expect("Failed to insert");

Sources: README.md:86-101

Insert Operation Flow

The process of inserting a document involves several components to ensure durability and consistency, particularly the Write-Ahead Log (WAL) and the Concurrency Control layer.

This diagram illustrates the planned high-level flow for an insert operation.

A more detailed sequence of events for a transactional insert is as follows:

This sequence diagram shows how the insert operation interacts with the TransactionLog and FileLockManager to ensure data is written durably and without conflicts.

Sources: IMPLEMENTATION_PLAN.md:100-119, IMPLEMENTATION_PLAN.md:144-161, IMPLEMENTATION_PLAN.md:166-182

Document Structure

When a document is inserted, it is stored as a JSON file. The implementation plan specifies that Sentinel will add internal metadata fields to the user-provided data.

Standard Document Fields

Sources: IMPLEMENTATION_PLAN.md:63-75

Example JSON Document

This is how a document user-123.json would look on disk after insertion, including the system-managed metadata.

{
  "_id": "user-123",
  "_ts": "2026-01-11T12:00:00Z",
  "_v": 1,
  "_hash": "sha256:abc123...",
  "_sig": "rsa2048:def456...",
  "name": "Alice",
  "email": "alice@example.com",
  "role": "admin",
  "encrypted_fields": []
}

Sources: IMPLEMENTATION_PLAN.md:63-75

Durability via Write-Ahead Logging (WAL)

To ensure that insert operations are not lost during a crash, Sentinel plans to use a Write-Ahead Log. Before the document file is written, a log entry is recorded.

WAL Entry for Insert

The TransactionLog will write a specific LogEntry enum variant for an insert operation.

// From IMPLEMENTATION_PLAN.md
pub enum LogEntry {
    Begin { transaction_id: String },
    Insert { collection: String, id: String, data: Value },
    Update { collection: String, id: String, data: Value },
    Delete { collection: String, id: String },
    Commit { transaction_id: String },
    Rollback { transaction_id: String },
}

The LogEntry::Insert variant captures all necessary information to replay the operation during recovery. The log itself is proposed to be a binary format for efficiency, using bincode for serialization.

Sources: IMPLEMENTATION_PLAN.md:149-157, IMPLEMENTATION_PLAN.md:163

Concurrency and Locking

To handle scenarios with multiple concurrent writers, the insert operation must acquire a lock before writing to the filesystem. The FileLockManager is responsible for this. For an insert, an Exclusive lock is required to prevent any other read or write operations on the same resource.

The FileLockManager is planned to use fs2 for cross-platform file locking.

Sources: IMPLEMENTATION_PLAN.md:166-182, Cargo.toml:10-20

Summary

The document insert operation is a foundational feature of Cyberpath Sentinel. It is designed to be a simple file-write at its core, but is architected with layers for transactionality, durability, and concurrency. By leveraging a Write-Ahead Log, file-level locking, and a simple API, the insert operation provides a robust and auditable method for creating new data while maintaining the project's core principles of transparency and security.

Sources: README.md:11-14, IMPLEMENTATION_PLAN.md:20-26