Reading Documents (Get & List)

Relevant source files

The following files were used as context for generating this wiki page:

• crates/sentinel/src/lib.rs
• README.md
• IMPLEMENTATION_PLAN.md
• Cargo.toml
• crates/sentinel/Cargo.toml
• rustfmt.toml
• CODE_OF_CONDUCT.md

Reading documents is a fundamental capability of the Cyberpath Sentinel DBMS. Because Sentinel uses the filesystem as its database, reading operations are designed to be transparent, auditable, and simple. A "collection" is a folder on disk, and each "document" is a file within that folder, typically in JSON format. The document's primary key is its filename.

This system prioritizes trust and inspectability over the high-speed performance of traditional databases. Reading a document can be as simple as using standard OS tools like cat, but the Sentinel library provides a structured API for these operations. This document outlines the two primary read operations: retrieving a single document (get) and listing all documents in a collection (list).

Sources: README.md:9-11, README.md:20-23, IMPLEMENTATION_PLAN.md:19-24

Core Concepts

The core design principle of Sentinel is "Filesystem is the Database". This means all data is stored as files on disk, making it inspectable and manageable with standard command-line tools.

Sources: README.md:111-112, IMPLEMENTATION_PLAN.md:26-30

Data Storage Model

A Store represents the root database directory. Within the store, each Collection is a subdirectory. Documents are individual files within their respective collection folder.

The diagram below illustrates the on-disk data structure.

This diagram shows a typical folder structure for a Sentinel database.
Sources: README.md:83-94, IMPLEMENTATION_PLAN.md:41-56

Document Structure

Each document is a JSON file that contains user data along with metadata managed by Sentinel. The proposed structure includes fields for versioning, timestamps, and integrity checks.

Field	Type	Description
_id	String	The document's unique identifier, matching the filename.
_ts	String	ISO 8601 timestamp of the last update.
_v	Integer	The version number of the document.
_hash	String	A cryptographic hash (e.g., SHA-256) of the document content for integrity.
_sig	String	A digital signature (e.g., RSA-2048) to verify the author.
(user_fields)	Any	User-defined data fields.

Sources: IMPLEMENTATION_PLAN.md:58-69

Document Retrieval (Get)

The get operation retrieves a single document from a collection by its ID. Since the document ID is the filename, this operation is a direct file path lookup, making it highly efficient.

Sources: README.md:160, IMPLEMENTATION_PLAN.md:113

API and Usage

The get method is part of the Collection struct. It takes the document ID as an argument and is expected to return the parsed document.

Proposed API Definition:

// From IMPLEMENTATION_PLAN.md
impl Collection {
    pub async fn get(&self, id: &str) -> io::Result<Option<Document>>
}

Sources: IMPLEMENTATION_PLAN.md:113

Example Usage:

// From README.md
let user = users.get("user-123").expect("Failed to get");
println!("{}", user);

Sources: README.md:72-73

Execution Flow

The get operation follows a simple, direct path to the file on disk. The process may be enhanced by a caching layer to avoid disk I/O for frequently accessed documents.

This flowchart shows the process of retrieving a document.

Performance

The get operation has a time complexity of O(1) because it's a direct filename lookup on the filesystem. This is one of the most performant operations in Sentinel.

Sources: README.md:160

Document Listing (List)

The list operation retrieves the IDs of all documents within a collection. This is accomplished by scanning the collection's directory and listing its files.

Sources: README.md:76, README.md:162

API and Usage

The list method is part of the Collection struct and is expected to return a vector of document IDs (strings).

Proposed API Definition:

// From IMPLEMENTATION_PLAN.md
impl Collection {
    pub async fn list(&self) -> io::Result<Vec<String>>
}

Sources: IMPLEMENTATION_PLAN.md:116

Example Usage:

// From README.md
let all_users = users.list().expect("Failed to list");
for user_id in all_users {
    println!("{}", user_id);
}

Sources: README.md:76-79

Execution Flow

The list operation involves iterating over the contents of the collection's directory. It is a more expensive operation than get as it requires a directory scan.

This flowchart shows the process of listing documents in a collection.

Performance

The list operation has a time complexity of O(n), where n is the number of documents in the collection. Performance can degrade in folders with millions of files, a known trade-off of the system. Proposed solutions include hash-based sharding to limit the number of files per directory.

Sources: README.md:162, README.md:46-47, IMPLEMENTATION_PLAN.md:308-316

Core Data Structures

The IMPLEMENTATION_PLAN.md outlines the primary Rust structs that will manage data access. The Store is the top-level object, which manages multiple Collection instances.

This class diagram shows the relationship between the main data structures.

Summary

Reading documents in Cyberpath Sentinel is designed around the core principle of using the filesystem as the database.

• The get operation is an efficient O(1) lookup, ideal for retrieving specific documents by their ID.
• The list operation is an O(n) directory scan, providing a simple way to enumerate all documents but with performance considerations for very large collections.

This approach offers unparalleled transparency and auditability, as every document is a plain file on disk. While it trades the raw query performance of traditional databases, it provides a robust foundation for systems where compliance and data integrity are paramount.

Sources: README.md:9-11, README.md:158-163, IMPLEMENTATION_PLAN.md:19-25