The Filesystem as a Database
Relevant source files
The following files were used as context for generating this wiki page:Cyberpath Sentinel is a document-based DBMS where the filesystem itself is the database. This design choice prioritizes trust, transparency, and compliance over raw speed. All data is stored as files on disk, tables are represented by folders, and each document's primary key is its filename. This approach makes every piece of data inspectable, auditable, and compliant by design, leveraging the reliability and tooling of the underlying operating system.
The core philosophy is to compose with standard UNIX tools (grep, rsync, tar) for operations, eliminating the need for a separate database daemon. This makes Sentinel particularly well-suited for audit logs, compliance data, certificate management, and edge devices where operational simplicity and data transparency are paramount. For more details on the project's goals, see the Implementation Plan.
Sources: README.md:9-11, README.md:162-166, IMPLEMENTATION_PLAN.md:13-19
Design Principles
The architecture is founded on a set of core principles that guide its development and use cases.
- Filesystem is the Database: Leverages OS reliability and standard tooling.
- Immutability by Default: Prioritizes audit trails and append-only patterns. Deletions are soft by default.
- Security First: Relies on file permissions, encryption, and cryptographic verification.
- UNIX Philosophy: Does one thing well (transparent storage) and composes with other tools.
- Zero External Dependencies: Designed to work offline and on edge devices without needing external services.
Sources: README.md:162-166
Architecture and Data Model
The system is designed as a library that interacts directly with the filesystem, with layers for querying, transactions, and caching built on top of the core file I/O layer.
Sources: IMPLEMENTATION_PLAN.md:21-36
Core Components
The following diagram illustrates the layered architecture of Sentinel.
This diagram shows the flow from the client application down through the logical layers to the physical filesystem storage.
Sources: README.md:168-179, IMPLEMENTATION_PLAN.md:21-36
Data Storage Model
Sentinel organizes data in a hierarchical folder structure. A Store is the root directory, and each sub-directory within it represents a Collection (analogous to a table). Documents are individual files within a collection's folder.
This diagram shows the on-disk layout for a Sentinel store with users and audit_logs collections.
Sources: README.md:124-135, IMPLEMENTATION_PLAN.md:41-59
Document Structure
Documents are stored in JSON format. The system reserves certain top-level keys, prefixed with an underscore, for metadata.
| Field | Type | Description |
|---|---|---|
_id | String | The document's primary key, matching the filename. |
_ts | String | ISO 8601 timestamp of the last update. |
_v | Integer | The version number of the document. |
_hash | String | A SHA-256 hash of the document's content for integrity checks. |
_sig | String | An optional RSA-2048 signature for tamper detection. |
Sources: IMPLEMENTATION_PLAN.md:61-71, IMPLEMENTATION_PLAN.md:121-122
Core Operations
All database operations map directly to filesystem actions, which can be performed using the Sentinel library.
CRUD Operations
| Operation | Filesystem Action | Description |
|---|---|---|
| Insert | write file | Creates a new file (e.g., data/users/user-123.json). |
| Get | read file | Reads a specific file by its name (primary key). |
| Update | atomic rename | Writes a new file and atomically renames it to replace the old one. |
| Delete | move file | Moves the file to a .deleted/ sub-directory (soft delete). |
| List | scan directory | Reads the filenames in a collection's directory. |
Sources: README.md:95-122, README.md:195-202
The following sequence diagram shows the process for inserting a new document.
This flow ensures durability through the Write-Ahead Log before the file is physically written to disk.
Sources: IMPLEMENTATION_PLAN.md:118, IMPLEMENTATION_PLAN.md:143-152, IMPLEMENTATION_PLAN.md:162-175
Core Data Structures
The IMPLEMENTATION_PLAN.md outlines the primary Rust structs that manage the database operations.
This diagram shows the relationship between the Store, Collection, and Document entities.
Sources: IMPLEMENTATION_PLAN.md:77-92, IMPLEMENTATION_PLAN.md:102-122
Durability and Concurrency
To handle the trade-offs of using a filesystem, Sentinel plans several key features.
Write-Ahead Logging (WAL)
To ensure durability and enable crash recovery, all write operations (insert, update, delete) are first recorded in a TransactionLog before being applied to the data files. In case of a crash, the log can be replayed to restore a consistent state.
- Log Entry Types:
Begin,Insert,Update,Delete,Commit,Rollback. - Serialization: Log entries are serialized using
bincodefor efficiency. - Integrity: Log entries include CRC checksums to detect corruption.
Sources: IMPLEMENTATION_PLAN.md:141-159
Concurrency Control
To manage multi-writer scenarios, a FileLockManager is proposed. It will use cross-platform file locking (fcntl on Unix, LockFile on Windows) to prevent race conditions.
- Lock Strategies:
Exclusive(write),Shared(read), and a plannedOptimisticMVCC. - Implementation: The
fs2crate is suggested for handling the OS-level locks.
Sources: IMPLEMENTATION_PLAN.md:161-178
Performance and Scalability
While not designed for high-frequency trading, several strategies are planned to optimize performance for its target use cases.
Query Performance
Initial queries require scanning files, which is an O(n) operation. To improve this, the following solutions are proposed:
- In-memory Caching: An LRU cache for frequently accessed documents.
- Lazy Indexing: Indices are built on the first query against a field and stored in a
.index/directory for future use. Index types include Hash, B-Tree, and Full-Text. - Hash-based Sharding: For collections exceeding a certain size (e.g., 4 million files), documents can be sharded into sub-directories to reduce scan times.
Sources: README.md:55-57, README.md:204-207, IMPLEMENTATION_PLAN.md:199-221, IMPLEMENTATION_PLAN.md:403-418
Performance Characteristics
The following table summarizes the expected time complexity for core operations.
| Operation | Time Complexity | Notes |
|---|---|---|
| Insert | O(1) | A single file write. |
| Get | O(1) | Direct filename lookup. |
| Delete | O(1) | A file rename operation. |
| Update | O(1) | An atomic file rename. |
| List | O(n) | Scans the directory for filenames. |
| Filter | O(n) | Scans all files in a collection (without an index). |
Sources: README.md:195-202
Conclusion
The "Filesystem as a Database" model is the foundational concept of Cyberpath Sentinel. It deliberately trades raw performance for unparalleled transparency, auditability, and operational simplicity. By leveraging the native capabilities of the filesystem and standard system tools, Sentinel provides a robust data storage solution for security, compliance, and auditable workloads. The architecture is designed to be secure by default, with clear plans to address inherent challenges like concurrency and query performance through mechanisms like Write-Ahead Logging and lazy indexing.
Sources: README.md:10-11, README.md:243-250