Full Roadmap
Relevant source files
The following files were used as context for generating this wiki page:The Cyberpath Sentinel project is a filesystem-backed document DBMS designed for security, auditability, and compliance. The development is structured into a multi-phase roadmap, starting with core database functionalities and progressively adding advanced features for durability, performance, security, and distribution. The ultimate vision is to create the gold standard for transparent and auditable data storage.
This roadmap outlines the planned features and technical specifications for each implementation phase, from the initial core DBMS to game-changing capabilities like distributed consensus and Merkle tree verification. The project prioritizes transparency and UNIX philosophy, leveraging standard tools and file-based operations wherever possible.
Sources: README.md:10-11, 194-201, IMPLEMENTATION_PLAN.md:10-16
Implementation Phases Overview
The roadmap is broken down into seven distinct phases, each building upon the last to deliver a comprehensive and robust system.
This diagram shows the planned progression of feature development through the seven phases.
Sources: README.md:123-155, IMPLEMENTATION_PLAN.md:369-491
Phase 1: Core DBMS
This initial phase focuses on establishing the fundamental database functionalities. The goal is to create a working system capable of basic document storage and retrieval.
Sources: README.md:124-129, IMPLEMENTATION_PLAN.md:371-393
Key Features
- Document Storage: Store documents as individual files on the filesystem.
- Collections: Represent collections as folders.
- CRUD Operations: Implement Create, Read, Update, and Delete operations. Soft deletes will be handled by moving files to a
.deleted/folder. - JSON Document Format: Use JSON as the primary data format.
- Query Engine: A basic query engine with in-memory filtering capabilities.
- Transaction Support: Initial planning for atomic operations.
The basic structure will involve a Store managing multiple Collection instances.
This diagram illustrates the core Store and Collection structures planned for Phase 1.
Sources: IMPLEMENTATION_PLAN.md:65-81, 98-121
A simple test case demonstrates the intended API usage:
#[cfg(test)]
mod tests {
#[tokio::test]
async fn test_insert_and_retrieve() {
let store = Store::new("/tmp/test").unwrap();
let users = store.collection("users").unwrap();
let doc = json!({ "name": "Alice", "email": "alice@example.com" });
users.insert("user-123", doc.clone()).await.unwrap();
let retrieved = users.get("user-123").await.unwrap();
assert_eq!(retrieved.data, doc);
}
}
Sources: IMPLEMENTATION_PLAN.md:381-392
Phase 2: Durability & Concurrency
This phase addresses the challenges of data integrity during crashes and handling simultaneous write operations.
Sources: README.md:131-136, IMPLEMENTATION_PLAN.md:395-416
Key Features
- Write-Ahead Logging (WAL): Ensures durability and enables crash recovery. All operations are written to a log before being applied to the files.
- File Locking: Manages concurrent writes using exclusive and shared locks to prevent data corruption.
- ACID Transactions: Provides atomicity, consistency, isolation, and durability for operations.
The WAL process is central to this phase, ensuring that in the event of a crash, the database can recover to a consistent state by replaying the log.
This diagram shows the sequence of an insert operation using a Write-Ahead Log.
Sources: IMPLEMENTATION_PLAN.md:144-161, 397-402
Phase 3: Performance & Scale
This phase introduces optimizations to handle large datasets and improve query performance.
Sources: README.md:131-136, IMPLEMENTATION_PLAN.md:418-441
Key Features
- In-Memory Caching: An LRU (Least Recently Used) cache for frequently accessed documents to reduce disk I/O.
- Indexing Strategies: Lazy-built hash indices for O(1) lookups and B-tree indices for O(log n) range queries.
- Full-Text Search: Advanced text search capabilities.
- Sharding: Hash-based sharding to distribute documents across sub-folders, overcoming filesystem limits on the number of files in a single directory.
Lazy indexing is a key strategy, where an index is built only when a field is queried for the first time.
This diagram illustrates the lazy indexing workflow.
Sources: IMPLEMENTATION_PLAN.md:529-539
Phase 4: Encryption & Security
This phase focuses on securing data at rest and ensuring its integrity.
Sources: README.md:143-148, IMPLEMENTATION_PLAN.md:443-458
Key Features
- Encryption at Rest: Per-document encryption using algorithms like AES-256-GCM or XChaCha20-Poly1305.
- Digital Signatures: RSA-2048 signing of documents to detect tampering.
- Access Control Lists (ACLs): Leveraging file-level ACLs and implementing Role-Based Access Control (RBAC).
The EncryptionManager will be a core component, handling all cryptographic operations.
pub struct EncryptionManager {
master_key: SecretKey,
key_derivation: KDF,
}
pub enum EncryptionAlgorithm {
AES256GCM,
ChaCha20Poly1305,
}
Sources: IMPLEMENTATION_PLAN.md:244-254
Phase 5: Replication & Distribution
This phase enables high availability and disaster recovery through data replication.
Sources: README.md:134, 150-151, IMPLEMENTATION_PLAN.md:460-475
Key Features
- Git-Based Replication: Using Git for versioning, history, and efficient push/pull synchronization between nodes.
- Multi-Node Sync: Peer discovery and state reconciliation using a gossip protocol.
- Distributed Consensus: An optional Raft-based implementation for strong consistency in a distributed cluster.
The Git-based approach is a unique feature, leveraging a well-understood tool for database replication.
This diagram shows the data flow for Git-based replication between a primary and secondary node.
Sources: README.md:180-187, IMPLEMENTATION_PLAN.md:300-318
Phase 6: Compliance & Audit
This phase builds features specifically for regulatory compliance and generating audit trails.
Sources: README.md:144-146, IMPLEMENTATION_PLAN.md:477-491
Key Features
- Immutable Audit Trails: A dedicated, append-only collection for logging all significant events.
- Compliance Reporting: Automated generation of reports for standards like GDPR, SOC 2, and HIPAA.
- Data Governance: Tools for data classification, retention policies, and tracking deletions.
An AuditTrail component will be responsible for logging events and generating reports.
Sources: IMPLEMENTATION_PLAN.md:320-349
Phase 7: Advanced & Game-Changing Features
The final planned phase introduces cutting-edge features to solidify Sentinel's position as a leader in trusted data storage.
Sources: README.md:149-155, IMPLEMENTATION_PLAN.md:493-511
Feature Matrix Summary
| Feature | Phase | Priority | Complexity |
|---|---|---|---|
| Basic CRUD | 1 | P0 | Low |
| WAL & Transactions | 2 | P0 | High |
| File Locking | 2 | P0 | Medium |
| LRU Caching | 3 | P1 | Medium |
| Hash Indices | 3 | P1 | Medium |
| AES-256 Encryption | 4 | P1 | Medium |
| Git Replication | 5 | P1 | High |
| Audit Trails | 6 | P1 | Medium |
| Full-Text Search | 7 | P2 | High |
| Merkle Tree Verification | 7 | P3 | High |
| MVCC | 7 | P3 | Very High |
| Temporal Queries | 4 | - | - |
This table summarizes the priority and complexity of key features across the roadmap.
Sources: IMPLEMENTATION_PLAN.md:513-527, README.md:155
Key Features
- Merkle Tree Verification: To cryptographically verify the integrity of an entire collection.
- Content-Addressable Storage: Storing documents based on their content hash, similar to Git, for automatic deduplication.
- Temporal Queries: Allowing queries against the state of the database at a specific point in time.
- Zero-Knowledge Proofs: Future capabilities for proving facts about data without revealing the data itself.
Conclusion
The Sentinel roadmap is an ambitious plan to build a new kind of database where trust, transparency, and auditability are the primary design goals. By progressing through these seven phases, the project will deliver a powerful tool for organizations managing sensitive security and compliance data, from simple document storage to a fully distributed, cryptographically verifiable system. The open-source nature of the project and its reliance on standard tools and formats ensure zero lock-in and encourage community contribution.
Sources: README.md:194-201, CODE_OF_CONDUCT.md, Cargo.toml