Scaling Sentinel

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• IMPLEMENTATION_PLAN.md
• Cargo.toml
• crates/sentinel/Cargo.toml
• crates/sentinel/src/lib.rs
• CODE_OF_CONDUCT.md
• rustfmt.toml

Cyberpath Sentinel is designed with a primary focus on trust, transparency, and compliance, rather than raw speed or high-throughput transactional workloads. However, the architecture includes a clear roadmap for scaling within its intended use cases, such as audit logs, configuration management, and compliance data storage. Scaling in Sentinel is not about competing with high-frequency trading platforms, but about overcoming inherent filesystem limitations and improving performance for large collections of documents.

The scaling strategy addresses three main areas: concurrency for multi-writer scenarios, query performance for large datasets, and storage limits to handle millions of files. The solutions involve a phased implementation of features like Write-Ahead Logging (WAL), in-memory caching, lazy indexing, and distributed replication.

Sources: README.md:12, README.md:23-25, IMPLEMENTATION_PLAN.md:13-17, IMPLEMENTATION_PLAN.md:323-326

Key Scaling Challenges and Solutions

Sentinel identifies several trade-offs and proposes specific solutions to mitigate them. These are critical for scaling the system beyond basic use.

Challenge	Description	Proposed Solution
Concurrency Complexity	Multi-writer scenarios can lead to data corruption or lost updates without proper controls.	Implement file locking, Write-Ahead Logging (WAL), and eventually Multi-Version Concurrency Control (MVCC).
Query Performance	Queries scan all files in a collection, which is slow for large collections. The system has no native B-tree indices initially.	Introduce in-memory caching (LRU), lazy indexing (hash-based), and full-text search capabilities.
Scaling Limits	A single folder's performance degrades significantly around 4 million files.	Employ hash-based sharding to distribute documents across multiple sub-folders.
Network Replication	Syncing large document stores across a network can be slow and error-prone.	Leverage Git-based replication for efficient delta syncing and version history. Plan for Raft-based consensus for distributed correctness.

Sources: README.md:62-73, IMPLEMENTATION_PLAN.md:320-366

Concurrency Scaling

Handling concurrent operations, especially writes, is a primary scaling challenge. The implementation plan details a multi-stage approach to ensure data integrity.

Sources: IMPLEMENTATION_PLAN.md:320-333

File Locking

The first line of defense against concurrent write issues is file locking. This strategy serializes writes to ensure that only one process can modify a file at a time.

• Exclusive Locks: Used for write operations to prevent any other read or write access.
• Shared Locks: Used for read operations, allowing multiple readers simultaneously but no writers.

The fs2 crate is planned for cross-platform file locking.

Sources: IMPLEMENTATION_PLAN.md:126-139, IMPLEMENTATION_PLAN.md:323-328, IMPLEMENTATION_PLAN.md:400

The diagram below shows a sequence for a locked write operation.

Write-Ahead Logging (WAL)

For enhanced durability and crash recovery, a Write-Ahead Log (WAL) is planned. All operations (Insert, Update, Delete) are first written to a log file before being applied to the actual document files. This ensures that in the event of a crash, the system can recover to a consistent state by replaying the log.

• Durability: Ensures that once a transaction is committed to the log, it is not lost.
• Recovery: On startup, the system can check the WAL and complete any unfinished operations.
• Checkpointing: A mechanism to periodically apply log entries to the main data files and truncate the log.

Sources: README.md:64, IMPLEMENTATION_PLAN.md:109-124, IMPLEMENTATION_PLAN.md:244-251

Multi-Version Concurrency Control (MVCC)

As a long-term, high-performance solution, MVCC is planned. This allows readers to access a consistent snapshot of the data without being blocked by writers. Writers create new versions of documents instead of modifying them in-place.

• Planned for Phase 3 (Enterprise) / Phase 7 (Advanced).
• Avoids the need for read locks, improving concurrency.
• High implementation complexity.

Sources: README.md:100, IMPLEMENTATION_PLAN.md:301, IMPLEMENTATION_PLAN.md:330-333

Query Performance Scaling

Without traditional database indexes, query performance is a known bottleneck. The strategy to address this involves caching, lazy indexing, and sharding.

Sources: README.md:68-69, IMPLEMENTATION_PLAN.md:335-353

Caching Layer

An in-memory LRU (Least Recently Used) cache is planned to store frequently accessed documents. This avoids repeated disk I/O for "hot" data.

• Component: Cache<K, V> struct.
• Policy: LRU eviction.
• Configuration: Cache size is configurable (cache_size_mb).
• Implementation: Planned for Phase 3, using libraries like lru and dashmap.

Sources: IMPLEMENTATION_PLAN.md:81, IMPLEMENTATION_PLAN.md:141-154, IMPLEMENTATION_PLAN.md:265-271

Lazy Indexing

To speed up queries without the overhead of maintaining indexes for all fields, Sentinel will use lazy indexing. An index for a field is only created the first time a query filters on that field.

• Index Types: Hash indices for O(1) exact matches and B-tree indices for O(log n) range queries are planned.
• Storage: Indices will be stored in a .index/ sub-directory within each collection.
• Maintenance: Indexes are automatically updated on subsequent write operations.

Sources: README.md:180, IMPLEMENTATION_PLAN.md:54, IMPLEMENTATION_PLAN.md:156-176, IMPLEMENTATION_PLAN.md:338-343

The following flowchart illustrates the lazy indexing process.

Storage and Replication Scaling

Sentinel's "filesystem is the database" principle faces a major hurdle with filesystem performance limits, particularly the number of files in a single directory.

Sources: README.md:88, IMPLEMENTATION_PLAN.md:79

Sharding Strategy

To overcome the performance degradation in folders with millions of files, hash-based sharding is the proposed solution. Documents will be distributed across a fixed number of sub-directories based on a hash of their ID.

• Default Limit: Performance degrades around 4 million files per folder.
• Solution: Distribute documents into sub-folders (e.g., 256 shards). This reduces the number of files to scan in any single directory.
• Implementation: Planned for Phase 3.

Sources: README.md:71, README.md:179, IMPLEMENTATION_PLAN.md:283-289, IMPLEMENTATION_PLAN.md:340-343

This diagram shows how documents would be distributed into shards.

Replication and Distribution

For high availability and disaster recovery, Sentinel plans for robust replication mechanisms.

Git-Based Replication

The primary method for replication is integrating directly with Git. Every data change can be an automatic commit, and replication is handled via git push and git pull.

• Mechanism: Use the git2 library for native Git operations.
• Strategy: Can be configured for push-pull, push-only, or pull-only sync.
• Benefit: Provides a complete, auditable history of all changes and leverages Git's efficient packfile format for network transfer.

Sources: README.md:191-198, IMPLEMENTATION_PLAN.md:208-226, IMPLEMENTATION_PLAN.md:291-297

Distributed Consensus

For more demanding scenarios requiring stronger consistency guarantees, a Raft-based consensus model is considered as an optional, advanced feature.

• Technology: raft-rs library.
• Use Case: Multi-node clusters requiring automated leader election and log replication.
• Implementation: Planned for Phase 5, marked as optional and high complexity.

Sources: README.md:104, IMPLEMENTATION_PLAN.md:214, IMPLEMENTATION_PLAN.md:299-305, IMPLEMENTATION_PLAN.md:410

Conclusion

Sentinel's approach to scaling is pragmatic and tailored to its core mission of providing a transparent and auditable data store. Instead of pursuing raw performance at all costs, it focuses on solving specific bottlenecks related to its filesystem-based architecture. Through a phased implementation of caching, lazy indexing, sharding, and robust replication strategies like Git-sync, Sentinel aims to provide a scalable solution for security, compliance, and configuration data management, where trust is more valuable than throughput.

Sources: README.md:12-13, README.md:223-228, IMPLEMENTATION_PLAN.md:13-17