Transactions
Relevant source files
The following files were used as context for generating this wiki page:
Transaction support is a planned core feature of Cyberpath Sentinel, designed to provide atomic operations for data manipulation. The primary goal is to ensure durability, crash recovery, and data integrity, which are critical for the system's focus on auditability and compliance. This functionality is part of the Phase 2 implementation plan, focusing on Durability & Concurrency.
The transaction system is centered around a Transaction Manager that utilizes Write-Ahead Logging (WAL) to guarantee that operations are either fully completed or not at all, even in the event of a system failure. This ensures that the database can be restored to a consistent state.
Sources: README.md:83, IMPLEMENTATION_PLAN.md:21, 313-316
Architecture and Components
The Transaction Manager is a central component in the Sentinel architecture, sitting between the client-facing API (Query Engine, Caching Layer) and the low-level File I/O layer. It works in concert with the Concurrency Control layer to manage data consistency.
Sources: IMPLEMENTATION_PLAN.md:31-45
The following diagram illustrates the position of the Transaction Manager within the system.
Key Components
| Component | Description |
|---|---|
| Transaction Manager | Orchestrates transactions, ensuring ACID properties. It is a planned core component. |
| Write-Ahead Log (WAL) | Provides durability by logging all changes before they are written to the main data files. This is the foundation for crash recovery. |
| File Lock Manager | Manages concurrent access to files, preventing race conditions and data corruption during multi-writer scenarios. |
| Collection | The Collection struct will expose a transaction method to allow grouping multiple operations atomically. |
Sources: IMPLEMENTATION_PLAN.md:34, IMPLEMENTATION_PLAN.md:143-144, IMPLEMENTATION_PLAN.md:170-173, IMPLEMENTATION_PLAN.md:313-316
Write-Ahead Logging (WAL)
The WAL is the cornerstone of Sentinel's durability and recovery strategy. Before any data modification (insert, update, delete) is applied to the actual document files, a corresponding entry is written to the transaction log.
Sources: IMPLEMENTATION_PLAN.md:141-143, IMPLEMENTATION_PLAN.md:321-324
WAL Process Flow
This sequence diagram shows how a transactional write is processed using the WAL.
Log Structure and Entries
The TransactionLog is designed to manage log files and the entries within them. Each entry represents a single step in a transaction.
Sources: IMPLEMENTATION_PLAN.md:145-149
Sources: IMPLEMENTATION_PLAN.md:145-162
The different types of log entries are detailed below:
| Log Entry | Description |
|---|---|
Begin | Marks the start of a new transaction. |
Insert | Records the data for a new document insertion. |
Update | Records the data for a document update. |
Delete | Records the ID of a document to be deleted. |
Commit | Marks the successful completion of a transaction. Once committed, the changes are considered durable. |
Rollback | Marks a transaction that was intentionally aborted. |
Sources: IMPLEMENTATION_PLAN.md:150-156
For efficiency, log entries are planned to be serialized using bincode and include a CRC32 checksum for integrity verification.
Sources: IMPLEMENTATION_PLAN.md:164-165, IMPLEMENTATION_PLAN.md:325-326
Recovery and Checkpointing
- Recovery: On startup, the
recover()method will be called to replay the WAL. It will re-apply changes from committed transactions that were not yet written to disk and roll back any uncommitted transactions. - Checkpointing: The
checkpoint()process will periodically flush all committed changes from the log to the main data files and then truncate the log. This prevents the log file from growing indefinitely. The default checkpoint interval is planned to be 5 minutes.
Sources: IMPLEMENTATION_PLAN.md:73, IMPLEMENTATION_PLAN.md:161-162, IMPLEMENTATION_PLAN.md:321-324
Concurrency Control and Locking
To handle multi-writer scenarios, Sentinel will implement a FileLockManager. The initial strategy is to use file-level locking to serialize writes.
Sources: README.md:52, IMPLEMENTATION_PLAN.md:168-170, IMPLEMENTATION_PLAN.md:353-356
The FileLockManager will support different locking strategies:
| Lock Strategy | Description |
|---|---|
Exclusive | A single-writer, no-readers lock. Used for write operations. |
Shared | Allows multiple readers but no writers. Used for read operations. |
OptimisticMVCC | A planned long-term solution for Multi-Version Concurrency Control, allowing for higher concurrency. |
Sources: IMPLEMENTATION_PLAN.md:174-178, IMPLEMENTATION_PLAN.md:362-367
The implementation will use the fs2 crate for cross-platform file locking.
Sources: IMPLEMENTATION_PLAN.md:185, IMPLEMENTATION_PLAN.md:471
Transactional API
The Collection struct will provide a transaction method to execute a series of operations atomically. This is part of the Phase 2 implementation plan.
// Planned API structure
impl Collection {
pub async fn transaction<F>(&self, operations: F) -> io::Result<()>
}
Sources: IMPLEMENTATION_PLAN.md:139, IMPLEMENTATION_PLAN.md:335-338
This API will ensure that all operations within the closure are treated as a single unit, achieving the ACID properties planned for Phase 2:
- Atomicity: Ensured by the WAL's commit/rollback mechanism.
- Consistency: The responsibility of the application logic within the transaction.
- Isolation: Initially provided by file locks.
- Durability: Guaranteed by writing to the WAL before acknowledging a commit.
Sources: IMPLEMENTATION_PLAN.md:335-338
Summary
Transactions are a critical, planned feature for Cyberpath Sentinel, forming the backbone of its promise of durability and data integrity. By leveraging a Write-Ahead Log (WAL) and file-level locking, the system aims to provide atomic operations essential for its target use cases in audit, security, and compliance. The implementation is scheduled for Phase 2 and is foundational for later features like MVCC and high-availability replication.
Sources: README.md:83, IMPLEMENTATION_PLAN.md:313-341