Guide: Backup and Recovery
Relevant source files
The following files were used as context for generating this wiki page:Cyberpath Sentinel's approach to backup and recovery is rooted in its core design principle: "Filesystem is the Database". This philosophy enables the use of standard, battle-tested UNIX tools for simple and transparent data management. The system is designed for operational simplicity, where backups can be as straightforward as creating a tarball, and disaster recovery can be achieved with a git clone command.
Sources: README.md:58-63, 138
This guide covers the primary mechanisms for backup, recovery, and data durability as outlined in the project's documentation. It includes both currently supported methods using standard tools and planned features like the integrated backup utility and Write-Ahead Logging (WAL) for crash recovery. For more details on the overall architecture, see the Implementation Plan.
Sources: README.md:201, IMPLEMENTATION_PLAN.md:162-174
Core Principles
Sentinel's backup strategy leverages its file-based architecture, providing several distinct advantages:
- Operational Simplicity: Standard UNIX tools like
rsyncandtarare the primary instruments for replication and backups. - Auditability: When using Git for replication, the entire history of the database is versioned, providing a transparent audit trail.
- Zero Lock-In: Data is stored in plain JSON/BSON, allowing backups to be easily inspected or migrated to other systems.
Sources: README.md:59-63, 78-81
Backup Mechanisms
Sentinel provides multiple strategies for creating backups, ranging from manual filesystem archives to integrated, automated solutions.
Filesystem-Level Backups (tar)
The most direct method for backing up a Sentinel store is to use standard filesystem archiving tools. Since all data is contained within a directory structure, a tool like tar can create a complete, point-in-time snapshot of the database.
Sources: README.md:60
The following diagram illustrates the tar backup process.
Sources: README.md:60
Git-Based Replication and Recovery
For continuous protection and simplified disaster recovery, Sentinel is designed to integrate with Git. Each change can be committed, and the entire data store can be managed as a Git repository. Disaster recovery is as simple as cloning the repository to a new machine.
Sources: README.md:62-63, 194-200, IMPLEMENTATION_PLAN.md:298-315
This diagram shows the disaster recovery flow using git clone.
Sources: README.md:62-63, 194-200
Integrated Backup Utility
An integrated backup utility is planned, which will provide streamlined commands for managing backups, including options for cloud storage integration and encryption.
Sources: README.md:114, 201-203
Planned Store API
The core Store struct is planned to include backup and restore methods to handle these operations programmatically.
This diagram shows the planned Store struct with its backup and restore methods.
Sources: IMPLEMENTATION_PLAN.md:78-91
Planned S3 Backup Command
The roadmap includes a command-line interface for backing up the data store directly to an encrypted cloud storage bucket, such as Amazon S3.
cyberpath-sentinel backup --path /data --s3-bucket compliance-backups --encryption AES256
Sources: README.md:202-203
The sequence diagram below illustrates the intended flow for this operation.
Sources: README.md:201-203, IMPLEMENTATION_PLAN.md:90
Recovery and Durability
Recovery mechanisms are designed to handle everything from accidental deletion to catastrophic system failure.
Disaster Recovery
As mentioned, the primary method for disaster recovery is to restore from a backup.
- Using
git:git clonethe remote repository to a new system. - Using
tar: Extract the archive to the desired data directory location. - Using
Store::restore(): A planned programmatic method to restore from a backup source.
Sources: README.md:62-63, IMPLEMENTATION_PLAN.md:91
Crash Recovery (Write-Ahead Logging)
To ensure durability and protect against data corruption from partial writes or power failures, a Write-Ahead Logging (WAL) system is a planned core feature. All operations (Insert, Update, Delete) are first written to a log file before being applied to the main data files. In the event of a crash, Sentinel can replay the log to bring the database back to a consistent state.
Sources: README.md:104, 118, IMPLEMENTATION_PLAN.md:162-174, 395-401
The crash recovery process is visualized below.
This flowchart shows how the Write-Ahead Log is used to recover the system state after a crash.
Sources: IMPLEMENTATION_PLAN.md:174, 398-400
Soft Deletes and Recovery
Sentinel employs a "soft delete" mechanism. When a document is deleted, it is not immediately removed from the filesystem. Instead, it is moved to a .deleted/ sub-folder within its collection. This allows for easy recovery of accidentally deleted documents.
Sources: README.md:157, 163-165, IMPLEMENTATION_PLAN.md:372
Summary of Backup & Recovery Methods
The following table summarizes the different methods available and planned for Sentinel.
| Method | Tool / Component | Use Case | Type | Status |
|---|---|---|---|---|
| Filesystem Archive | tar, zip | Full, point-in-time backup | Manual | Available |
| Replication | rsync | Hot-site replication | Continuous | Available |
| Version Control | git | Disaster recovery, audit trail | Continuous | Available |
| Integrated Backup | cyberpath-sentinel backup | Automated, encrypted cloud backups | Automated | Planned |
| Programmatic API | Store::backup(), Store::restore() | In-application backup management | Automated | Planned |
| Crash Recovery | Write-Ahead Log (WAL) | Durability, power failure safety | Automatic | Planned |
| File Recovery | .deleted/ folder | Recovering single deleted documents | Manual | Available |
Sources: README.md:60-63, 118, 157, 194, 202, IMPLEMENTATION_PLAN.md:90-91, 162, 372, 580
Conclusion
Cyberpath Sentinel's backup and recovery strategy is a direct reflection of its transparent and simple design. By leveraging the filesystem and standard tools, it offers robust and easy-to-understand procedures. The planned additions of an integrated backup utility and a Write-Ahead Log will further enhance its durability and ease of use, making it a reliable choice for storing critical, auditable data.
Sources: README.md:21-23, 58-63, IMPLEMENTATION_PLAN.md:21-26