Backup & Recovery

Relevant source files

The following files were used as context for generating this wiki page:

• src/backup.rs
• src/crash.rs
• src/failure.rs
• src/main.rs
• src/setup.rs
• src/raw.rs

The Backup & Recovery system provides durability and restoration for the raw.db and knowledge.db stores. It exports database records as sealed, checksummed, and compressed segments. When corruption occurs, the system automatically quarantines damaged files and rebuilds the database from backup segments to ensure data integrity and persistence across system crashes or power losses.

Sources: src/backup.rs:1-12, src/main.rs:114-116

Backup Architecture

Oboete backs up raw.db records and operations (ops) as independent segments. This separation ensures that operations can be recovered even if they were appended when no new records were present. Backups occur periodically during worker execution and upon idle exit.

Backup Segments

Segments are stored as zstd compressed files with associated .sha256 checksum files. Each segment represents a range of sequence numbers for a specific device.

The diagram shows the flow of data from the raw database into typed, compressed, and hashed segment files within the backup directory.
Sources: src/backup.rs:40-60, src/backup.rs:149-155

Backup Configuration

Backup behavior is controlled via config.toml. If no directory is specified, the system defaults to a backups folder within the home directory.

Parameter	Type	Default Value	Description
dir	PathBuf	<home>/backups	Location where backup segments are stored.
SEGMENT_BYTES	usize	8 MB	Maximum uncompressed record bytes per segment.
EVERY	Duration	30 minutes	Interval at which running workers perform backups.

Sources: src/backup.rs:15-38

Corruption Detection & Quarantine

Oboete monitors database health during file operations. If a file is identified as damaged—indicated by SQLite corruption codes or failed integrity checks—the system initiates a quarantine process.

Quarantine Logic

1. Identity Sidecars: The system identifies SQLite sidecar files (-wal, -shm).
2. Rename: It renames the main database file and its sidecars with a .quarantined-<timestamp> suffix.
3. Isolation: Renaming ensures that subsequent SQLite attempts do not accidentally replay an old Write-Ahead Log (WAL) into a new database file.

Sources: src/backup.rs:211-231, src/backup.rs:370-388

Health Checks

The system performs several tiers of checks to identify issues:

• Quick Check: Performed during database opening to catch obvious header or structure corruption.
• Integrity Check: A full read of every database page, typically triggered manually via oboete doctor.
• Verification: Backup segments are verified by comparing their current SHA256 hash against their .sha256 sidecar file.

Sources: src/backup.rs:188-210, src/backup.rs:356-368, src/setup.rs:1040-1060

Recovery & Restoration

Recovery is the process of rebuilding a functional database from backup segments or other sources.

Rebuilding raw.db

The restore function automates the reconstruction of raw.db.

The sequence diagram illustrates the automated workflow for detecting corruption, isolating damaged files, and rebuilding the database from backup segments.
Sources: src/backup.rs:251-330, src/backup.rs:390-410

Rebuilding knowledge.db

Unlike raw.db, knowledge.db contains derived data (claims, digests, indexes). If it is damaged, it is quarantined and restarted as an empty store. Every consumer then rebuilds its state by scanning raw.db from sequence 0.

Sources: src/backup.rs:414-432, src/main.rs:723-727

Failure Management

The system tracks persistent failures to prevent recursive error loops and alert the user.

Recording Failures

If the system cannot write to the store (e.g., due to a full disk or permission error), it records the failure in the data directory.

• Marker File: A recording-failed file is created.
• Metadata: The file contains the error message and the timestamp of the failure.
• User Alert: Commands like oboete inject and oboete doctor check for this marker to notify the user.

Sources: src/failure.rs:10-40, src/setup.rs:1015-1025

Restore Requests

If a hook finds raw.db damaged while a worker holds the lock, it creates a restore-wanted file. The worker checks for this file upon its next cycle to initiate the restoration.

Sources: src/backup.rs:341-354

Summary

The Backup & Recovery system ensures Oboete's reliability by maintaining compressed, hashed segments of all records and operations. It features automated corruption detection that isolates damaged databases through quarantine and recovers state via segment-based rebuilding. For derived data, the system prioritizes consistency by allowing consumers to re-process the raw history. This architecture protects against data loss from hardware failure, software crashes, and unexpected power interruptions.

Sources: src/backup.rs:1-12, src/backup.rs:251-260, src/failure.rs:1-5