Persistence & WAL
Relevant source files
The following files were used as context for generating this wiki page:
EdgeVec implements a robust persistence layer designed for "Nvidia Grade" correctness and durability across multiple platforms, including Browser (IndexedDB), native filesystems, and memory. The system relies on two primary mechanisms: Write-Ahead Logging (WAL) for incremental updates and crash recovery, and Atomic Snapshots for full state serialization.
Sources: GATE_1_COMPLETE.md:144-148, GATE_1_COMPLETE.md:214-217
Write-Ahead Log (WAL)
The Write-Ahead Log is the primary mechanism for ensuring data durability between snapshots. It captures sequential operations (inserts, deletes) before they are fully committed to the index, allowing the system to replay these operations to recover state after a crash.
Sources: src/persistence/wal.rs:136-139, tests/integration_storage_durability.rs:15-20
WAL Entry Structure
Each entry in the WAL is serialized using a manual little-endian format to ensure cross-platform compatibility. An entry consists of a fixed-size header, a variable-length payload, and a CRC32 checksum for integrity verification.
| Field | Size | Description |
|---|---|---|
| Sequence | 8 bytes | Incremental u64 sequence number |
| Type | 1 byte | Entry type (e.g., 0=Insert, 1=Delete) |
| Padding | 3 bytes | Explicit zero padding for alignment |
| Length | 4 bytes | u32 length of the following payload |
| Payload | Variable | The actual data (Max 16MB) |
| CRC32 | 4 bytes | Checksum of Header + Payload |
Sources: src/persistence/wal.rs:7-13, src/persistence/wal.rs:88-100, src/persistence/wal.rs:184-187
Append and Replay Logic
The WalAppender handles writing new entries to the log, while WalIterator provides a streaming interface to read and validate entries during recovery.
The diagram shows the sequential recovery flow where each entry is validated against its CRC before being applied to the in-memory storage.
Sources: src/persistence/wal.rs:82-134, tests/integration_storage_durability.rs:27-30
Atomic Snapshots
Snapshots represent a point-in-time serialization of the entire HnswIndex and VectorStorage. These are used to provide fast loading of large datasets without replaying extensive WAL logs.
Snapshot Workflow
- Creation: The current index state is serialized and written to a
StorageBackend. - Persistence: Snapshots are designed to be atomic; in a filesystem context, this typically involves writing to a temporary file and renaming it.
- Recovery: During startup, the system first loads the most recent snapshot and then replays any WAL entries created after the snapshot timestamp.
Sources: GATE_1_COMPLETE.md:144-148, examples/filter_persistence.rs:141-144, examples/filter_persistence.rs:147-151
Storage Backends
EdgeVec abstracts I/O operations through a StorageBackend trait, enabling the same persistence logic to function across different environments.
- FileBackend: Used for native OS filesystem persistence.
- IndexedDB (WASM): Provides persistent storage within web browsers.
- MemoryBackend: A volatile implementation used primarily for unit testing and transient caches.
Sources: GATE_1_COMPLETE.md:144-148, tests/unit_wal_recovery.rs:10-15
Integrity and Error Handling
Integrity is enforced through strict validation rules during both writing and reading phases.
| Error Type | Trigger Condition |
|---|---|
ChecksumMismatch | Calculated CRC32 does not match the stored CRC in the WAL. |
PayloadTooLarge | A payload exceeds the MAX_PAYLOAD_SIZE of 16MB. |
Truncated | The file ends unexpectedly in the middle of a header or payload. |
Io | Underlying hardware or OS-level read/write failure. |
Sources: src/persistence/wal.rs:16-56, tests/proptest_wal.rs:43-77
Torn Write Protection
The system is tested extensively for "torn writes"—scenarios where a crash occurs mid-write. The WalIterator is designed to detect these partial entries as Truncated errors and allows the system to recover the valid prefix of the log while discarding the corrupted tail.
Sources: tests/proptest_wal.rs:103-157, tests/integration_storage_durability.rs:65-71
Summary
The Persistence and WAL system ensures that EdgeVec remains reliable even in the face of unexpected shutdowns. By combining sequential logging for durability and snapshots for performance, the architecture meets high-grade reliability standards while remaining platform-agnostic through abstract storage backends.
Sources: GATE_1_COMPLETE.md:214-217, GATE_2_BLOCKED.md:80-86