Migration Guide
Relevant source files
The following files were used as context for generating this wiki page:
The EdgeVec Migration Guide provides technical instructions and architectural context for transitioning between data format versions and migrating from external vector search libraries. As an embedded, WASM-native vector database, EdgeVec prioritizes backward compatibility and transparent data migration to ensure stability for browser-based applications.
This guide specifically covers the internal transition from format version v0.3 to v0.4, which introduced native metadata support, and provides high-level context for users moving from server-side libraries like hnswlib or voy.
Sources: tests/migration_v03_v04.rs:1-6, README.md:16-30
Version Migration: v0.3 to v0.4
The transition from v0.3 to v0.4 is a critical upgrade path that enables metadata filtering. EdgeVec handles this migration transparently during the loading process.
Backward Compatibility Mechanism
The v0.4 reader is designed to handle older snapshots that lack metadata. When a v0.3 file is detected, the system loads the vector data and HNSW graph while initializing an empty metadata store.
The following flowchart illustrates how EdgeVec handles different format versions during a read operation:
The system distinguishes versions using the version_minor field in the file header and utilizes a HAS_METADATA flag in the snapshot header to determine if auxiliary metadata blocks should be processed.
Sources: tests/migration_v03_v04.rs:17-100, GATE_1_COMPLETE.md:100-115
Migration Workflow
To fully migrate a v0.3 index to v0.4 capabilities, the index must be loaded and then saved using the new writer. This process updates the file header and sets the appropriate feature flags.
- Load: Call
read_snapshoton the legacy data. - Enrich (Optional): Add metadata to existing vectors using
insert_with_metadata. - Persist: Call
write_snapshotto save the index in thev0.4format.
Sources: tests/migration_v03_v04.rs:108-150
Format Invariants and Validation
Migrating data requires strict adherence to memory layouts and structural invariants to prevent corruption. EdgeVec uses compile-time verification to ensure that headers remain consistent across versions.
Header Structure Invariants
The system enforces a 64-byte FileHeader and a 24-byte SectionHeader. During migration, the CRC32 checksum must be recalculated if the header fields (such as version numbers) are manually modified for testing or recovery purposes.
| Header Component | Offset | Size | Requirement |
|---|---|---|---|
version_minor | 5 | 1 byte | Must match 3 or 4 |
flags | 6 | 2 bytes | HAS_METADATA (0x01) |
CRC32 | 44 | 4 bytes | Checksum of first 64 bytes |
Sources: GATE_1_COMPLETE.md:25-35, tests/migration_v03_v04.rs:77-85, GATE_2_BLOCKED.md:20-25
Data Layout Logic
The transition between versions involves navigating the following storage layers:
The migration to v0.4 specifically adds the "Metadata HashMap" component to the snapshot format, while preserving the "Contiguous Vector Arena" layout for SIMD compatibility.
Sources: GATE_1_COMPLETE.md:85-98, tests/migration_v03_v04.rs:240-265
Migrating from External Libraries
Users moving from hnswlib or voy to EdgeVec must adjust their implementation to account for EdgeVec's WASM-native features like Binary Quantization (BQ) and integrated metadata filtering.
Feature Comparison for Migration
| Feature | External (hnswlib/voy) | EdgeVec (v0.6+) |
|---|---|---|
| Quantization | Scalar or None | Binary (32x reduction) |
| Filtering | Manual / Post-process | Native SQL-like Expressions |
| Persistence | File System | IndexedDB (Browser) / File |
| Deletion | Rebuild required | O(1) Soft Delete + Compaction |
Sources: README.md:16-30, README.md:175-185
API Transition Example
When migrating, replace manual post-filtering logic with EdgeVec's internal filtered search to reduce JS-WASM boundary overhead.
// Legacy approach (Manual filtering)
const results = index.search(query, 100);
const filtered = results.filter(r => r.metadata.category === 'books');
// EdgeVec approach (Native migration)
const results = db.searchWithFilter(query, 'category = "books"', 10);
Sources: README.md:140-155, CONTRIBUTING.md:45-60
Conclusion
Migration within EdgeVec is governed by the "Nvidia Grade" protocol, ensuring that format changes are always accompanied by backward compatibility tests. The transition from v0.3 to v0.4 established the foundation for metadata-rich indices, while the current architecture supports seamless integration of newer features like SIMD acceleration and Binary Quantization without breaking legacy snapshots.