Metadata Filtering
Relevant source files
The following files were used as context for generating this wiki page:
Metadata filtering in EdgeVec allows users to combine high-performance vector similarity search with SQL-like structured queries. This system enables precise retrieval by restricting search results to vectors that satisfy specific metadata criteria, such as categories, price ranges, or boolean flags.
The system is designed to be "WASM-first," providing a type-safe metadata storage engine in Rust that is exposed via high-level TypeScript and JavaScript APIs. It supports complex boolean logic (AND, OR, NOT), various comparison operators, and multiple execution strategies to optimize performance based on query selectivity.
Sources: src/wasm/mod.rs:1351-1355, wasm/examples/filter-playground.html:26-30
Metadata Storage Engine
EdgeVec implements a strongly typed metadata storage system that attaches key-value pairs to unique vector IDs. This system is centralized in the MetadataStore, which manages CRUD operations for metadata associated with the HNSW index.
Supported Data Types
The engine supports five core value types, allowing for a wide range of application metadata:
| Type | Rust Implementation | JS/JSON Representation | Typical Use Case |
|---|---|---|---|
| String | String | string | Titles, descriptions, categories |
| Integer | i64 | number | Timestamps, counts, discrete IDs |
| Float | f64 | number | Scores, weights, prices |
| Boolean | bool | boolean | Status flags (active, verified) |
| StringArray | Vec<String> | string[] | Tags, multi-category labels |
Sources: src/metadata/mod.rs:43-50, src/wasm/mod.rs:1505-1510
System Invariants and Limits
To prevent memory bloat in browser environments, the metadata system enforces strict validation limits:
- Keys per Vector: Maximum of 64 unique keys.
- Key Length: Maximum of 256 bytes per key.
- String Values: Maximum size of 64KB per string.
- Array Size: Maximum of 1,024 elements per
StringArray.
Sources: src/metadata/mod.rs:56-62
Filter Execution Workflow
The filtering process involves parsing a query string into an Abstract Syntax Tree (AST), which is then evaluated against the metadata associated with candidate vectors found during the HNSW search.
The diagram shows the transition from a raw filter string to an executable search strategy.
Sources: src/wasm/mod.rs:1375-1390, wasm/examples/filter-playground.html:1120-1135
Filter Strategies
EdgeVec supports multiple strategies to balance search speed and recall accuracy:
- Pre-Filtering: Evaluates the filter against the entire metadata store before performing the vector search. This ensures 100% recall but can be slower for large datasets with low selectivity.
- Post-Filtering: Performs the vector search first (retrieving more than $k$ results) and then filters the results. This is extremely fast but may return fewer than $k$ results if the filter is highly selective.
- Hybrid: A middle-ground approach that uses oversampling factors to improve the chances of returning $k$ valid results while maintaining high performance.
- Auto: The default strategy where the engine attempts to select the best method based on observed selectivity.
Sources: src/wasm/mod.rs:1382-1395, pkg/edgevec-wrapper.ts:20-30
Search Syntax and Operators
EdgeVec provides a rich set of operators for building complex filter expressions. These can be used directly as strings or via a programmatic FilterBuilder.
Operator Reference
| Operator | Description | Example |
|---|---|---|
=, != | Equality and Inequality | status = "active" |
>, <, >=, <= | Numeric Comparison | price < 100.0 |
BETWEEN | Range Selection (Inclusive) | year BETWEEN 2020 AND 2024 |
IN, NOT IN | Set Membership | category IN ["gpu", "cpu"] |
CONTAINS | Substring/Array Match | tags CONTAINS "ai" |
STARTS_WITH | Prefix Matching | sku STARTS_WITH "NV-" |
IS NULL | Nullability Check | deleted_at IS NULL |
AND, OR, NOT | Logical Composition | (a = 1 OR b = 2) AND NOT c |
Sources: wasm/examples/filter-playground.html:1050-1080, examples/filter_documents.rs:115-130
API Integration Examples
JavaScript/TypeScript Usage
In the browser environment, users interact with the EdgeVec class. The searchFiltered method accepts a query vector and a configuration object containing the filter string.
const results = await index.searchFiltered(queryVector, 10, {
filter: 'category = "electronics" AND price < 500',
strategy: 'auto',
includeMetadata: true
});
// Access results
results.results.forEach(r => {
console.log(`ID: ${r.id}, Price: ${r.metadata.price}`);
});
Sources: pkg/edgevec-wrapper.ts:250-270, wasm/examples/filter-playground.html:1210-1225
Rust (Backend/Library) Usage
For native Rust applications, the FilteredSearcher is used to coordinate between the HnswIndex and the MetadataStore.
let mut searcher = FilteredSearcher::new(&index, &storage, &metadata_store);
let filter = parse(r#"author = "Dr. Smith" AND year >= 2023"#).unwrap();
let result = searcher.search_filtered(
&query_vec,
10,
Some(&filter),
FilterStrategy::Auto
).expect("Search failed");
Sources: examples/filter_documents.rs:150-165, tests/integration_filtered_search.rs:115-125
Error Handling and Validation
Filtered searches are fallible, especially when handling user-provided strings. The system provides detailed error feedback, including the character position of syntax errors and suggested fixes.
The FilterException class in the TypeScript wrapper parses WASM-generated error JSON to provide:
- Error Code: Unique identifier for the error type (e.g., E001).
- Position: Line and column number where the parser failed.
- Suggestion: Context-aware recommendations (e.g., "Add quotes around string values").
Sources: pkg/edgevec-wrapper.ts:100-125, wasm/examples/filter-playground.html:1000-1020
Summary
The Metadata Filtering system in EdgeVec provides a powerful bridge between unstructured vector data and structured metadata. By implementing a type-safe storage engine and multiple search strategies, it ensures that developers can build complex, context-aware search features that run efficiently in both native and browser environments. The use of an AST-based parser allows for flexible SQL-like queries while maintaining the performance required for real-time similarity search applications.