Scalar Querying
Relevant source files
The following files were used as context for generating this wiki page:
Scalar Querying in the Milvus Rust SDK refers to the process of retrieving data from collections based on boolean expressions and specific primary keys, rather than vector similarity. This feature allows for precise data retrieval using filtering logic (e.g., age > 20) and is fundamental for CRUD operations and metadata management.
The system supports two primary modes of scalar retrieval: standard queries using filtering expressions and point lookups using specific IDs. Furthermore, the SDK provides an iterator pattern for processing large result sets that exceed standard gRPC message limits.
Sources: src/query.rs:1-15, examples/query_search.rs:37-45
Core Query Mechanisms
The SDK implements scalar querying primarily through the query and get methods on the Client struct. These methods translate high-level Rust options into gRPC QueryRequest messages defined in the Milvus protocol.
Boolean Expression Filtering
The query method evaluates a boolean expression against the scalar fields in a collection. It is used for filtering data based on logical conditions.
Sources: src/query.rs:604-620, examples/index_example.rs:99-105
Primary Key Retrieval
The get method is a specialized version of a query that retrieves entities by their primary keys. It supports both Int64 and VarChar ID types.
Sources: src/query.rs:806-815, examples/query_search.rs:81-91
The diagram shows the logic flow from high-level client calls to the underlying gRPC request construction.
Sources: src/query.rs:624-651, src/query.rs:820-845
Configuration and Options
Queries are configured using QueryOptions (also aliased as GetOptions), which control consistency, pagination, and output formatting.
QueryOptions Parameters
| Parameter | Type | Description |
|---|---|---|
output_fields | Vec<String> | Specific fields to return. If empty, all fields may be returned. |
partition_names | Vec<String> | Optional list of partitions to restrict the query. |
limit | i64 | Maximum number of records to return. |
offset | i64 | Number of records to skip for pagination. |
consistency_level | i32 | Consistency guarantee (Strong, Bounded, Eventually, Session). |
expr_template_values | HashMap | Values for parameterized query templates. |
Sources: src/query.rs:260-268, src/query.rs:434-510
Consistency Levels
The SDK supports four consistency levels that determine how fresh the queried data must be:
- Strong: Uses
STRONG_TIMESTAMP(0), ensuring the query sees all preceding writes. - Bounded: Uses
BOUNDED_TIMESTAMP(2), providing staleness guarantees. - Eventually: Uses
EVENTUALLY_TIMESTAMP(1). - Session: Uses the timestamp from the client's collection cache.
Sources: src/query.rs:40-60, src/query.rs:586-602
Iterative Querying
For large datasets, the SDK provides QueryIterator, which allows for paginated streaming of results. This prevents gRPC timeout errors and memory exhaustion when retrieving millions of entities.
QueryIterator Workflow
The sequence diagram illustrates how the iterator maintains state via session_ts and primary key tracking to provide seamless pagination.
Sources: src/iterator.rs:253-272, src/iterator.rs:563-630
Iterator Checkpointing
The QueryIterator can optionally save its progress to a file using the iterator_cp_file option. This allows a process to resume querying from the last successful primary key if interrupted.
Sources: src/iterator.rs:412-440
Implementation Details
Result Processing
Query results are returned as a Vec<FieldColumn>. Each FieldColumn contains metadata (name, field type) and a ValueVec containing the actual data for that column across all returned rows.
Sources: src/query.rs:653, src/proto/milvus.proto.milvus.rs:689-700
Primary Key Packing
When performing a get operation, the SDK must translate a list of IDs into a Milvus-compatible boolean expression. For Int64 keys, it produces an in expression; for VarChar, it quotes the strings appropriately.
Sources: src/query.rs:777-804
// Example: Packing primary keys (src/query.rs:777-804)
fn pack_pks_expr(&self, collection: &Collection, pks: Vec<String>) -> Result<String> {
let primary_field = self.extract_primary_field(collection)?;
let pk_field_name = primary_field.name.clone();
let data_type = primary_field.dtype;
if data_type == DataType::VarChar {
let ids: Vec<String> = pks.iter().map(|entry| format!("'{}'", entry)).collect();
let expr = format!("{pk_field_name} in {:?}", ids);
Ok(expr)
} else {
// ... handling for i64 ...
let expr = format!("{pk_field_name} in {:?}", ids);
Ok(expr)
}
}
Summary
Scalar querying provides the backbone for structured data retrieval within the Milvus Rust SDK. Through the combination of boolean expressions, primary key lookups, and the robust QueryIterator pattern, developers can efficiently manage metadata and perform precise filtering alongside high-performance vector operations.
Sources: src/query.rs:1-12, src/iterator.rs:222-235