Indexing Vectors and Scalars

Relevant source files

The following files were used as context for generating this wiki page:

• src/index/mod.rs
• src/index/utils.rs
• examples/index_example.rs
• src/proto/milvus.proto.milvus.rs
• examples/query_search.rs
• tests/collection.rs

Indexing in the Milvus Rust SDK is a critical operation that enables high-performance similarity searches for vector data and efficient filtering for scalar data. By creating indexes on specific fields within a collection, the system can significantly accelerate query execution compared to brute-force scans. The SDK provides a structured way to define index parameters, including index types and metric types, and manage the index lifecycle from creation to deletion.

The indexing system supports a variety of specialized algorithms for floating-point and binary vectors, such as IVF_FLAT, HNSW, and ANNOY, as well as scalar-specific indexes like Trie for VARCHAR fields. These indexes are typically created after data ingestion and must be built before a collection is loaded into memory for searching.

Sources: src/index/mod.rs, src/index/utils.rs, examples/index_example.rs

Index Architecture and Data Structures

The SDK defines several core data structures to represent index configurations and metadata. The primary structure is IndexParams, which encapsulates the configuration for a specific index.

Key Data Structures

• IndexType: An enum representing the underlying algorithm used for the index (e.g., IvfFlat, HNSW, Trie).
• MetricType: Defines how distances are calculated between vectors (e.g., L2 for Euclidean distance, IP for Inner Product).
• IndexParams: A structure containing the index name, type, metric, and a map of algorithm-specific hyperparameters.
• IndexInfo: Represents the current status and metadata of an existing index on the server.

The following class diagram illustrates the relationship between these structures:

Index Management Lifecycle

The lifecycle of an index involves creation, monitoring the build progress, describing metadata, and eventually dropping the index when no longer needed.

1. Creating an Index

The create_index function is an asynchronous operation that sends a request to the Milvus server. In the Rust SDK implementation, this function includes a polling loop that waits for the index state to reach Finished or Failed.

2. Supported Index and Metric Types

The SDK supports a wide range of indexing strategies tailored for different performance and accuracy requirements.

Category	Index Types	Metric Types
Vector	FLAT, IVF_FLAT, IVF_PQ, IVF_SQ8, HNSW, ANNOY	L2, IP, HAMMING, JACCARD
Scalar	Trie (VARCHAR), BITMAP	N/A

Sources: src/index/mod.rs:11-64, examples/index_example.rs:69-82

3. Modifying Index Properties

Indexes can be altered to update specific metadata or properties using alter_index_properties. This sends an AlterIndexRequest containing key-value pairs of properties to update or keys to delete.

Sources: src/index/utils.rs:152-179, src/proto/milvus.proto.milvus.rs:460-474

Implementation Details

Configuration Example

Creating an index requires specifying hyperparameters. For instance, an IVF_FLAT index requires nlist (number of cluster units).

let index_params = IndexParams::new(
    "embeddings_index".to_string(),
    IndexType::IvfFlat,
    MetricType::L2,
    HashMap::from([("nlist".to_string(), "32".to_string())]),
);

client.create_index(COLLECTION_NAME, "embeddings", index_params).await?;

Sources: examples/index_example.rs:62-72

Listing and Describing Indexes

The SDK provides utilities to list index names for a field or the entire collection. describe_index returns detailed metadata, including the current build state.

Method	Return Type	Description
list_indexes	Result<Vec<String>>	Returns names of all indexes on a field or collection.
describe_index	Result<Vec<IndexInfo>>	Returns full metadata for indexes on a field.

Sources: src/index/utils.rs:91-118, src/index/utils.rs:205-231

Polling Logic for Index Building

Because index building is an asynchronous server-side process, the SDK implements a robust polling mechanism in src/index/utils.rs. It repeatedly calls DescribeIndex and checks the IndexState field.

• Finished: The index is ready; the function returns success.
• Failed: The index build failed; the function returns Error::IndexBuildFailed.
• InProgress/Unissued: The function sleeps for a configured duration before the next check.

Sources: src/index/utils.rs:68-88, src/proto/milvus.proto.common.rs:275-285

Conclusion

Indexing is a prerequisite for efficient search operations in the Milvus Rust SDK. By abstracting the complex gRPC interactions into the IndexParams and providing synchronous-like behavior through automated polling, the SDK ensures that developers can reliably prepare their vector and scalar data for high-performance retrieval. Proper selection of IndexType and MetricType is essential for balancing search speed, memory usage, and accuracy.

Sources: src/index/mod.rs, src/index/utils.rs, examples/query_search.rs