Collection Management
Relevant source files
The following files were used as context for generating this wiki page:
Collection Management is the core functional area of the Milvus Rust SDK responsible for defining, creating, and maintaining the lifecycle of data collections. A collection in Milvus is a logical grouping of entities, analogous to a table in a relational database, where data is organized according to a specific schema containing both scalar and vector fields.
The SDK facilitates administrative operations such as creating schemas, managing aliases, altering collection properties, and controlling the memory state of collections through loading and releasing mechanisms. These operations interact with the Milvus server via gRPC calls defined in the underlying protocol buffers.
Lifecycle and Architecture
The lifecycle of a collection involves definition, instantiation, and operational maintenance. The Client struct acts as the primary interface for these gRPC-based interactions, while a local CollectionCache is used to store metadata for efficiency.
The flow shows the standard progression from initial schema definition to the terminal state of dropping a collection.
Sources: src/collection.rs:253-300, examples/collection.rs:21-35, tests/common/mod.rs:32-84
Data Structures
The SDK uses several key structures to represent and manage collections:
| Structure | Purpose | Source |
|---|---|---|
Collection | Represents a collection's metadata (ID, Name, Shards, Fields). | src/collection.rs:46-56 |
DescribeCollection | Response object containing detailed metadata from the server. | src/collection.rs:59-68 |
CollectionCache | Thread-safe local storage for collection metadata using DashMap. | src/collection.rs:71-75 |
CollectionSchema | Defines the structure, including fields and auto-ID settings. | src/collection.rs:258-270 |
Schema Definition and Creation
Before a collection can be created, a CollectionSchema must be built. This defines the primary key, vector fields, and scalar fields.
The relationship between the builder and field definitions for schema construction.
Sources: examples/collection.rs:24-34, tests/client.rs:69-77, src/proto/milvus.proto.milvus.rs:65-85
Key Creation Functions
create_collection: Serializes the schema into protobuf bytes and sends aCreateCollectionRequestto the Milvus server. Sources: src/collection.rs:253-277has_collection: Checks existence viaHasCollectionRequest. Sources: src/collection.rs:356-376drop_collection: Deletes the collection and all associated data viaDropCollectionRequest. Sources: src/collection.rs:288-305
Memory Management: Load and Release
Milvus requires collections to be loaded into memory (Query Nodes) before search or query operations can be performed.
The sequence demonstrates the asynchronous nature of loading, where the SDK polls the server's load state before returning.
Sources: src/collection.rs:432-470, src/proto/milvus.proto.milvus.rs:189-215
Operational Methods
- Load: The
load_collectionmethod includes a blocking loop that waits until theLoadStatereachesLoaded. Sources: src/collection.rs:458-470 - Release: The
release_collectionmethod removes the collection from Query Node memory to free resources. Sources: src/collection.rs:524-539 - Refresh:
refresh_loadupdates the loaded data to reflect recent inserts or changes. Sources: src/collection.rs:512-516
Collection Modification and Aliases
The SDK supports altering existing collections and managing aliases for abstraction.
Alteration Operations
Collections can be modified through the following methods:
alter_collection_field: Changes field-level properties (e.g.,max_lengthfor VARCHAR ormmap_enabledfor vectors). Sources: src/collection.rs:551-584, tests/client.rs:136-156alter_collection_properties: Modifies collection-level attributes like TTL (collection.ttl.second) or MMap settings. Sources: src/collection.rs:596-620, tests/client.rs:181-190rename_collection: Changes the collection name viaRenameCollectionRequest. Sources: src/collection.rs:378-409
Alias Management
Aliases allow applications to refer to collections by a symbolic name, facilitating seamless collection swaps.
create_alias: Links an alias to a collection.alter_alias: Redirects an existing alias to a different collection.drop_alias: Removes the symbolic link.
Sources: tests/client.rs:103-120, src/proto/milvus.proto.milvus.rs:8-63
Compaction and Statistics
To maintain performance and optimize storage, the SDK provides tools for data compaction and health monitoring.
| Feature | Description | Source |
|---|---|---|
get_collection_stats | Returns metrics like row_count via GetCollectionStatisticsRequest. | src/collection.rs:418-430 |
manual_compaction | Triggers segment merging and clustering to optimize data layout. | src/collection.rs:722-747 |
get_compaction_state | Polls the progress of a compaction task (Executing, Completed, Failed). | src/collection.rs:749-758 |
flush | Seal segments and ensure data is persisted to storage. | src/collection.rs:656-711 |
Sources: src/collection.rs:165-179, tests/collection.rs:38-42
Summary
Collection Management in the Milvus Rust SDK provides a robust abstraction over the Milvus gRPC API. It handles the complexities of schema serialization, asynchronous state transitions (loading/flushing), and metadata caching. By using the Client interface, developers can programmatically control the entire lifecycle of their vector data, from initial schema definition to long-term storage optimization through compaction and property management.
Sources: src/collection.rs, examples/collection.rs, tests/collection.rs