Data Mutations (Insert, Upsert, Delete)
Relevant source files
The following files were used as context for generating this wiki page:
Data mutations in the Milvus Rust SDK encompass the operations required to modify the state of entities within a collection. This includes adding new entities (Insert), updating existing entities or inserting them if they do not exist (Upsert), and removing entities based on specific criteria (Delete). These operations are primarily facilitated through the Client struct and interact with the Milvus server via gRPC.
The SDK ensures consistency by updating a local collection_cache with timestamps returned from successful mutation results, allowing subsequent operations to maintain synchronized states with the Milvus server. Sources: src/mutate.rs:77-78, src/mutate.rs:113-114.
Insert Operations
The insert operation allows for the addition of new entities into a specified collection. It accepts a list of FieldColumn objects representing the data for each field and optional InsertOptions to specify target partitions.
Insert Architecture and Data Flow
The SDK converts high-level FieldColumn data into a gRPC-compatible InsertRequest. It calculates the number of rows based on the first column provided and populates the request with collection and partition metadata.
The diagram shows the internal flow of an insert call, from parameter mapping to cache updates. Sources: src/mutate.rs:64-92.
Configuration: InsertOptions
InsertOptions provides a builder pattern for configuring the behavior of an insertion, specifically targeting a partition.
| Field | Type | Description |
|---|---|---|
partition_name | String | The name of the partition where data will be inserted. |
Sources: src/mutate.rs:16-19.
Upsert Operations
Upsert (Update or Insert) operations allow developers to either update an existing entity or insert a new one if the primary key is not found. The upsert function utilizes an UpsertRequest structure which mirrors the insert request but with an additional flag for partial updates.
Implementation Details
The current implementation in the Rust SDK sets partial_update to false by default. Like the insert operation, it updates the local collection timestamp upon completion.
pub async fn upsert<S>(
&self,
collection_name: S,
fields_data: Vec<FieldColumn>,
options: Option<InsertOptions>,
) -> Result<crate::proto::milvus::MutationResult>
Sources: src/mutate.rs:155-161, src/proto/milvus.proto.milvus.rs:815-827.
Delete Operations
Delete operations remove entities from a collection based on primary keys or a boolean filter expression.
Delete Logic and Expression Composition
If a filter is not explicitly provided in DeleteOptions, the SDK attempts to "compose" an expression by retrieving the primary key field from the collection cache and creating an IN expression using the provided IDs.
The diagram illustrates how the SDK automatically generates a delete expression when only entity IDs are supplied. Sources: src/mutate.rs:119-153.
Delete Options
| Parameter | Type | Description |
|---|---|---|
ids | ValueVec | A vector of IDs to delete (used when no filter is provided). |
filter | String | A boolean expression defining which entities to delete. |
partition_name | String | Optional partition name to limit the scope of the deletion. |
Sources: src/mutate.rs:38-42.
Mutation Results
Every mutation operation returns a MutationResult message from the server, containing metadata about the success and scale of the operation.
| Field | Type | Description |
|---|---|---|
status | Status | The status of the gRPC call. |
i_ds | IDs | The primary keys of the affected entities. |
succ_index | Vec<u32> | Indices of successfully processed rows. |
insert_cnt | i64 | Total number of inserted rows. |
delete_cnt | i64 | Total number of deleted rows. |
upsert_cnt | i64 | Total number of upserter rows. |
timestamp | u64 | The hybrid timestamp assigned to the operation. |
Sources: src/proto/milvus.proto.milvus.rs:830-850.
Performance and Testing Insights
Aggressive stress testing revealed critical behaviors for mutations under high load:
- Rate Limiting: High-concurrency inserts can trigger
RateLimiterrors if server configurations are not adjusted. Sources: tests/aggressivehpctesting/aggressivetesting.md:100-102. - Partition Churn: Simultaneously writing to multiple unique partitions in a high-concurrency environment can lead to silent data loss in some Milvus versions (e.g., v2.5.15). Sources: tests/aggressivehpctesting/aggressivetesting.md:106-110.
- Consistency: The SDK defaults to
Strongconsistency for delete operations to ensure that once a delete returns, the change is immediately visible. Sources: src/mutate.rs:105.
Summary
The Data Mutation module provides a robust interface for managing entity lifecycles. By abstracting gRPC request construction and handling expression generation for deletions, the SDK simplifies interaction with Milvus while maintaining performance via timestamp-based cache synchronization.