HPC Phase 1: High-Concurrency CRUD

Relevant source files

The following files were used as context for generating this wiki page:

• tests/aggressivehpctesting/phase_1_crud_and_indexing.rs
• tests/aggressivehpctesting/aggressivetesting.md
• tests/aggressivehpctesting/phase_2_search_and_query.rs
• tests/aggressivehpctesting/phase_3_soak_test.rs
• src/proto/milvus.proto.milvus.rs

Introduction

HPC Phase 1 is a high-performance computing stress test designed to verify the robustness of fundamental Create, Read, Update, and Delete (CRUD) operations, alongside indexing, under heavy concurrent load. This phase focuses on multi-threaded stability and ensuring that the Milvus Rust SDK can handle large data volumes without data loss or performance degradation in a real-world high-traffic environment.

The scope includes the creation of large shared collections, concurrent batch inserts across multiple writer tasks, random deletions, and index lifecycle management (creation and recreation). This phase serves as the foundation for subsequent stability assessments such as Phase 2: High-Volume Search and Query Under Load and Phase 3: Soak Test for Stability.

Sources: [tests/aggressivehpctesting/aggressivetesting.md:21-25](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/aggressivetesting.md#L21-L25), [tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:21-46](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/phase_1_crud_and_indexing.rs#L21-L46)

Architecture and Test Flow

The architecture of Phase 1 utilizes an asynchronous runtime via tokio to spawn multiple concurrent task groups. These groups interact with a single large collection shared across all threads.

Operational Lifecycle

The test follows a structured sequence: setup of schema and collection, execution of concurrent task groups, indexing, and final logical verification.

The flow ensures that writers and deleters operate simultaneously to simulate system "churn" before indexing occurs.
Sources: [tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:24-110](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/phase_1_crud_and_indexing.rs#L24-L110), [tests/aggressivehpctesting/aggressivetesting.md:26-55](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/aggressivetesting.md#L26-L55)

Component Breakdown

Schema Definition

Phase 1 uses a complex schema to test various data types and their alignment during high-speed ingestion.

Field Name	Data Type	Constraints
id	Int64	Primary Key, AutoID: False
DEFAULT_VEC_FIELD	FloatVector	Dimension: 128 (default)
varchar_field	VarChar	Max Length: 256

Sources: [tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:28-36](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/phase_1_crud_and_indexing.rs#L28-L36), [tests/aggressivehpctesting/phase_2_search_and_query.rs:41-43](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/phase_2_search_and_query.rs#L41-L43)

Concurrent Task Groups

The testing suite defines specific task counts and batch sizes to saturate the connection pool and server-side ingestion buffers.

• Writers: 20 tasks, each inserting 10,000 entities in batches of 1,000.
• Deleters: 5 tasks, each selecting random batches of 100 IDs for removal.
• Client Configuration: The ClientBuilder is configured with a 60-second timeout to handle gRPC bottlenecks observed under high load.

Sources: [tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:14-23](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/phase_1_crud_and_indexing.rs#L14-L23), [tests/aggressivehpctesting/aggressivetesting.md:87-89](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/aggressivetesting.md#L87-L89)

Data Flow: Ingestion and Mutation

The ingestion process involves constructing FieldColumn objects for each batch and calling the insert API. To maintain thread safety for verification, inserted IDs are tracked via an Arc<Mutex<Vec<i64>>>.

Verification and Indexing Logic

Once concurrent operations finish, a flush is performed to ensure all segments are sealed. Verification is performed using a logical count (count(*)) rather than physical statistics to account for deletions that may not yet be reflected in the server's background compaction.

Consistency Levels

For accurate verification in high-concurrency scenarios, the test employs ConsistencyLevel::Strong. This ensures that the query returns the most up-to-date logical view of the data.

Sources: [tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:113-125](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/phase_1_crud_and_indexing.rs#L113-L125), [src/proto/milvus.proto.milvus.rs:1034-1036](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/src/proto/milvus.proto.milvus.rs#L1034-L1036)

Performance Findings and Limitations

During the execution of Phase 1, several critical environmental and server-side behaviors were identified:

1. Rate Limiting: Default server limits often triggered RateLimit errors during the 20-task writer burst.
2. Partition Bottleneck: A specific bottleneck was discovered in Milvus v2.5.15 where high-concurrency inserts into multiple unique partitions resulted in data loss; Phase 1 stability was achieved by writing directly to the collection instead.
3. gRPC Timeouts: Heavy load necessitated increasing the client-side timeout from default values to 60 seconds.

Sources: [tests/aggressivehpctesting/aggressivetesting.md:82-94](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/aggressivetesting.md#L82-L94)

Summary

HPC Phase 1 validates the core mutation capabilities of the Milvus Rust SDK. By utilizing tokio::spawn and shared mutexes, it ensures that the SDK can orchestrate complex, overlapping CRUD operations. The successful completion of this phase, involving over 200,000 entity operations per cycle, confirms the SDK's readiness for high-traffic production environments.

Sources: [tests/aggressivehpctesting/aggressivetesting.md:80-81](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/aggressivetesting.md#L80-L81), [tests/aggressivehpctesting/phase_3_soak_test.rs:106-112](https://github.com/marlon-costa-dc/milvus-sdk-rust/blob/19f0460c4cdafcffc4b1c4b1239ef17f14c7cb39/tests/aggressivehpctesting/phase_3_soak_test.rs#L106-L112)