Aggressive HPC Testing Overview

Relevant source files

The following files were used as context for generating this wiki page:

• tests/aggressivehpctesting/aggressivetesting.md
• tests/aggressivehpctesting/mod.rs
• tests/aggressivehpctesting/phase_1_crud_and_indexing.rs
• tests/aggressivehpctesting/phase_2_search_and_query.rs
• tests/aggressivehpctesting/phase_3_soak_test.rs
• src/proto/milvus.proto.milvus.rs

The Aggressive High-Performance Computing (HPC) Testing suite is a rigorous validation framework designed to ensure the Milvus Rust SDK is stable, performant, and production-ready. Its primary objective is to subject the SDK to extreme stress scenarios, including maximal concurrency, large data volumes, and long-duration execution to identify edge-case issues such as memory leaks or performance degradation under load.

Sources: tests/aggressivehpctesting/aggressivetesting.md:3-8

Testing Architecture and Principles

The testing framework is built on four guiding principles:

• Maximal Concurrency: Utilizing tokio::spawn and multi-threading to simulate high-traffic environments.
• Large Data Volumes: Using datasets significantly larger than standard unit tests to push SDK and server boundaries.
• Complex Interactions: Mimicking real-world application logic with overlapping and interdependent operations.
• Long-Duration Testing: Implementing soak tests to detect subtle, time-dependent regressions.

Sources: tests/aggressivehpctesting/aggressivetesting.md:12-17

High-Level Test Flow

The following diagram illustrates the sequence of operations across the three phases of aggressive testing.

---

Phase 1: High-Concurrency CRUD and Indexing

This phase verifies the robustness of fundamental Create, Read, Update, and Delete (CRUD) operations and indexing under heavy parallel load. It utilizes a shared large collection with a schema including Int64, FloatVector, and VarChar fields.

Sources: tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:26-38, tests/aggressivehpctesting/aggressivetesting.md:23-28

Component Distribution

Component	Task Count	Responsibility
Writers	20	Inserting 10,000 entities in batches of 1,000 using client.insert.
Deleters	5	Randomly selecting and deleting batches of 100 recently inserted IDs.
Upserters	5	Updating existing entities via upsert operations.
Indexer	1	Creating and dropping IVF_FLAT indices dynamically.

Sources: tests/aggressivehpctesting/aggressivetesting.md:33-51, tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:16-20

Technical Findings

During execution, several bottlenecks were identified and addressed:

• gRPC Timeouts: High load required increasing client timeouts via ClientBuilder::timeout.
• Rate Limiting: Default Milvus server limits were exceeded, requiring custom user.yaml configurations.
• Data Loss in Partitions: A performance bottleneck was discovered when writing to multiple unique partitions concurrently; stability was only achieved by writing directly to the collection.

Sources: tests/aggressivehpctesting/aggressivetesting.md:86-103, tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:24-25

---

Phase 2: High-Volume Search and Query Under Load

The second phase stresses search and query functionality while the collection is undergoing active data churn (continuous inserts and deletes).

Sources: tests/aggressivehpctesting/aggressivetesting.md:58-62

Concurrency Model

The test executes for a duration of 60 seconds, spawning 75 total concurrent tasks:

1. Searchers (50 tasks): Perform search with random vectors and top_k=10, plus range searches.
2. Queriers (20 tasks): Execute query with complex boolean filter expressions (e.g., id > 10000 and varchar_field like "prefix%").
3. Churners (5 tasks): Continuously perform inserts, deletes, and upsets to ensure the data environment is dynamic.

Sources: tests/aggressivehpctesting/phase_2_search_and_query.rs:20-25, tests/aggressivehpctesting/aggressivetesting.md:67-75

---

Phase 3: Soak Test for Stability

The soak test is a long-running execution (1 hour) designed to identify memory leaks, connection drops, and performance degradation. It cycles through the entire SDK lifecycle repeatedly.

Sources: tests/aggressivehpctesting/aggressivetesting.md:81-84, tests/aggressivehpctesting/phase_3_soak_test.rs:18-20

SDK Operational Cycle

Each cycle within the soak test performs the following steps in order:

1. Collection Setup: create_collection and create_partition.
2. Ingestion: insert a batch of 1,000 entities followed by flush.
3. Index Management: create_index and load_collection.
4. Data Retrieval: Concurrent search and query operations.
5. Teardown: release_collection, drop_index, delete entities, and drop_collection.

Sources: tests/aggressivehpctesting/phase_3_soak_test.rs:82-178, tests/aggressivehpctesting/aggressivetesting.md:112-115

Error Handling Logic

// Strategic timeout and flush to prevent drop hangs observed in server v2.5.15
if let Ok(Ok(true)) = timeout(Duration::from_secs(10), client.has_collection(&collection_name)).await {
    let _ = timeout(Duration::from_secs(30), client.flush(&collection_name)).await;
    let _ = timeout(Duration::from_secs(15), client.drop_collection(&collection_name)).await;
}

Sources: tests/aggressivehpctesting/phase_3_soak_test.rs:52-66

---

Test Parameters Summary

The following table summarizes the configuration parameters used across the aggressive testing suite.

Parameter	Phase 1 Value	Phase 2 Value	Phase 3 Value
Batch Size	1,000	1,000	1,000
Total Entities	200,000	100,000 (initial)	1,000 per cycle
Writer Tasks	20	5 (churn)	1 (serial loop)
Searcher Tasks	N/A	50	1 (serial loop)
Duration	Task completion	60 seconds	1 hour
Index Type	IVF_FLAT	IVF_FLAT	IVF_FLAT
Metric Type	L2	L2	L2

Sources: tests/aggressivehpctesting/phase_1_crud_and_indexing.rs:16-20, tests/aggressivehpctesting/phase_2_search_and_query.rs:20-25, tests/aggressivehpctesting/phase_3_soak_test.rs:18-20

Conclusion

The Aggressive HPC Testing suite demonstrated that the Milvus Rust SDK is capable of handling over 290 full operational cycles without panics or memory leaks during a 60-minute soak test. While Phase 1 revealed specific server-side partition bottlenecks and the need for increased gRPC timeouts, the SDK proved highly stable in dynamic environments with heavy concurrent search and query loads.

Sources: tests/aggressivehpctesting/aggressivetesting.md:105-115