HPC Phase 2: Search and Query Under Load

Relevant source files

The following files were used as context for generating this wiki page:

• tests/aggressivehpctesting/phase_2_search_and_query.rs
• tests/aggressivehpctesting/aggressivetesting.md
• src/query.rs
• src/proto/milvus.proto.milvus.rs
• tests/aggressivehpctesting/phase_1_crud_and_indexing.rs
• examples/query_search.rs

Introduction

HPC Phase 2 is a high-performance computing stress test designed to verify the stability and performance of search and query functionalities within the Milvus Rust SDK. This phase specifically focuses on subjecting the system to high volumes of concurrent read operations while the underlying collection is undergoing active data modification, also known as "churn." This simulates a dynamic, high-traffic production environment where data is frequently updated or deleted.

The primary objective of Phase 2 is to ensure that search and query operations remain consistent, plausible, and error-free even under heavy load. This follows Phase 1: High-Concurrency CRUD and Indexing, which establishes the initial dataset and indexing structures required for retrieval testing.

Sources: tests/aggressivehpctesting/aggressivetesting.md:46-52, tests/aggressivehpctesting/phase_2_search_and_query.rs:26-30

Architecture and Execution Flow

The test architecture utilizes tokio::spawn to manage multiple pools of concurrent tasks. The execution is divided into three distinct stages: setup, concurrent execution, and verification.

Test Workflow

The following diagram illustrates the lifecycle of the Phase 2 test execution.

Component Breakdown

1. Initial Data Population

Before stress testing begins, the collection is populated with a baseline of 100,000 entities. This is performed using a batch size of 1,000 entities per insert operation, distributed across multiple spawned tasks to minimize setup time.
Sources: tests/aggressivehpctesting/phase_2_search_and_query.rs:21-23, tests/aggressivehpctesting/phase_2_search_and_query.rs:55-88

2. Concurrent Task Groups

The core of Phase 2 consists of three distinct task groups running simultaneously for a duration of 60 seconds.

Task Group	Concurrency	Primary Operation	Parameters
Searchers	50 Tasks	client.search()	Top-K: 10, Radius: 1.0, Metric: L2
Queriers	20 Tasks	client.query()	Random ID expression (id == random_id)
Churn	5 Tasks	client.insert() & client.delete()	Inserts 1 new ID, Deletes 1 random ID per loop

Sources: tests/aggressivehpctesting/phase_2_search_and_query.rs:21-25, tests/aggressivehpctesting/phase_2_search_and_query.rs:108-154, tests/aggressivehpctesting/aggressivetesting.md:58-66

3. Search and Query Logic

The retrieval operations leverage the SearchOptions and QueryOptions structures provided by the SDK.

• Search Operation: Uses random vectors generated with rand::rng().random() to simulate diverse query patterns. It targets a specific vector field (e.g., DEFAULT_VEC_FIELD).
• Query Operation: Utilizes complex filter expressions and specifies output_fields to retrieve both the primary key and metadata (e.g., varchar_field).

// Searcher Task Example
let search_options = SearchOptions::new()
    .limit(10)
    .radius(1.0)
    .add_param("anns_field", DEFAULT_VEC_FIELD);
let result = client.search(&collection_name, vec![query_vector.into()], Some(search_options)).await;

// Querier Task Example
let query_options = QueryOptions::new()
    .output_fields(vec!["id".to_string(), "varchar_field".to_string()]);
let result = client.query(&collection_name, &expr, &query_options).await;

Sources: tests/aggressivehpctesting/phase_2_search_and_query.rs:114-136, src/query.rs:250-264

Data Consistency and Load Handling

Phase 2 tests the SDK's ability to handle high-frequency interactions with the Milvus server's gRPC endpoints. Key technical considerations addressed during this phase include:

Dynamic Environment (Churn)

The "Churn" pool ensures the collection's state is not static. By removing IDs from a shared Arc<Mutex<Vec<i64>>> list and deleting them from the collection, the test verifies that retrieval operations do not crash or return invalid data when segments are being compacted or updated.
Sources: tests/aggressivehpctesting/phase_2_search_and_query.rs:141-164, tests/aggressivehpctesting/aggressivetesting.md:65-66

Performance Bottlenecks and Timeouts

Phase 2 execution revealed that high load can cause client-side gRPC timeouts. This is mitigated by configuring a 60-second timeout in the ClientBuilder.
Sources: tests/aggressivehpctesting/aggressivetesting.md:95-97, tests/aggressivehpctesting/phase_2_search_and_query.rs:32-34

Verification and Results

Verification in Phase 2 is performed through assertions within each concurrent task and a final join of all handles.

• Error Checking: Every search, query, and churn operation must return Ok(()). Any Err results in a test failure.
• Stability Result: Phase 2 has successfully passed, demonstrating that the SDK maintains stability while handling high volumes of concurrent operations on an actively modified collection. No silent data loss or panics were observed during the read-heavy retrieval phase.

Sources: tests/aggressivehpctesting/phase_2_search_and_query.rs:125-168, tests/aggressivehpctesting/aggressivetesting.md:104-108

Conclusion

HPC Phase 2: Search and Query Under Load serves as a critical benchmark for the Milvus Rust SDK's reliability. By simulating 50 simultaneous searchers and 20 queriers alongside active data mutation, it confirms that the SDK's retrieval logic is robust enough for high-traffic environments. The successful completion of this phase ensures the SDK's read-path performance is production-ready.
Sources: tests/aggressivehpctesting/aggressivetesting.md:104-108, tests/aggressivehpctesting/phase_2_search_and_query.rs:172-176