HPC Phase 3: Soak Testing

Relevant source files

The following files were used as context for generating this wiki page:

• tests/aggressivehpctesting/phase_3_soak_test.rs
• tests/aggressivehpctesting/aggressivetesting.md
• tests/aggressivehpctesting/mod.rs
• src/proto/milvus.proto.milvus.rs
• Cargo.toml

HPC Phase 3: Soak Testing is the final stage of the Milvus Rust SDK "Aggressive Testing Plan." Its primary objective is to verify long-term stability and detect subtle resource issues such as memory leaks, connection degradation, or performance drops that only manifest over extended operational periods. Unlike Phase 1 (Concurrency) and Phase 2 (Volume), Phase 3 focuses on the reliability of the entire lifecycle of SDK operations.

Sources: tests/aggressivehpctesting/aggressivetesting.md:1-10, tests/aggressivehpctesting/aggressivetesting.md:79-81

Test Architecture and Scope

The soak test is designed as a long-running, iterative process that executes a complete cycle of major SDK functions. By repeatedly creating and destroying resources, the test ensures that the client-side library and the server-side interaction remain stable without leaking memory or hanging during teardown.

Operational Parameters

The test configuration is characterized by a fixed long duration and aggressive error handling to prevent the test suite from hanging indefinitely.

Parameter	Value	Description
Duration	60 Minutes	The total time the soak test runs.
Cycle Delay	5 Seconds	Sleep interval between iterative cycles.
gRPC Timeout	60 Seconds	Client-side timeout for gRPC operations.
Batch Size	1,000 Entities	Number of vectors inserted per cycle.

Sources: tests/aggressivehpctesting/phase_3_soak_test.rs:18-24, tests/aggressivehpctesting/aggressivetesting.md:83-86

Execution Lifecycle

The soak test operates in a loop, where each iteration executes run_full_sdk_cycle. This function exercises the majority of the SDK's public API surface area in a specific sequence to simulate a dynamic production environment.

Lifecycle Flow Diagram

This diagram illustrates the sequential operations performed in every iteration of the soak test.

Critical Logic: Idempotency and Cleanup

A key architectural requirement discovered during development is the necessity of "Flush before Drop." If a collection is not flushed before being dropped, the operation may hang or lead to duplicate collection errors in subsequent cycles. The test implements aggressive tokio::time::timeout wrappers around collection cleanup to prevent a single failure from stalling the entire suite.

Sources: tests/aggressivehpctesting/phase_3_soak_test.rs:69-82, tests/aggressivehpctesting/phase_3_soak_test.rs:101-118

SDK Components Exercised

The soak test utilizes a broad range of Milvus SDK modules and gRPC requests.

Data Schemas

Every cycle builds a new CollectionSchema with the following field types:

• Primary Key: Int64 (with auto_id set to true).
• Vector Field: FloatVector using DEFAULT_DIM (typically 128 or 256).

Sources: tests/aggressivehpctesting/phase_3_soak_test.rs:119-125

API Sequence and Logic

The following table maps the soak test steps to the underlying gRPC requests defined in the protobuf service:

Step	gRPC Method	Request Message Type
Setup	CreateCollection / CreatePartition	CreateCollectionRequest / CreatePartitionRequest
Mutation	Insert / Delete	InsertRequest / DeleteRequest
Maintenance	Flush / CreateIndex	FlushRequest / CreateIndexRequest
Querying	Search / Query	SearchRequest / QueryRequest
Teardown	ReleaseCollection / DropCollection	ReleaseCollectionRequest / DropCollectionRequest

Sources: src/proto/milvus.proto.milvus.rs:2537, src/proto/milvus.proto.milvus.rs:2564, src/proto/milvus.proto.milvus.rs:2715, tests/aggressivehpctesting/phase_3_soak_test.rs:101-189

Verification and Monitoring

The test success is measured by two primary criteria:

1. Cycle Completion: The test must complete over 290 full operational cycles within the 60-minute window without panics.
2. Resource Stability: Observation of memory usage for both the Rust test process and the Milvus server to ensure no linear growth occurs over time.

Error Handling Strategy

The test uses a Result<()> return type for the run_full_sdk_cycle function. Any error encountered—even intermittent network issues—is caught, logged, and leads to a panic to fail the test immediately. This ensures that "flaky" behavior is treated as a critical stability failure.

// Snippet of the error-checked loop in phase_3_soak_test.rs
while start_time.elapsed() < test_duration {
    cycle_count += 1;
    let cycle_result = run_full_sdk_cycle(&client, &collection_name, &partition_name).await;

    if let Err(e) = cycle_result {
        eprintln!("Soak test cycle {} failed with error: {:?}. Aborting test.", cycle_count, e);
        panic!("Soak test cycle failed.");
    }
}

Sources: tests/aggressivehpctesting/phase_3_soak_test.rs:43-55, tests/aggressivehpctesting/aggressivetesting.md:102-106

Summary of Phase 3 Results

The execution of the soak test resulted in a PASS. The SDK successfully completed the full one-hour duration, completing over 290 cycles. Key findings include:

• Stability: Zero panics or unhandled exceptions occurred during high-frequency resource creation and destruction.
• Memory: No observable memory leaks were detected in the Rust client.
• Infrastructure: The test confirmed that using the stable embedded etcd and providing custom server configurations (configs/user.yaml) prevents environment-related timeouts.

Sources: tests/aggressivehpctesting/aggressivetesting.md:126-130, tests/aggressivehpctesting/aggressivetesting.md:112-117