Benchmarking Suite
Relevant source files
The following files were used as context for generating this wiki page:
The Benchmarking Suite in cargo-cooldown is designed to measure the performance and accuracy of the cooldown resolution logic against real-world data from crates.io. Unlike the deterministic integration test suite which uses local registries, the benchmarking suite intentionally uses real crates.io packages to perform non-deterministic smoke checks and performance probes.
Sources: examples/README.md:3-6, tests/integration.rs:1-1
Core Components
The suite consists of shell scripts, Rust benchmark harnesses, and dedicated benchmark workspaces. It is designed to simulate various cooldown scenarios, ranging from small "smoke" checks to aggressive 60-day cooldown windows on large dependency graphs.
Sources: examples/README.md:8-16, benches/crates_io_cooldown.rs:1-5
Benchmark Execution Flow
The following diagram illustrates how a benchmark run is orchestrated, from the initial cargo bench command to the final reporting of durations and HTTP fallback usage.
The benchmark runner performs a sequence of setup steps including cache warming and prefetching before executing timed iterations.
Sources: examples/run-crates-io-benchmark.sh:110-310, benches/crates_io_cooldown.rs:11-30
Benchmark Configurations
The suite supports multiple scenarios and is highly configurable through environment variables. The primary entry point for performance measurement is the run-crates-io-benchmark.sh script, which serves as a shared wall-clock benchmark runner.
Sources: examples/README.md:12-14, examples/run-crates-io-benchmark.sh:6-21
Scenarios
The suite defines specific workloads to test different aspects of the cargo-cooldown executor:
- Default/Smoke: Small
crates.io-backed workspace for fast manual checks. - Large 60-day: Uses a larger dependency graph (found in
crates-io-large-benchmark-workspace/) with an aggressive 60-day cooldown window.
Sources: examples/README.md:8-11, benches/crates_io_cooldown.rs:43-47
Configuration Parameters
| Variable | Description | Default |
|---|---|---|
SAMPLES | Number of iterations to run for averaging. | 3 |
COOLDOWN_MINUTES | The age requirement for registry packages. | 131401 |
BENCH_OFFLINE | Force Cargo to run in offline mode during measurements. | 0 |
BENCH_PREFETCH | Preload cooldown candidate crates before measuring. | 0 |
BENCH_ISOLATED_CARGO_HOME | Use a temporary CARGO_HOME to avoid cache pollution. | 0 |
Sources: examples/run-crates-io-benchmark.sh:8-18, benches/crates_io_cooldown.rs:104-110
Measurement and Reporting
The benchmarking suite focuses on two primary metrics: wall-clock duration and Registry API fallback usage.
Timing Logic
The runner uses high-precision timers (via Python's time.perf_counter_ns()) to measure the duration of the cargo-cooldown update command. It calculates an average duration across all samples to provide a reliable performance baseline.
Sources: examples/run-crates-io-benchmark.sh:78-83, examples/run-crates-io-benchmark.sh:220-226, examples/run-crates-io-benchmark.sh:285-285
Fallback Inspection
A key feature of the suite is the "Fallback Report." Since cargo-cooldown prefers reading timestamps from the local index, the benchmark tracks how often it must fall back to the Registry API to retrieve missing metadata.
Sources: examples/run-crates-io-benchmark.sh:236-281, examples/run-crates-io-benchmark.sh:312-317
The benchmark identifies instances where local metadata was insufficient, requiring network requests to the registry API.
Sources: examples/run-crates-io-benchmark.sh:246-271, tests/integration.rs:114-135
Artifact Collection
Each benchmark run generates a set of artifacts stored in a directory named with a UTC timestamp (BENCH_RUN_ID). This allows developers to audit the results of the cooldown process.
- Cargo.lock: The final cooled lockfile for each sample.
- cooldown.log: Captured verbose output from the tool execution.
- samples.tsv: A tab-separated file containing per-sample metrics (duration, fallback counts).
Sources: examples/run-crates-io-benchmark.sh:49-62, examples/run-crates-io-benchmark.sh:150-163
Conclusion
The Benchmarking Suite provides a critical feedback loop for the project, ensuring that the cooldown logic remains performant as dependency graphs grow in complexity. By simulating real-world crates.io interactions and providing detailed fallback reports, it allows for the optimization of metadata retrieval and the validation of the batch solver's efficiency.