Benchmarking & Performance Tuning
Relevant source files
The following files were used as context for generating this wiki page:
Benchmarking and performance tuning in the oxidize workspace ensure that LLM inference remains efficient across different hardware backends. The project employs a multi-layered strategy involving local criterion benchmarks, CI-gated regression detection, and remote NUMA-aware hardware benchmarking. These tools allow developers to track prefill and decode throughput (tokens per second), memory usage, and numerical accuracy across various model architectures.
Performance-sensitive changes to kernels, quantization schemes, or memory layouts require benchmark evidence to be merged. The system uses automated autotuning to resolve optimal thread counts and memory mapping strategies based on the detected hardware inventory and model fingerprint.
Sources: CONTRIBUTING.md:65-72, scripts/AGENTS.md:6-15, oxidize-cli/src/main.rs:207-215
Benchmark Architecture and Flow
The benchmarking workflow integrates local developer checks with centralized CI enforcement. Developers run local benchmarks using cargo bench and regression scripts before submitting pull requests. CI systems then execute the same suite to prevent performance regressions.
The diagram shows the iterative path from code changes to successful merge, highlighting the different benchmark suites used for validation.
Sources: CONTRIBUTING.md:74-88, scripts/AGENTS.md:18-25
Automated Hardware Tuning
The autotune module detects system capabilities and model characteristics to generate a TuningPlan. This plan dictates the optimal execution environment, including thread allocation and memory management.
Tuning Logic for CPU
For dense models, the system prioritizes NUMA locality. If a model fits within a single NUMA node, it pins threads to that node to avoid cross-node memory thrashing. For larger models exceeding node memory, it switches to interleave mode.
| Metric | Condition | Action |
|---|---|---|
| NUMA Mode | Model fits in node | single |
| NUMA Mode | Model > 192 GiB or > 80% RAM | interleave |
| Threads | Huge model (> 48 logical cores) | Cap at 48 threads |
| Threads | Standard model | Physical core count |
| Hugepages | Model >= 200 GiB + Avail | Enable 2MiB Hugepages |
Sources: oxidize-cpp/src/autotune.cpp:78-120, oxidize-cli/src/main.rs:217-230
Autotune Flow
The sequence shows how the system resolves hardware and model information into a runnable plan.
Sources: oxidize-cpp/src/autotune.cpp:58-75, oxidize-cli/src/main.rs:431-445
Metrics and Regression Detection
The project tracks several key performance indicators (KPIs) to ensure quality. The regression script (ci_benchmark_regression.py) compares current runs against historical baselines.
Primary Metrics
- Prefill Speed: Measured in tokens per second during the initial prompt processing.
- Decode Speed: Measured in tokens per second during sequential token generation.
- RSS (Resident Set Size): The actual physical memory consumed by the process.
- Perplexity: A measure of the model's numerical accuracy and output quality, typically run on datasets like
wikitext2orc4.
Sources: scripts/bench_qwen36_c_remote.sh:205-225, oxidize-core/src/util/benchmark_suite.rs:55-75, oxidize-tui/src/engine/metrics.test.ts:5-15
Regression Gating
The regression script identifies significant performance drops. It expects a JSON or JSONL format containing the benchmark results.
# Example schema for benchmark records
{
"engine": "oxidize-c",
"case": "pp64/tg32",
"prefill_tok_per_s": 4540.92,
"decode_tok_per_s": 42.5,
"rss_kb": 2048,
"timing": {"elapsed_s": 1.2}
}
Sources: scripts/bench_qwen36_c_remote.sh:64-70, oxidize-tui/src/engine/metrics.test.ts:7-13
Remote and NUMA Benchmarking
For large-scale hardware like dual-socket Xeon servers, specialized scripts manage remote execution and resource isolation.
CPU Affinity and Pinning
Benchmark tasks are pinned to specific CPU lists and NUMA nodes using taskset and numactl. This prevents the OS scheduler from migrating workers across NUMA boundaries, which would invalidate memory throughput measurements.
# Example benchmark contract execution
taskset -c 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30 \
numactl --membind=0 \
./oxidize-c bench --model /path/to/model.gguf --threads 16
Sources: scripts/bench_qwen36_c_remote.sh:5-10, oxidize-cli/src/main.rs:248-255
Remote Host Monitoring
Before starting a benchmark, the script verifies the remote host's load average. It defers execution if the one-minute load is too high (e.g., >= 96.0 on a 96-core system) or if competing inference processes are detected.
Sources: scripts/bench_qwen36_c_remote.sh:116-125
Conclusion
Benchmarking in oxidize is a rigorous process designed to protect the "zero-cost" and high-performance promises of the Rust core. By combining hardware-aware autotuning with strict CI regression gates and remote NUMA validation, the project maintains predictable performance across varying deployment targets.
Sources: PRD.md:15-20, CONTRIBUTING.md:90-95