Hardware Autotuning
Relevant source files
The following files were used as context for generating this wiki page:
Hardware Autotuning identifies host hardware capabilities and model characteristics to generate an optimized inference TuningPlan. The system automatically configures execution parameters such as thread counts, NUMA strategies, and memory mapping options to ensure optimal performance across different platforms, including Rust, C++, Go, and Python ports.
Sources: oxidize-core/src/autotune/AGENTS.md:5-10, oxidize-cli/src/main.rs:136-150
Architecture and Data Flow
The autotuning system operates as a tiered pipeline. It first probes the host environment to create a HardwareInventory, then analyzes the targeted model to create a ModelFingerprint. A pure-function planner consumes these inputs to produce a TuningPlan, which is then applied as overrides to the inference engine.
The flow of information from hardware detection and model fingerprinting to the final inference plan.
Sources: oxidize-core/src/autotune/AGENTS.md:12-20, oxidize-core/src/autotune/mod.rs
Hardware Detection
The detect() function probes the operating system for hardware specifications. Probes are designed to be efficient, completing in under 50 milliseconds to avoid delaying startup. The C++ implementation specifically checks for /proc/meminfo to determine total RAM and probes /sys/kernel for hugepage availability.
HardwareInventory Data Structure
| Field | Description |
|---|---|
logical_cores | Total number of logical processing units. |
physical_cores | Number of physical CPU cores (excludes SMT/Hyper-threading). |
numa_nodes | Number of detected NUMA nodes. |
total_ram_bytes | Total system memory available. |
hugepages_2mib_avail | Boolean indicating if 2 MiB hugepages are free for use. |
simd_backend | Supported instruction sets (AVX2, AVX512, NEON, etc.). |
Sources: oxidize-core/src/autotune/AGENTS.md:23-30, oxidize-cpp/src/autotune.cpp:52-64, oxidize-c/include/oxidize/autotune_rules.h:37-51
Model Fingerprinting
The system fingerprints GGUF files to extract metadata essential for performance tuning. This analysis occurs after the model is memory-mapped but before generation begins.
Key fingerprinting attributes include:
- Architecture: Identifies the model type (e.g., Llama, DFlash, MoE).
- Dimensions: Hidden sizes, number of heads, and KV head counts.
- Quantization: The dominant bit-depth of the weights.
- Scale: Total file size and parameter count.
Sources: oxidize-core/src/autotune/AGENTS.md:31-35, oxidize-cli/src/main.rs:290-305, oxidize-c/include/oxidize/autotune_rules.h:53-62
Planning Rules
The planning logic is implemented as a pure function to ensure deterministic results. It evaluates tiers of rules to resolve the final TuningPlan. Every decision made by the planner is recorded in the plan.rationale field for transparency.
NUMA and Threading Logic
The planner follows specific heuristics for multi-socket systems:
- Dense Models (≤192 GB): Uses
singleNUMA mode and limits threads to the logical cores of one node (typically 16). - Large Models (>192 GB or >80% RAM): Uses
interleaveNUMA mode and scales threads up to 48 logical cores to avoid cross-node thrashing. - Single Node: Uses physical core count by default for threads.
Memory Optimization
If a model exceeds 200 GiB and 2 MiB hugepages are available, the planner enables mmap_hugepages. In the Rust implementation, it also calculates a context cap based on available RAM, reserving approximately 8 GiB for the operating system and workspace overhead.
Sources: oxidize-core/src/autotune/rules.rs, oxidize-cpp/src/autotune.cpp:77-118, oxidize-cli/src/main.rs:340-360
CLI Integration and Overrides
Users can control the autotuner via CLI flags. Explicit user-provided flags always take precedence over autotuned values.
| Flag | Effect |
|---|---|
--auto | Enables hardware detection and tuning (default). |
--no-auto | Opts out of auto-tuning, using only explicit flags. |
--print-plan | Prints the resolved plan to stderr before execution. |
--print-plan=json | Emits the plan in machine-readable JSON format. |
Sources: oxidize-cli/src/main.rs:136-150, oxidize-core/src/autotune/AGENTS.md:40-42
// Snippet illustrating the logic where explicit flags beat implicit tuning
let mut args = Args {
n_gpu_layers_set: user_passed_flag(&std::env::args().collect(), "--n-gpu-layers"),
kv_cache_dtype_set: user_passed_flag(&std::env::args().collect(), "--kv-cache-dtype"),
..args
};
Sources: oxidize-cli/src/main.rs:188-196
Summary
Hardware Autotuning in oxidize provides a zero-configuration entry point for optimal inference performance. By combining deep hardware probing with model metadata analysis, it balances thread parallelism and memory management strategies across diverse hardware environments. The system maintains portability through parallel implementations in Rust, C, and C++, ensuring consistent behavior across all deployment targets.
Sources: oxidize-core/src/autotune/AGENTS.md:5-10, oxidize-cpp/src/autotune.cpp:115-120