Hardware & Execution Providers
Relevant source files
The following files were used as context for generating this wiki page:
Hardware and Execution Providers (EPs) form the foundational layer of Olive Studio, determining how optimization recipes are executed and deployed across diverse silicon targets. The system manages the mapping between high-level optimization intent and specific hardware capabilities, including CPU, NVIDIA GPUs (CUDA/TensorRT), Intel (OpenVINO), and Qualcomm (QNN).
The core scope of this module includes hardware probing to detect local capabilities, managing vendor-specific library dependencies (such as TensorRT and CUDA), and validating recipe compatibility against the detected or targeted hardware environment.
Hardware Probing & Detection
The project utilizes a multi-stage probing mechanism to identify available compute resources. This includes checking system metrics, querying vendor-specific CLIs like nvidia-smi or rocm-smi, and inspecting the local Python environment for installed ONNX Runtime providers.
Probing Workflow
The backend performs an asynchronous probe that aggregates data from the operating system and installed drivers to recommend the optimal Execution Provider.
The system captures specific metrics such as VRAM in MB, driver versions, and system RAM in GB.
Sources: server.ts:600-660, src/lib/hardwareProbe.ts
Detected Providers
The system categorizes providers based on their architectural target:
- CPU:
CPUExecutionProvider - NVIDIA:
CUDAExecutionProvider,TensorrtExecutionProvider,NvTensorRTRTXExecutionProvider - Intel:
OpenVINOExecutionProvider - Qualcomm:
QNNExecutionProvider - AMD:
ROCMExecutionProvider
Sources: README.md:40-50, server.ts:664-672
Execution Provider Dependencies
Certain Execution Providers require specific external libraries and environment configurations. The system manages these through "pinned" dependencies to ensure stability between the ONNX Runtime version and native GPU drivers.
Dependency Management Matrix
| Provider | Core Dependency | Version/Source |
|---|---|---|
| CUDA | onnxruntime-gpu | Pinned stable build |
| TensorRT | tensorrt | Pinned v10.x (nvinfer_10) |
| TensorRT RTX | tensorrt-rtx | Targeted for consumer GeForce |
| OpenVINO | openvino, optimum[openvino] | Standard PyPI |
Sources: src/lib/tensorrtDeps.ts:13-25, src/lib/tensorrtRtxDeps.ts:10-20, server.ts:335-375
TensorRT Library Verification
The system performs a "probe load" of native libraries using Python's ctypes to ensure that DLLs/SOs are actually loadable by the runtime before starting an optimization job.
Sources: server.ts:503-530, server.ts:553-575
Compatibility & Validation Logic
Recipes are validated against hardware through a tier-based assessment. This prevents users from attempting optimizations that their hardware cannot support (e.g., running AWQ quantization on a CPU-only machine).
Hardware Compatibility Tiers
- Compatible: Hardware and drivers verified as present.
- Unknown: Hardware might be present but unverified (e.g., remote targets).
- Unavailable: Hardware missing or library load failure.
Sources: src/lib/recipeHardwareCompatibility.ts:15-30, src/components/features/InputEnvironmentPanel.tsx:165-180
Validation Rules
The system enforces rules such as:
- AWQ/Quantization: Requires an NVIDIA GPU.
- OpenVINO IR: Requires the OpenVINO Execution Provider.
- VRAM Estimates: Compares estimated model footprint against detected VRAM to warn of potential OOM (Out of Memory) conditions.
Sources: README.md:55-60, src/components/features/ExecutionWorkspace.tsx:845-860
Environment Configuration
When executing an Olive run, the system must prepend native library paths to the system PATH (Windows) or LD_LIBRARY_PATH (Linux) so the ONNX Runtime can find vendor-specific binaries like nvinfer_10.dll.
Environment Path Construction
The server identifies the directory of installed site-packages (like nvidia, tensorrt_libs, and onnxruntime/capi) and constructs a clean environment for the Python subprocess.
async function buildOliveRunEnvironment(
python: string,
provider: IHVProvider,
base: NodeJS.ProcessEnv,
): Promise<NodeJS.ProcessEnv> {
if (!isGpuExecutionProvider(provider)) {
return base;
}
const libPaths = await getNativeGpuLibPaths(python);
return envWithPrependedPaths(base, libPaths);
}
Sources: server.ts:435-445, src/lib/tensorrtDeps.ts:27-40
Conclusion
The Hardware & Execution Provider system provides a robust abstraction layer that handles the complexities of heterogeneous compute. By combining proactive hardware probing, library verification, and tiered compatibility rules, it ensures that model optimization workflows are both valid and performant for the specific hardware environment.