Hardware Probing System
Relevant source files
The following files were used as context for generating this wiki page:
The Hardware Probing System is a critical diagnostic and decision-making module in Olive Studio designed to detect local machine capabilities and align them with Microsoft Olive optimization workflows. Its primary purpose is to identify available ONNX Runtime Execution Providers (EPs), GPU resources (NVIDIA/AMD), and CPU architectures to prevent the configuration of "impossible" optimization passes before execution.
By probing the local environment, the system provides real-time feedback on hardware compatibility, estimates VRAM usage, and recommends the optimal Execution Provider for a given model. This ensures that users do not attempt to run hardware-specific optimizations, such as NVIDIA TensorRT or Intel OpenVINO, on incompatible systems.
Sources: ABOUT.md:12-15, src/components/features/IHVIntegrationPanel.tsx:142-145
Architecture and Data Flow
The system operates across the frontend (React) and backend (Express/Node.js). The frontend initiates probes through the fetchHardwareProbe utility, which calls the server-side API. The server executes specialized shell commands (e.g., nvidia-smi, rocm-smi) and Python scripts to interrogate the hardware and Python environment.
Probing Workflow
The following diagram illustrates how the system transitions from a raw environment scan to a validated hardware recommendation.
This flow ensures that detection includes both raw hardware (GPU presence) and software readiness (drivers and Python package loadability).
Sources: server.ts:544-633, src/lib/hardwareProbe.ts:56-80
Key Components
Backend Detection Logic
The server implementation in server.ts handles the heavy lifting of environment interrogation. It uses a series of specialized functions to detect different hardware tiers:
- GPU Probing: Uses
execFileto runnvidia-smi(for NVIDIA) androcm-smi(for AMD). It parses CSV output for GPU names, driver versions, and total VRAM. - Python Runtime Probe: Spawns a Python process to import
onnxruntimeand callort.get_available_providers(). It also checks for the availability ofopenvino. - Library Loadability: Specifically checks if
nvinfer_10.dll(TensorRT) ortensorrt_rtxlibraries are loadable viactypesto ensure the Execution Providers are actually functional, not just installed.
Sources: server.ts:494-542, server.ts:581-615
Frontend Integration
The hardware probe results are consumed by several UI components:
- IHVIntegrationPanel: Surfaces detected providers and flags incompatibilities.
- InputEnvironmentPanel: Uses probe data to hide incompatible recipes from the starter catalog.
- VramEstimateBanner: Combines probe data with model metadata to provide memory usage warnings.
Sources: src/components/features/IHVIntegrationPanel.tsx:162-185, src/components/features/InputEnvironmentPanel.tsx:160-175
Data Structures
HardwareProbeResult
The central data structure returned by the API contains exhaustive details about the environment.
| Field | Type | Description |
|---|---|---|
platform | object | OS, Architecture, CPU model, and System RAM (GB). |
nvidia | object | List of GPUs (name, vramMb) and detected CUDA version/tag. |
rocm | object | List of AMD GPUs detected via rocm-smi. |
openvino | object | Availability status and version of the OpenVINO package. |
detectedProviders | string[] | List of functional ONNX Runtime execution providers. |
recommendedProvider | IHVProvider | The system's "best guess" for the primary accelerator. |
Sources: src/lib/hardwareProbe.ts:6-45, server.ts:544-555
Execution Provider Mapping
The system maps detected capabilities to the internal IHVProvider types defined in the PROVIDER_CATALOG.
The selection logic prioritizes hardware in the following order: TensorRT RTX -> TensorRT -> CUDA -> ROCm -> OpenVINO -> CPU.
Sources: src/lib/providerCatalog.ts:4-10, src/lib/hardwareProbe.ts:110-125
API Endpoints
GET /api/system/hardware-probe
Retrieves the full hardware profile of the machine.
- Query Parameters:
refresh(boolean): Iftrue, bypasses the 30-second server cache to re-run all detection scripts.
- Caching: The server maintains a 30,000ms cache (
HARDWARE_PROBE_CACHE_MS) to prevent excessive shell spawning.
Sources: server.ts:635-647
Logic and Calculations
Provider Recommendation
The pickRecommendedProvider function implements a hierarchy of preference for optimization targets:
- NvTensorRTRTXExecutionProvider: Recommended if NVIDIA hardware and the RTX-specific library are detected.
- TensorrtExecutionProvider: Recommended if NVIDIA hardware and the standard TensorRT SDK are functional.
- CUDAExecutionProvider: The default for general NVIDIA GPU acceleration.
- ROCMExecutionProvider: Recommended for AMD hardware.
- OpenVINOExecutionProvider: Recommended for Intel hardware/CPUs if the OpenVINO runtime is present.
- CPUExecutionProvider: The final fallback.
Sources: src/lib/hardwareProbe.ts:110-128
Local Detection Logic
The isProviderDetectedLocally utility determines if a specific EP can run on the current machine. It doesn't just check if the EP is in the detectedProviders list; it also validates underlying hardware requirements (e.g., requiring an NVIDIA GPU for the CUDA provider).
Sources: src/lib/hardwareProbe.ts:85-108
Conclusion
The Hardware Probing System serves as the foundation for the "Hardware-Aware" optimization philosophy of Olive Studio. By automating the detection of GPUs, CUDA versions, and library loadability, it effectively bridges the gap between high-level optimization intent and the low-level constraints of the user's local hardware environment.