GPU Launcher & Dependencies

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/olive_gpu_launcher.py
• src/lib/oliveGpuRuntime.ts
• server.ts
• src/lib/tensorrtDeps.ts
• src/lib/tensorrtRtxDeps.ts
• README.md

The GPU Launcher and its associated dependency management system constitute the core execution backend for hardware-accelerated model optimization in Olive Studio. This system automates the setup of specialized Python environments, manages vendor-specific GPU libraries (NVIDIA CUDA, TensorRT), and ensures that the Microsoft Olive CLI runs with the correct environmental overrides to access local hardware accelerators.

The system is designed to be "local-first," probing the user's machine for available ONNX Runtime (ORT) execution providers and automatically installing required wheels into a project-local virtual environment (.venv) upon the first execution request.

Sources: README.md:14-17, server.ts:219-254, ABOUT.md:12-15

Environment Management & Initialization

Olive Studio manages its own Python lifecycle to ensure stability and isolation from system-wide Python installations. The server handles the creation of a .venv and the installation of the base olive-ai package before any optimization passes are executed.

Virtual Environment (venv) Workflow

When a user triggers a "Live Execution," the server initiates a sequential setup process:

1. System Python Probe: Searches for python or python3 on the system PATH.
2. Venv Creation: If missing, it creates a local .venv folder.
3. Core Installation: Installs olive-ai and requests.
4. Status Streaming: All setup progress is streamed to the frontend via Server-Sent Events (SSE) to provide real-time feedback.

Sources: server.ts:219-265, README.md:85-92

GPU Dependency Management

Olive Studio utilizes a "pinned" dependency strategy to avoid version mismatches between ONNX Runtime, CUDA, and TensorRT. These pins are centralized in TypeScript library files and used during the pip install phase.

Pinned Package Specifications

Category	Package / Argument	Source
ONNX Runtime	onnxruntime-gpu==1.20.1	src/lib/oliveGpuRuntime.ts:16
Classic TensorRT	tensorrt==10.9.0.34	src/lib/tensorrtDeps.ts:15
TensorRT RTX	tensorrt-rtx	src/lib/tensorrtRtxDeps.ts:10
CUDA 12 Runtime	nvidia-cudnn-cu12, nvidia-cublas-cu12	src/lib/oliveGpuRuntime.ts:23-28

Sources: src/lib/oliveGpuRuntime.ts:16-30, src/lib/tensorrtDeps.ts:15-20, server.ts:405-412

CUDA Version Detection

The system automatically detects the appropriate CUDA version to install based on the local hardware:

• Auto-detection: Executes nvidia-smi to parse the driver's reported CUDA version.
• Tag Selection: Maps versions to specific PyPI index URLs (e.g., cu121, cu124, cu118).
• Fallback: Defaults to cpu if no NVIDIA GPU is detected.

Sources: server.ts:602-632

The GPU Launcher Script

The scripts/olive_gpu_launcher.py script serves as a wrapper around the standard olive run command. It is used specifically when a recipe targets a GPU execution provider (CUDA or TensorRT) to solve path registration and DLL loading issues, particularly on Windows.

Implementation Logic

The launcher performs the following critical steps before calling the Olive CLI:

1. Path Resolution: It locates the .venv and site-packages.
2. DLL Preloading: It scans for native GPU libraries (e.g., nvinfer_10.dll, cudnn64_9.dll) within the virtual environment.
3. OS PATH Extension: It temporarily prepends the directories containing these DLLs to the operating system's PATH environment variable.
4. Execution: Spawns the Olive process as a subprocess, ensuring it inherits the corrected environment.

Sources: scripts/olive_gpu_launcher.py, server.ts:446-455

Hardware-Specific Runtimes

Olive Studio provides specialized support for different NVIDIA software stacks.

TensorRT (Classic)

Targeted at data center and general-purpose TensorRT workflows. It requires the nvinfer_10.dll library.

• Verification: The server runs a Python probe attempting to ctypes.CDLL the TensorRT library before starting a run.
• Path Setup: Prepends tensorrt_libs directory to the system PATH.

Sources: server.ts:503-535, src/lib/tensorrtDeps.ts:32-35

TensorRT RTX

Specialized for consumer GeForce RTX hardware using the NvTensorRTRTXExecutionProvider.

• Package: Uses tensorrt-rtx.
• Constraint: Primarily targets Windows environments for consumer AI applications.

Sources: src/lib/tensorrtRtxDeps.ts:1-15, server.ts:566-575

Summary

The GPU Launcher & Dependency system ensures that Olive Studio can reliably execute complex model optimizations on local hardware without manual configuration by the user. By pinning dependencies like onnxruntime-gpu and tensorrt, and utilizing the olive_gpu_launcher.py script to manage native library paths, the project provides a robust bridge between the web-based UI and the underlying Python-based optimization engine.

Sources: README.md:130-138, server.ts:468-475