VRAM Estimation
Relevant source files
The following files were used as context for generating this wiki page:
VRAM Estimation in Olive Studio provides developers with heuristic-based insights into the memory requirements of AI models throughout the optimization pipeline. This system estimates memory footprints for both the optimization run (Peak VRAM) and the final deployed artifact (Inference VRAM), comparing these requirements against the user's detected hardware capabilities.
The feature serves as a pre-flight validation tool, alerting users if a specific optimization pass (such as Quantization or LoRA tuning) or a target model is likely to exceed the available GPU VRAM or system RAM. It integrates directly with the hardware probe system to provide real-time feedback on whether a model "fits," is a "tight fit," or has "insufficient" memory.
Sources: src/lib/vramEstimate.ts:4-13, src/components/features/VramEstimateBanner.tsx:90-112, src/lib/aiWorkspaceContext.ts:182-192
Core Architecture and Data Flow
The VRAM estimation logic follows a structured pipeline that transforms model identifiers and user-selected optimization passes into memory metrics.
Estimation Pipeline Flow
The following diagram illustrates how the system resolves model size and applies multipliers based on optimization passes.
The logic first determines the "Base Weight" in GB. For local files, this is derived from actual file sizes. For Hugging Face or Azure models, the system uses regex and name matching to infer parameter counts (e.g., "7b" = 7 billion parameters).
Sources: src/lib/vramEstimate.ts:51-105, src/components/features/InputEnvironmentPanel.tsx:288-305
Key Data Structures
| Type | Field | Description |
|---|---|---|
VramEstimate | peakRunGb | Maximum VRAM/RAM required during the Olive optimization process. |
VramEstimate | inferenceGb | Estimated VRAM/RAM required to run the final optimized model. |
VramEstimate | sourceWeightGb | The initial size of the model weights before optimization. |
VramEstimate | confidence | The reliability of the estimate (high, medium, or low). |
VramEstimate | usesGpu | Boolean indicating if the selected Execution Provider utilizes a GPU. |
| Sources: src/lib/vramEstimate.ts:6-13 |
Memory Calculation Logic
The system calculates memory requirements based on three primary factors: the number of parameters, the data type (dtype) precision, and the overhead introduced by specific optimization passes.
Parameter and Precision Resolution
The number of bytes per parameter is determined by the conversionInputTargetTypes for source weights and quantPrecision for deployed artifacts.
- INT4: 0.5 bytes per parameter
- INT8: 1 byte per parameter
- FP16/BF16: 2 bytes per parameter
- FP32: 4 bytes per parameter
Sources: src/lib/vramEstimate.ts:112-140
Pass Multipliers
To estimate peakRunGb, the system applies multipliers to the source weight to account for calibration buffers, gradients during PEFT, and optimization overhead.
| Pass / Condition | Multiplier / Effect |
|---|---|
| Default Baseline | 1.35x |
| AWQ Quantization | 2.2x |
| QAT (Quantization Aware Training) | 2.5x |
| QLoRA | 2.8x |
| LoRA | 3.2x |
| Pruning | Adds 0.2 to multiplier |
| Sources: src/lib/vramEstimate.ts:153-164 |
Hybrid Memory Pool
When memory offloading is active (e.g., using Hugging Face device_map), the system calculates a hybridPoolGb that combines GPU VRAM and System RAM with safety buffers:
hybridPoolGb = (GPU_VRAM * 0.9) + (System_RAM * 0.75)
Sources: src/lib/vramEstimate.ts:223-225, src/components/features/VramEstimateBanner.tsx:61-63
Hardware Integration and UI Feedback
The VramEstimateBanner component bridges the gap between calculated estimates and physical hardware constraints.
Comparison Logic
The system classifies the "fit" of a model using a comparison between neededGb and availableGb.
Hardware Probing
The system uses the HardwareProbeResult to determine available VRAM. It specifically looks for NVIDIA or ROCm GPUs based on the selected IHVProvider.
- NVIDIA: Extracts
vramMbfromprobe.nvidia.gpus. - ROCM: Extracts
vramMbfromprobe.rocm.gpus.
Sources: src/lib/vramEstimate.ts:197-215
Catalog Preset Estimates
For curated recipes found in the RecipeHub, the system provides pre-calculated estimates based on the metadata associated with the catalog item. If a recipe is tagged with specific model sizes or architectures (e.g., "Llama 3 8B"), the system applies specialized heuristics from presetVramEstimate.ts to show requirements before the user even loads the recipe.
Sources: src/lib/presetVramEstimate.ts, src/components/features/InputEnvironmentPanel.tsx:162-164
Summary
VRAM Estimation in Olive Studio is a heuristic validation layer that ensures optimization workflows are compatible with target hardware. By calculating model footprints across different precisions and applying overhead multipliers for passes like AWQ and QLoRA, it provides developers with critical feedback on memory feasibility. This prevents runtime Out-of-Memory (OOM) errors by identifying potential bottlenecks during both the optimization (Peak) and deployment (Inference) phases.