Performance Metrics & Comparison
Relevant source files
The following files were used as context for generating this wiki page:
Performance Metrics & Comparison within Olive Studio refers to the system's ability to extract, analyze, and display technical efficiency data from Microsoft Olive optimization runs and in-browser inference tests. This module captures critical data points such as latency, throughput, memory usage, and model compression to help developers evaluate the success of their optimization recipes.
The system operates across three primary contexts: real-time extraction from Olive CLI logs, batch job serialization comparisons, and browser-based benchmarking using WebGPU and ONNX Runtime Web. These tools provide a technical audit trail from initial model conversion to final deployment readiness.
Sources: src/components/features/PerformanceMetrics.tsx:8-11, src/components/features/ExecutionWorkspace.tsx:501-510, ABOUT.md:7-12
Metric Extraction Architecture
The core of the performance analysis is the extraction of metrics from raw stdout logs produced by the Olive engine. This is handled by a specialized utility that parses text streams for specific performance indicators.
Log Parsing Logic
The system uses parseOliveMetricsFromLogs to scan execution logs for keywords like "latency", "throughput", "memory", and "compression". This allows the UI to surface structured data even when working with arbitrary CLI output.
The diagram shows how unstructured CLI logs are transformed into structured data for the performance dashboard.
Sources: src/lib/oliveLogMetrics.ts:1-20, src/components/features/PerformanceMetrics.tsx:8-13
Metric Definitions
| Metric | Description | Source Type |
|---|---|---|
| Latency | Time taken for a single inference pass, typically measured in milliseconds (ms). | Olive Logs / Browser Benchmark |
| Throughput | Number of inferences processed per second (it/s). | Olive Logs / Browser Benchmark |
| Memory | VRAM or System RAM footprint during model execution. | Olive Logs / VRAM Estimator |
| Compression | The ratio of the optimized model size compared to the original source. | Olive Logs |
Sources: src/components/features/PerformanceMetrics.tsx:14-25, src/components/features/WebGpuBenchmarkPanel.tsx:28-40
Batch Performance Comparison
Olive Studio allows users to queue multiple optimization jobs to compare how different passes or hardware providers affect performance. This is managed via the Batch Processing Panel, which tracks the history and results of sequential runs.
Sequential Processing Flow
When a batch queue is started, the system executes jobs one-by-one, capturing the final exit code and parsing the resulting logs for comparison.
This sequence details how the frontend monitors job progress and extracts performance data upon completion.
Sources: src/components/features/BatchProcessingPanel.tsx:81-160, src/components/features/ExecutionWorkspace.tsx:432-480
Multi-Job Analytics
The BatchProcessingPanel displays a summary for each job, including its status and key performance indicators (KPIs) if available. This enables at-a-glance comparison between different optimization strategies, such as comparing INT8 quantization against FP16 conversion.
Sources: src/components/features/BatchProcessingPanel.tsx:327-360, src/components/features/BatchProcessingPanel.tsx:442-470
Browser-Based Benchmarking
For models intended for web deployment, the InBrowserValidation and WebGpuBenchmarkPanel components provide direct performance metrics using the local hardware via ONNX Runtime Web.
Benchmarking Parameters
Users can configure benchmark "presets" to determine the rigor of the performance test:
- Quick: 10 iterations, 2 warmup runs.
- Standard: 50 iterations, 5 warmup runs.
- Thorough: 200 iterations, 10 warmup runs.
Sources: src/components/features/WebGpuBenchmarkPanel.tsx:50-55
Latency Distribution Analysis
The system tracks individual iteration latencies to calculate statistical distributions beyond simple averages. This helps identify "jitter" or performance spikes.
The benchmark panel calculates these percentiles to provide a more accurate picture of real-world performance stability.
Sources: src/components/features/WebGpuBenchmarkPanel.tsx:103-112, src/components/features/WebGpuBenchmarkPanel.tsx:260-280
Infrastructure & Memory Estimation
Beyond runtime metrics, the system provides "pre-flight" metrics in the form of VRAM estimates to prevent Out-Of-Memory (OOM) failures before execution starts.
Memory Guidance
The VramEstimateBanner uses catalog presets and hardware probe results to provide rough memory guidance. It classifies compatibility into tiers:
- Compatible: Fit is verified for the detected hardware.
- Unknown: Hardware specs could not be fully probed.
- Unavailable: The model/recipe exceeds detected hardware capabilities.
Sources: src/components/features/InputEnvironmentPanel.tsx:140-165, src/components/features/ExecutionWorkspace.tsx:516-525
Hardware-Specific Performance Factors
The performance metrics module accounts for different Execution Providers (EPs), as shown in the WebGpuBenchmarkPanel. It explicitly tracks which EP was used (e.g., webgpu, wasm) as this is the primary variable affecting latency and throughput in the browser.
Sources: src/components/features/WebGpuBenchmarkPanel.tsx:220-235, src/components/features/InBrowserValidation.tsx:170-185
Conclusion
Performance Metrics & Comparison in Olive Studio provides a feedback loop for model optimization. By combining real-time log parsing from the Olive engine with sophisticated in-browser benchmarking, the system ensures that developers can quantitatively measure the impact of their optimization recipes on target hardware, from server-side CUDA environments to client-side WebGPU deployments.