Model Benchmarking
Relevant source files
The following files were used as context for generating this wiki page:
The Model Benchmarking system is a core performance evaluation module within the ONNX-Station workbench. It provides developers with the ability to measure inference performance of ONNX models directly in the browser using the ONNX Runtime (ORT). The system supports benchmarking both public models from the Hugging Face Hub and local .onnx files, providing metrics such as latency, throughput, and estimated memory usage.
Sources: src/lib/benchmark.ts:1-29, src/WorkbenchContext.tsx:340-345
Benchmark Architecture and Flow
The benchmarking process is managed via the WorkbenchContext and executed by the runBenchmark function. It involves resolving the execution provider (WASM or WebGPU), loading model binaries, generating synthetic input tensors based on the model's modality, and performing timed inference iterations.
Core Execution Logic
The following diagram illustrates the sequence of operations performed during a benchmark run:
The system performs 3 warmup iterations to stabilize the environment before executing 20 timed iterations to calculate statistical averages.
Sources: src/lib/benchmark.ts:101-220, src/WorkbenchContext.tsx:340-394
Metric Generation and Data Structures
The system calculates several key performance indicators (KPIs) to help developers evaluate the efficiency of their ONNX models.
Benchmark Result Metrics
| Metric | Description | Derived From |
|---|---|---|
| Mean Latency | The average time taken for a single inference pass in milliseconds. | Sum of latencies / iterations |
| p50/p95 Latency | Median and 95th percentile latencies to identify performance tail variance. | Sorted latency array |
| Throughput | Inference speed measured in units per second (tokens/sec, images/sec, or frames/sec). | batchSize / (meanLatency / 1000) |
| Est. Memory | Estimated memory footprint based on model size (calculated as 1.5x model bytes). | modelSizeBytes |
| File Size | The physical size of the ONNX model binary. | Content-Length header or local file size |
Sources: src/lib/benchmark.ts:25-36, src/lib/benchmark.ts:223-255, src/components/CenterCanvas.tsx:238-243
Input Tensor Synthesis
A critical component of the benchmarking engine is the generateSyntheticInputs function. Since ONNX-Station benchmarks models without requiring specific datasets, it dynamically generates synthetic tensors that match the expected input shapes and types for different modalities.
The logic differentiates between int64 types (often used for token IDs and masks in text models) and float32 types (used for pixel values or spectral features).
Sources: src/lib/benchmark.ts:38-99
Configuration Parameters
Benchmarks are influenced by the ExportConfig defined in the workbench, which dictates the hardware target and model precision.
Key Benchmark Parameters
| Parameter | Type | Impact on Benchmark |
|---|---|---|
device | cpu | cuda | Determines if wasm or webgpu execution provider is used. |
dtype | fp32 | fp16 | bf16 | Influences memory bandwidth and kernel execution speed. |
optimize | none to O4 | Sets the ORT graphOptimizationLevel (disabled, basic, extended, or all). |
modality | text | vision | audio | Determines the shape and units of synthetic inputs and throughput. |
Sources: src/lib/benchmark.ts:4-16, src/types.ts:40-48, src/lib/benchmark.ts:162-167
Error Handling and Simulation
The benchmark runner includes robust fallback mechanisms:
- Provider Fallback: If WebGPU initialization fails for a
cudatarget, the system automatically degrades to the WASM provider to ensure the benchmark can still run. - Sandbox Simulation: For models that cannot be fetched due to CORS restrictions or non-public assets, the workbench activates a high-fidelity profiling predictor that estimates performance based on hardware and precision configurations.
- Path Resolution: The system automatically resolves Hugging Face paths to standard ONNX locations (e.g.,
resolve/main/onnx/model.onnx).
Sources: src/lib/benchmark.ts:114-125, src/lib/benchmark.ts:175-188, src/components/CenterCanvas.tsx:250-254
Summary
The Model Benchmarking system provides an integrated performance profiling environment. By combining dynamic synthetic input generation with direct ONNX Runtime integration, it allows developers to obtain immediate performance feedback on their optimization choices—such as changing Opset versions, applying graph fusions, or overriding precision—without leaving the browser.
Sources: src/lib/benchmark.ts:213-257, src/WorkbenchContext.tsx:405-410