Benchmarking Engine Library
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The Benchmarking Engine Library is a core component of the ONNX-Station workbench designed to evaluate the performance of ONNX models directly within the browser environment. By leveraging onnxruntime-web, the library enables developers to measure critical execution metrics such as mean latency, throughput, and estimated memory usage across various hardware targets including CPU (via WASM) and GPU (via WebGPU).
The library supports both live execution for CORS-compliant models or local files and high-fidelity simulated profiling when direct execution is restricted. It integrates deeply with the workbench's configuration system, allowing for performance validation across different modalities (text, vision, audio) and optimization levels.
Sources: src/lib/benchmark.ts:1-35, src/WorkbenchContext.tsx:251-265, src/utils/modelInfo.ts:5-15
Core Architecture and Execution Flow
The benchmarking system follows a multi-stage pipeline: resolving execution providers, fetching model binaries, generating synthetic input tensors, and performing timed inference iterations.
Performance Execution Pipeline
The library first determines the best available execution provider. While it attempts to use WebGPU for CUDA-targeted configurations, it gracefully degrades to WASM if the browser lacks WebGPU support or initialization fails.
The diagram shows the sequential stages of the benchmarking process from provider resolution to metric calculation.
Sources: src/lib/benchmark.ts:117-236
Input Generation Logic
Inputs are dynamically generated based on the model's metadata (session.inputNames) and the user-provided configuration. The engine maps modality-specific dimensions (e.g., sequence length for text, image dimensions for vision) to the required tensor shapes.
| Modality | Default Shape Dimensions | Default Data Type |
|---|---|---|
| Text | [batchSize, sequenceLength] | int64 (ids/masks) or float32 |
| Vision | [batchSize, channels, height, width] | float32 |
| Audio | [batchSize, featureSize, nbMaxFrames] | float32 |
Sources: src/lib/benchmark.ts:37-90
Data Structures and Interfaces
Benchmark Configuration (BenchmarkParams)
Defines the parameters required to initialize and run the benchmark sequence.
Sources: src/lib/benchmark.ts:3-17, src/types.ts:40-75
Benchmark Results (BenchmarkResult)
Contains the execution metrics returned to the UI after a successful run.
| Field | Type | Description |
|---|---|---|
status | 'ok' | 'error' | Execution status of the benchmark sequence. |
meanLatencyMs | number | Average time per inference across 20 iterations. |
p50LatencyMs | number | Median latency (50th percentile). |
p95LatencyMs | number | Tail latency (95th percentile). |
throughput | number | Items processed per second (tokens, images, or frames). |
memoryUsageMb | number | Estimated VRAM/RAM footprint of the model. |
isSimulated | boolean | Flag indicating if metrics are estimated via profiling logic. |
Sources: src/lib/benchmark.ts:19-31
Simulated Profiling Engine
When live execution is not possible (e.g., CORS restrictions on public assets), the library utilizes a high-fidelity predictor to estimate performance. This engine adjusts metrics based on:
- Model Architecture: Detected via
modelPathkeywords (Whisper, Llama, BERT, ViT, etc.). - Optimization Levels: Scales operator counts and memory based on O1-O4 optimization flags.
- Precision: Applies
dtypeScale(e.g., 0.5x for fp16/bf16) to memory and bandwidth calculations.
This diagram illustrates how the simulation logic derives model info without direct binary execution.
Sources: src/utils/modelInfo.ts:10-150
Integration with Workbench
The library is consumed by the WorkbenchProvider, which manages the state of the benchmarking process.
Sequence Diagram: Benchmark Execution
The sequence diagram details the interaction between the UI, the workbench context, and the underlying benchmark library.
Sources: src/WorkbenchContext.tsx:251-305, src/components/CenterCanvas.tsx:185-225
Conclusion
The Benchmarking Engine Library provides a robust framework for performance analysis within ONNX-Station. By combining actual runtime execution via onnxruntime-web with a sophisticated simulation fallback, it ensures developers have access to actionable performance data regardless of environment constraints. Its modular design allows for the calculation of complex metrics like p95 latency and modality-specific throughput, making it an essential tool for optimizing ONNX deployments.
Sources: src/lib/benchmark.ts:220-245, src/utils/modelInfo.ts:180-195