Extending Benchmarks

Relevant source files

The following files were used as context for generating this wiki page:

• src/lib/benchmark.ts
• src/WorkbenchContext.tsx
• src/types.ts
• src/components/CenterCanvas.tsx
• src/utils/modelInfo.ts
• src/components/Sidebar.tsx

The benchmarking system in OPTIMUM.WORKBENCH is designed to evaluate the performance of ONNX models across different execution providers (CPU and WebGPU) and hardware configurations. It provides high-fidelity performance metrics, including mean latency, throughput, and estimated memory usage, either through live execution in the browser or via a profiling predictor when direct execution is unavailable.

Extending the benchmarking capabilities involves interfacing with the runBenchmark utility, configuring modality-specific synthetic input generation, and utilizing the WorkbenchContext to manage execution states. The system supports a variety of modalities such as text, vision, and audio, each requiring specific tensor shapes and types for accurate performance profiling.

Sources: src/lib/benchmark.ts:140-224, src/WorkbenchContext.tsx:313-365, src/components/CenterCanvas.tsx:210-220

Core Architecture & Data Flow

The benchmarking process is initiated from the UI, which triggers a sequence of operations starting with environment initialization and ending with the aggregation of performance metrics.

The flow shows the lifecycle of a benchmark request from provider resolution (WASM vs WebGPU) to final metric calculation.
Sources: src/lib/benchmark.ts:83-162

Key Components

Component	Description
runBenchmark	The primary async function that handles session creation, input generation, and performance timing.
generateSyntheticInputs	Logic for creating modality-specific tensors (int64 for text IDs, float32 for vision/audio).
BenchmarkParams	Interface defining input parameters such as modelPath, device, dtype, and shape overrides.
BenchmarkResult	Interface for the returned data, including latency percentiles (p50, p95) and throughput.

Sources: src/lib/benchmark.ts:3-31, src/lib/benchmark.ts:33-81

Execution Environment & Providers

The system dynamically resolves the execution provider based on the hardware target and browser capabilities. It prioritizes webgpu (mapped to the "CUDA" selection in UI) if available, falling back to wasm if initialization fails.

• WASM Provider: Utilizes multi-threaded CPU instructions. The number of threads is automatically set to navigator.hardwareConcurrency.
• WebGPU Provider: Leverages GPU acceleration. The system checks for navigator.gpu support before attempting session creation.
• Optimization Levels: Mapped from UI settings (none, O1, O2, O3, O4) to ONNX Runtime graphOptimizationLevel settings (disabled, basic, extended, all).

Sources: src/lib/benchmark.ts:88-103, src/lib/benchmark.ts:133-145

Input Generation Logic

To benchmark models without requiring real datasets, the system generates synthetic tensors. The shapes and data types are determined by the modality and session.inputNames.

The sequence diagram illustrates how the system dynamically matches synthetic data to the model's required input schema.
Sources: src/lib/benchmark.ts:33-58

Modality-Specific Shapes

• Text: Uses [batchSize, sequenceLength]. Inputs containing "id", "mask", or "token" default to int64.
• Vision: Uses [batchSize, channels, height, width]. Inputs like "pixel_values" use float32.
• Audio: Uses [batchSize, featureSize, frames]. Inputs like "input_features" use float32.

Sources: src/lib/benchmark.ts:47-70

Performance Metrics Calculation

Metrics are derived from 20 timed iterations following 3 warmup runs to ensure JIT optimizations and GPU kernels are fully resident.

• Mean Latency: Simple average of iteration times.
• Percentiles (p50, p95): Calculated by sorting the latencies array and selecting the corresponding indices.
• Throughput: Calculated based on modality units:
  • Text: tokens/sec (based on batchSize * sequenceLength).
  • Vision: images/sec (based on batchSize).
  • Audio: frames/sec (based on batchSize * nbMaxFrames).

Sources: src/lib/benchmark.ts:175-207

State Management & UI Integration

The WorkbenchContext manages the lifecycle of the benchmark, providing abort controllers for cancellation and logging for observability.

Configuration Schema

The benchmark parameters are derived from the ExportConfig object, which includes several shape override fields:

Field	Type	Modality	UI Label
sequence_length	string	text	SEQUENCE_LENGTH
batch_size	string	all	BATCH_SIZE
width/height	string	vision	WIDTH/HEIGHT
feature_size	string	audio	FEATURE_SIZE

Sources: src/types.ts:44-67, src/components/CenterCanvas.tsx:185-205

Cancellation Logic

Benchmarks can be cancelled via the cancelBenchmark function, which triggers an AbortController. This prevents state updates if the user navigates away or stops the process manually.

const cancelBenchmark = () => {
  if (benchmarkAbortControllerRef.current) {
      benchmarkAbortControllerRef.current.abort();
      benchmarkAbortControllerRef.current = null;
  }
  setIsBenchmarking(false);
  addLog('info', 'Benchmark execution cancelled by user.');
};

Sources: src/WorkbenchContext.tsx:196-203

Simulated Benchmarking

When direct browser execution is not possible (e.g., due to CORS restrictions on public assets), the system activates a "Simulated Sandbox" mode. This uses generateCompiledModelInfo to predict latency and throughput based on the model architecture (e.g., number of transformer layers determined by model name) and optimization settings.

Sources: src/utils/modelInfo.ts:8-45, src/components/CenterCanvas.tsx:210-218

The benchmarking module serves as a critical feedback loop in the OPTIMUM.WORKBENCH, allowing developers to validate that their ONNX exports meet performance targets before deployment. By extending the synthetic input generation or adding new execution providers, the workbench can be adapted for specialized hardware or emerging model architectures.