Benchmarking Suite
Relevant source files
The following files were used as context for generating this wiki page:
The Benchmarking Suite in vectorbt provides high-performance tooling designed to compare the execution efficiency of Numba-compiled Python kernels against their optional Rust-based counterparts. These benchmarks are categorized as correctness-aware microbenchmarks, meaning they not only measure execution time but also verify output parity between the two engines using deterministic inputs. The suite is essential for validating performance improvements introduced by the vectorbt-rust extension and ensuring that the Rust engine remains a viable drop-in replacement for Numba in production environments.
The suite generates comprehensive reports in both CSV and Markdown formats. These reports highlight "speedup" values, defined as the ratio of Numba execution time to Rust execution time ($speedup = numba_s / rust_s$). Values above 1.00x indicate that the Rust implementation is faster, while values below 1.00x favor Numba. The suite supports multiple memory layout modes and test suites, ranging from core logic to extended investigation cases.
Sources: benchmarks/README.md:3-12, benchmarks/README.md:89-98, rust/README.md:1-12
Architecture and Components
The benchmarking system is built around two primary scripts: bench_engine.py, which handles individual function comparisons and correctness checks, and bench_matrix.py, which automates the execution of these benchmarks across a variety of input configurations (rows vs. columns) to produce performance matrices.
Benchmark Runner Flow
The following diagram illustrates the lifecycle of a benchmark execution for a single pair of Numba and Rust functions.
The runner ensures that memory overhead—such as copying non-contiguous arrays—can be either isolated or included in the measurement through specific layout modes. This allows developers to distinguish between raw kernel speed and data preparation costs.
Sources: benchmarks/bench_engine.py:32-40, benchmarks/bench_engine.py:1107-1115, benchmarks/README.md:14-22
Data Flow and Matrix Generation
The suite is designed to evaluate performance across a scaling spectrum of data sizes. The bench_matrix.py script orchestrates multiple runs of the engine benchmark using pre-defined configurations ranging from small sets ($100 \times 1$) to large-scale matrices ($100,000 \times 10$).
Input Configurations
| Configuration Label | Rows | Columns | Description |
|---|---|---|---|
| 100x1 | 100 | 1 | Smallest test case |
| 10Kx1 | 10,000 | 1 | Medium single-column |
| 100Kx1 | 100,000 | 1 | Large single-column |
| 1Kx100 | 1,000 | 100 | High column width scenario |
| 10Kx100 | 10,000 | 100 | Maximum scale matrix |
Sources: benchmarks/bench_matrix.py:20-33
Statistics Calculation
For every configuration, the system calculates descriptive statistics including minimum, median, mean, and maximum execution times. The best measured time (minimum) after warmup is typically used to represent the steady-state performance of the kernel.
Sources: benchmarks/bench_matrix.py:73-82, benchmarks/README.md:95-97
Core Benchmark Suites and Layouts
Benchmarks are categorized into suites to allow targeted testing and provide layout options that mimic real-world usage patterns.
Layout Modes
The memory layout of 1D column inputs significantly impacts performance, particularly for strided operations.
view: Passes strided column views. This is the default as it represents common vectorbt usage where columns are sliced from a 2D matrix.contiguous: Passes C-contiguous 1D arrays. This represents the best-case baseline for kernel performance.copy-included: Includes the cost of copying non-contiguous arrays into the timed call.
Sources: benchmarks/README.md:46-56, benchmarks/bench_engine.py:44-48
Benchmark Suites
Cases are tagged with metadata to filter them based on the scope of investigation:
core: The default suite. It excludes O(1) operations, metadata-only lookups, and fixed-input cases to focus on computationally intensive kernels.extended: Includes all available test cases, including scalar and metadata lookups.
Sources: benchmarks/README.md:61-71, benchmarks/bench_engine.py:53-56
Implementation Details
BenchmarkCase Structure
Each benchmark is defined by a BenchmarkCase dataclass, which maps a Numba function to its Rust equivalent along with their respective arguments.
@dataclass(frozen=True)
class BenchmarkCase:
name: str
nb_func: Callable
nb_args: tuple
rs_func: Callable
rs_args: tuple
check: bool = True
tags: tuple[str, ...] = ()
Sources: benchmarks/bench_engine.py:32-41
Parity Verification
Output verification is performed using np.testing.assert_allclose with a relative tolerance of 1e-6 and absolute tolerance of 1e-12. This ensures that floating-point arithmetic differences between Numba/LLVM and Rust/LLVM remain within acceptable bounds. The verification logic recursively handles dictionaries, tuples, and nested structures to support complex indicator caches.
Sources: benchmarks/bench_engine.py:1082-1104
Rust Implementation Efficiency
The Rust kernels evaluated by this suite (implemented in rust/src/portfolio.rs and other modules) utilize low-level memory operations, such as raw pointer writes for log records and specialized math utilities for numerical stability.
#[inline(always)]
fn is_close(a: f64, b: f64) -> bool {
if a.is_nan() || b.is_nan() { return false; }
if a == b { return true; }
(a - b).abs() <= ABS_TOL.max(REL_TOL * a.abs().max(b.abs()))
}
Sources: rust/src/portfolio.rs:60-68
Conclusion
The vectorbt Benchmarking Suite is a critical infrastructure component that enables data-driven development of the library's performance engines. By providing a structured way to measure and verify speedups, it ensures that the transition to Rust for computationally expensive operations (like portfolio simulation and technical indicators) yields tangible performance gains without compromising the accuracy of financial calculations. Developers can use this suite to identify performance bottlenecks and validate that new Rust kernels outperform the existing Numba implementations across varying data scales.