Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• tests/conftest.py
• tests/test_utils.py
• tests/test_indicators.py
• tests/test_plotting.py
• rust/README.md
• benchmarks/README.md
• benchmarks/bench_engine.py

The testing strategy for vectorbt focuses on ensuring cross-engine parity, correctness of vectorized trading logic, and regression testing for complex data analysis workflows. The project employs a multi-tiered approach encompassing unit tests for utilities, integration tests for financial indicators, regression tests for plotting, and specialized parity tests to ensure that the optional Rust acceleration engine produces identical results to the reference Numba implementation.

The testing infrastructure leverages pytest for orchestration, and includes specific mechanisms for handling multi-engine environments (Numba vs. Rust). It also utilizes automated benchmarking as a form of "correctness-aware" performance monitoring, ensuring that speed optimizations do not introduce logic discrepancies.

Sources: rust/README.md, benchmarks/README.md, tests/test_plotting.py:1-15

Core Testing Infrastructure

The project uses a sophisticated pytest configuration to handle the optional nature of the Rust engine and to normalize data comparison across different environments.

Engine-Aware Test Selection

The test suite automatically adapts based on whether the vectorbt-rust extension is installed. If Rust is unavailable, tests explicitly requiring the Rust engine are dynamically marked for skipping during collection. This is managed via pytest_collection_modifyitems, which scans test source code for references to engine="rust".

Data Normalization and Compatibility

Because vectorbt processes complex financial time series, standard equality checks for pandas objects can fail due to metadata or subtle index differences. The project overrides pd.testing methods to provide engine-agnostic comparison logic.

Component	Description
normalize_test_index	Standardizes Datetime, Timedelta, and MultiIndex objects to nanosecond precision and object dtypes for string comparisons.
normalize_test_pandas	Handles unit conversion (e.g., to 'ns') for Series and DataFrames to ensure precision matches across engines.
assert_series_equal_compat	A wrapper that attempts standard pandas equality first, falling back to normalized comparison if metadata mismatches occur.

Sources: tests/conftest.py:20-56, tests/conftest.py:110-125

Parity and Regression Tiers

Testing is organized into logical tiers, as evidenced in the plotting and indicator test suites.

Tiered Testing Structure

The regression suite for plotting illustrates the hierarchy of validation:

1. Tier 1 (Factories): Validates basic figure and subplot generation via make_figure.
2. Tier 2 (Core Methods): Checks high-level accessors like OHLCV, Orders, and Drawdowns.
3. Tier 3 (Trace Wrappers): Unit tests for specific visual components like Scatter, Bar, and Heatmap.
4. Tier 4 (Orchestrators): Validates complex plot coordination in Portfolio.plots().
5. Tier 5 (Smoke Tests): Basic "runs without error" verification for all public plotting APIs.

Sources: tests/test_plotting.py:7-15

Multi-Engine Parity

A critical component of the strategy is the validation of the Rust engine against the reference Numba implementation. New Rust kernels must follow a strict process:

1. Treat Numba as the "source of truth".
2. Implement Rust kernel with identical arguments and return shapes.
3. Execute parity tests (assert_same) that compare nested results, including arrays, scalars, and cache dictionaries.

Sources: rust/README.md, benchmarks/bench_engine.py:848-870

Benchmark-Driven Correctness

Benchmarks in vectorbt are not just for performance; they are "correctness-aware microbenchmarks."

The Benchmark Pipeline

The bench_engine.py tool optionally verifies output parity before measuring execution time. This ensures that performance comparisons are only performed on logically equivalent code.

Layout Sensitivity

Tests and benchmarks explicitly account for memory layout modes (view, contiguous, copy-included). This ensures that the testing strategy captures performance and logic edge cases related to non-contiguous 1D column views, which are common in vectorized operations.

Sources: benchmarks/README.md

Specialized Validation Utilities

The project includes a robust set of custom check functions used within tests to validate data structures and function signatures.

Key Validation Functions

• checks.assert_numba_func: Verifies that a function is a compiled Numba kernel.
• checks.assert_engine_func: Validates compatibility with specific engines (auto, numba, rust).
• checks.is_deep_equal: Recursively compares complex structures, including captured values in function closures.
• array_.is_sorted_nb: Performance-optimized sortedness check for arrays.

Sources: tests/test_utils.py:640-675, tests/test_utils.py:535-585, tests/test_utils.py:875-885

Conclusion

The testing strategy of vectorbt is built around the dual requirements of high-performance computation and mathematical correctness. By enforcing engine parity through automated verifiers and utilizing tiered regression testing, the project maintains stability across its diverse ecosystem of pandas accessors, Numba kernels, and Rust extensions. The integration of correctness checks into the benchmarking pipeline further ensures that optimizations do not compromise the integrity of financial analysis.