Rust Precompiled Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Rust Precompiled Engine is an optional acceleration layer for vectorbt that provides high-performance, precompiled alternatives to the library's default Numba-based kernels. It targets performance-critical paths, such as portfolio simulation and large-scale indicator calculations, by eliminating Just-In-Time (JIT) compilation overhead and leveraging Rust's memory safety and execution speed.
Sources: rust/README.md:1-12, README.md:27-30
Architecture and Dispatch Logic
The engine operates through a centralized resolution system located in vectorbt/_engine.py. This system determines whether to use Rust, fall back to Numba, or automatically select the best available engine based on the environment and input data compatibility.
Engine Selection Flow
When a function is called with an engine parameter (or uses the global setting), the resolver evaluates several conditions: availability of the vectorbt-rust package, version compatibility, and support for the specific argument types and shapes.
The diagram represents the decision matrix used by the resolve_engine function to route calls.
Sources: vectorbt/_engine.py:270-305, rust/README.md:58-75
Engine Modes
| Mode | Behavior |
|---|---|
numba | Forces the Numba implementation. |
rust | Forces Rust; raises an error if Rust is unavailable or the specific call/data is unsupported. |
auto | Uses Rust if available and compatible; otherwise falls back to Numba without error. |
Sources: rust/README.md:70-75, vectorbt/_engine.py:270-305
Data Compatibility and Preparation
Rust kernels are deterministic and array-oriented, primarily targeting NumPy arrays. Because Rust is statically typed, the engine requires strict adherence to specific data types (typically float64 for prices/returns and int64 for indices/counts).
Type and Shape Validation
The RustSupport dataclass tracks whether a specific Python call can be offloaded to Rust. Common validation requirements include:
- Array Dimensionality: Rust kernels generally require 1D or 2D arrays.
- Casting Safety: Arrays must be convertible to the target dtype (e.g.,
float32tofloat64) using "safe" casting. - Memory Layout: While strided views are supported via internal preparation, contiguous arrays often provide the best-case baseline.
Sources: vectorbt/_engine.py:44-60, vectorbt/_engine.py:126-140, benchmarks/bench_engine.py:118-124
Flexible Arrays (FlexArray)
To handle broadcasting efficiently between Python and Rust, the engine uses a "compact flexible array" pattern. This allows Rust to receive a smaller array (like a scalar or a single-column array) and broadcast it logically during the simulation loop without allocating the full memory shape in advance.
Sources: vectorbt/_engine.py:171-180, rust/src/portfolio.rs:1040-1065
Portfolio Simulation Engine
The core of the Rust engine is implemented in rust/src/portfolio.rs. It handles complex backtesting logic including order execution, signal processing, and stop-loss management.
Key Data Structures (Rust)
| Struct | Purpose |
|---|---|
Order | Contains order parameters (size, price, fees, slippage, etc.). |
OrderResult | Stores the outcome of an execution (filled size, price paid, status). |
ProcessOrderState | Running state of a simulation (cash, position, debt, value). |
OrderRecord | Structured record of a successfully filled order. |
Sources: rust/src/portfolio.rs:133-165, rust/src/portfolio.rs:205-215, rust/src/portfolio.rs:245-257
Execution Logic
The simulation supports shared cash groups and automated call sequences. In a shared cash group, multiple columns (assets) compete for the same pool of capital.
This diagram illustrates the simulation lifecycle for signal-based backtesting in the Rust engine.
Sources: rust/src/portfolio.rs:1255-1440, rust/src/portfolio.rs:1520-1650
Performance and Benchmarking
The project includes correctness-aware microbenchmarks to compare Numba against Rust. Speedup is calculated as numba_runtime / rust_runtime.
Layout Modes for 1D Inputs
Benchmarks can be run in different memory layout modes to analyze overhead:
- view: Passes strided column views (default, represents real-world usage).
- contiguous: Passes contiguous 1D arrays (best-case kernel baseline).
- copy-included: Includes the cost of copying non-contiguous arrays into the measurement.
Sources: benchmarks/README.md:43-55, benchmarks/bench_engine.py:73-77
Version Compatibility
The engine resolver compares the major and minor version prefixes of vectorbt and vectorbt-rust. If a mismatch is detected (e.g., vectorbt 0.26.0 vs vectorbt-rust 0.25.0), the system automatically falls back to Numba to prevent undefined behavior from binary interface changes.
Sources: vectorbt/_engine.py:83-100, rust/pyproject.toml:6-10
Conclusion
The Rust Precompiled Engine significantly enhances vectorbt's scalability by providing a precompiled execution path that bypasses Python's interpreter and JIT compilation lag. By strictly enforcing data types and utilizing optimized Rust loops for portfolio simulation and record management, it allows for high-throughput backtesting of thousands of strategy configurations simultaneously. Its transparent integration via the engine resolver ensures that users can toggle between engines with minimal code changes while maintaining output parity with the reference Numba implementations.