Broadcasting & Indexing
Relevant source files
The following files were used as context for generating this wiki page:
Broadcasting and Indexing form the architectural backbone of vectorbt, enabling high-performance vectorized backtesting across multiple assets and parameters. The system extends the standard NumPy broadcasting rules to Pandas objects, ensuring that time-series indices and hierarchical columns are correctly aligned, stacked, or tiled during multi-object operations. This allows the library to "think in matrices," processing thousands of strategy configurations simultaneously by packing them into high-dimensional arrays.
The indexing subsystem provides custom accessors (loc, iloc) and metadata-aware selection logic that preserves the relationship between raw data and its labels (timestamps and symbols). By maintaining an internal ArrayWrapper and ColumnGrouper, vectorbt ensures that any slice or transformation of data correctly updates its associated metadata, including frequency information and group structures used for capital sharing.
Data Alignment & Broadcasting Logic
The broadcasting logic in vectorbt is designed to bring multiple objects (scalars, 1D arrays, and 2D frames) to a common shape for vectorized computation. This is primarily handled by the broadcast function and its specialized variants.
The Broadcasting Pipeline
The process involves aligning indices, matching column structures, and expanding shapes. The system uses global settings to determine if indices should be aligned strictly or if columns should be stacked or tiled.
The flow above illustrates how disparate inputs are processed into a unified matrix format.
Sources: vectorbt/base/reshape_fns.py:844-885, vectorbt/_settings.py:126-136
Key Functions and Parameters
The following table summarizes the primary controls for broadcasting behavior:
| Parameter | Type | Description |
|---|---|---|
index_from | str | Source for index alignment: 'strict', 'stack', 'tile', or index name. |
columns_from | str | Source for column alignment: 'strict', 'stack', 'tile', or column name. |
align_index | bool | Whether to join indices if they don't match. |
align_columns | bool | Whether to join columns if they don't match. |
ignore_sr_names | bool | If True, Series names are ignored during column broadcasting. |
Sources: vectorbt/base/reshape_fns.py:756-782, vectorbt/_settings.py:126-135
Metadata Management with ArrayWrapper
The ArrayWrapper class is the central component that bridges raw NumPy arrays with Pandas-style labeling. It stores the shape, index, and columns of the underlying data and provides methods to "wrap" results back into Pandas formats after Numba-accelerated computations.
Indexing and Selection
Vectorbt implements custom indexing via PandasIndexer, which enables metadata-aware selection. When a user selects a subset of columns, the wrapper calculates the new column labels and updates any group information.
Frequency Handling
A critical feature of ArrayWrapper is its ability to parse and validate the frequency (freq) of the data index. This is used throughout the library to calculate performance metrics (like annualization) and to handle time-deltas.
Sources: vectorbt/base/array_wrapper.py:350-410, vectorbt/base/index_fns.py:550-600
Column Grouping and Capital Sharing
The ColumnGrouper class manages relationships between columns, which is essential for multi-asset portfolios where columns might represent different assets belonging to the same trading group.
Grouping Mechanics
Columns can be grouped based on their values, hierarchical levels, or explicit group arrays. This grouping dictates how operations like "sum" or "mean" are applied across asset configurations.
| Component | Description |
|---|---|
group_lens | An array of integers representing the number of columns in each group. |
group_select | Setting that determines if indexing should select columns or entire groups. |
allow_mixed | Whether to allow groups of different sizes. |
Sources: vectorbt/base/column_grouper.py:25-80, vectorbt/base/indexing.py:310-350
The diagram shows how individual columns map to groups for shared resource management.
Sources: vectorbt/base/column_grouper.py:150-200
Index Manipulation Functions
Vectorbt provides a suite of functions in index_fns.py and reshape_fns.py to manipulate Pandas indices and NumPy arrays at a low level to support the high-level broadcasting requirements.
Stacking and Tiling
- Stacking: Repeats the index/array elements multiple times.
stack_indexis used when multiple parameter sets are applied to the same data. - Tiling: Repeats the entire index/array structure.
tile_indexis used to expand data to match the length of a stacked parameter set. - Repeating:
repeat_indexduplicates individual index elements.
Sources: vectorbt/base/index_fns.py:210-280, vectorbt/base/reshape_fns.py:310-380
Flexible Selection
Numba kernels require fast indexing that can handle both 1D and 2D inputs interchangeably. flex_select_auto_nb is a Numba-compatible function that resolves coordinates based on the input's dimensionality and whether it is a scalar.
Sources: vectorbt/base/reshape_fns.py:1050-1100
Configuration and Global Settings
Broadcasting and indexing behaviors are globally configurable through vectorbt.settings.
# Default broadcasting settings in vectorbt/_settings.py
broadcasting = dict(
align_index=False,
align_columns=True,
index_from="strict",
columns_from="stack",
ignore_sr_names=True,
drop_duplicates=True,
keep="last",
drop_redundant=True,
ignore_default=True,
)
Sources: vectorbt/_settings.py:126-136
Summary of key settings:
align_index: If False, indices must match exactly or broadcasting fails (performance optimization).index_from: Determines which object's index is used as the base when shapes differ.columns_from: Determines how column labels are combined (stacked to form MultiIndex or tiled).
Sources: vectorbt/base/reshape_fns.py:760-800, vectorbt/_settings.py:126-135
Conclusion
Broadcasting and Indexing in vectorbt provide the necessary infrastructure for scaling backtests. By abstracting the complexity of aligning time-series data and parameter configurations into a robust metadata-aware system, it allows developers to focus on strategy logic while the library handles the underlying matrix transformations. The combination of ArrayWrapper for metadata preservation and reshape_fns for efficient array manipulation ensures that the library remains both flexible and performant.