Records Management
Relevant source files
The following files were used as context for generating this wiki page:
Records Management in vectorbt is a high-performance framework designed to represent complex, sparse information in a dense, memory-efficient format. While the library primarily uses matrices for uniform feature data, records allow for the storage of structured data where multiple entries (such as trades or orders) can map to a single element in a time-series matrix. This representation avoids the memory waste of large, sparse matrices and provides a fixed schema for diverse attributes.
Sources: vectorbt/records/base.py:5-41
The system is built around the Records class and its subclasses, which wrap structured NumPy arrays. It utilizes a "MapReduce" paradigm to process these arrays without converting them back to dense matrices, enabling scalable analysis of trading strategies and signals. The infrastructure supports mapping, filtering, grouping, and indexing, with specialized implementations for ranges, trades, and portfolio logs.
Sources: vectorbt/records/base.py:85-110, vectorbt/records/mapped_array.py:34-70
Architecture and Core Components
The records management system follows a hierarchical structure where generic records are specialized into domain-specific types like trading data or time-range events.
Class Hierarchy
The core of the system is the Records class, which inherits from Wrapping (to handle metadata and column alignment) and mixins for statistics and plotting.
The diagram shows the inheritance from base metadata wrappers to specific record types.
Sources: vectorbt/records/base.py:270-280, vectorbt/generic/ranges.py:116-120, vectorbt/portfolio/trades.py:108-115
Key Data Structures
The system relies on structured NumPy arrays with specific fields to maintain identity and alignment.
| Field | Description | Type |
|---|---|---|
id | Unique identifier for the record. | np.int64 |
col | The column (feature/asset) index the record belongs to. | np.int64 |
idx | The timestamp or row index in the original time-series. | np.int64 |
Sources: vectorbt/records/base.py:46-51, vectorbt/records/base.py:291-298
Column Mapping and Alignment
The ColumnMapper is a critical internal component that manages the relationship between the flat records array and the column structure defined by the ArrayWrapper.
This flow illustrates how ColumnMapper optimizes data retrieval based on the sort state of the records.
Sources: vectorbt/records/base.py:330-348, vectorbt/records/col_mapper.py
The ColumnMapper identifies whether records are sorted by column. If sorted, it uses range-based indexing (slices) for extremely fast access. If unsorted, it maintains a map to retrieve records corresponding to specific asset columns.
Sources: vectorbt/records/base.py:433-445
Mapping and Reduction (MapReduce)
The system transforms raw records into values for analysis using the MappedArray class.
The Mapping Process
Mapping extracts a single attribute from records or applies a function to each record to produce a one-dimensional array of values aligned with the record IDs.
map_field: Extracts a specific field from the structured array.map: Applies a Numba-jitted function to each individual record.apply: Applies a function to records grouped by column or asset.
Sources: vectorbt/records/base.py:110-138, vectorbt/records/base.py:488-531
Reduction Operations
MappedArray provides high-performance reduction methods (mean, std, sum, etc.) that can operate on groups of columns.
| Method | Description |
|---|---|
reduce | Custom reduction using Numba-jitted functions. |
to_pd | Converts the sparse records back into a dense Pandas Series or DataFrame. |
value_counts | Performs categorical analysis on mapped values. |
Sources: vectorbt/records/mapped_array.py:34-60, vectorbt/records/mapped_array.py:438-510
Specialized Record Types
Range Records (Ranges)
Ranges records capture duration-based information, defined by a start index, an end index, and a status (Open or Closed). They are used to analyze intervals like signal distances or drawdown periods.
Sources: vectorbt/generic/ranges.py:5-30
Trade Records (Trades)
Trades extend Ranges by adding financial metrics. They include entry/exit prices, fees, PnL, and returns.
EntryTrades: Created for every order that adds to a position.ExitTrades: Created for every order that reduces or closes a position.Positions: Aggregated sequences of entry and exit trades representing the full lifecycle of a holding.
Sources: vectorbt/portfolio/trades.py:5-40, vectorbt/portfolio/trades.py:650-710
Log Records (Logs)
Logs capture the internal state of a portfolio simulation, including requested vs. filled sizes, cash levels, and rejection reasons.
Sources: vectorbt/portfolio/logs.py:5-30
Performance Optimization
The system leverages several techniques to maintain speed at scale:
- Numba Integration: Most mapping and reduction operations are executed via Numba-jitted kernels for C-level speed.
- Caching: Computations like
is_sortedor statistics are cached usingcached_methodandcached_propertydecorators. - Sorted Optimizations: Operations on records sorted by column use optimized dispatch functions (
record_col_range_select) that avoid full array scans.
Sources: vectorbt/records/base.py:228-235, vectorbt/records/base.py:433-445, vectorbt/records/mapped_array.py:270-285
Conclusion
Records Management provides the flexible, dense storage layer necessary for vectorbt's large-scale backtesting engine. By decoupling structured data from the time-series matrix and utilizing a Numba-accelerated MapReduce pipeline, the system allows for the analysis of millions of trades or signals with minimal memory overhead and high computational efficiency.
Summary of Significance:
- Dense storage of sparse event data.
- Hierarchical support for trade and range analysis.
- Seamless integration with Pandas and Numba.
Sources: vectorbt/records/base.py:5-25, vectorbt/records/mapped_array.py:5-30