Data Fetching & Processing

Relevant source files

The following files were used as context for generating this wiki page:

• vectorbt/data/base.py
• vectorbt/generic/accessors.py
• vectorbt/generic/stats_builder.py
• vectorbt/generic/plots_builder.py
• apps/candlestick-patterns/app.py

The Data Fetching & Processing system in vectorbt is designed to manage, download, and transform financial data at scale. It leverages a vectorized approach, packing data into NumPy and Pandas structures to enable high-performance analysis of thousands of configurations simultaneously. The system provides a unified interface for various data sources, ensuring data alignment across different symbols and timeframes.

At its core, the system utilizes the Data class to handle the lifecycle of financial datasets, including initial downloading, incremental updates, and complex processing through specialized accessors. This architecture allows human researchers and AI agents to perform large-scale parameter sweeps and backtesting with minimal overhead.

Data Management Architecture

The architecture revolves around the Data class, which acts as a container for Series or DataFrames keyed by symbol. It inherits from Wrapping, StatsBuilderMixin, and PlotsBuilderMixin to provide integrated statistical analysis and visualization capabilities.

Core Components

Component	Description
Data Class	Base class for storing, downloading, and updating data keyed by symbol.
ArrayWrapper	Manages index and column metadata for aligned data operations.
symbol_dict	A specialized dictionary used to pass symbol-specific keyword arguments.
GenericAccessor	Provides a pandas-native API for advanced vectorized operations (e.g., rolling/expanding windows).

Sources: vectorbt/data/base.py:214-230, vectorbt/generic/accessors.py:186-215

Data Flow for Downloading and Alignment

When data is requested for multiple symbols, vectorbt ensures that all pandas objects share a common index and column structure through alignment functions.

The alignment process can handle missing data points by setting them to NaN, dropping them, or raising an error based on configuration.
Sources: vectorbt/data/base.py:270-330, vectorbt/data/base.py:440-470

Data Transformation & Vectorization

The system utilizes custom pandas accessors via the vbt namespace to perform high-performance transformations. These operations are often dispatched to Numba or Rust engines for maximum speed.

Accessor Operations

The GenericAccessor allows for complex data manipulation without manual loops:

• Rolling/Expanding Windows: Methods like rolling_mean, expanding_std, and ewm_mean are implemented using vectorized kernels.
• Reductions: Custom reduction functions can be applied across groups of data.
• Crossovers: Generating signals from crossovers (e.g., fast_ma.ma_crossed_above(slow_ma)).

Statistical Analysis & Metrics

The StatsBuilderMixin enables the computation of various metrics on fetched data. It supports grouping and column selection to analyze specific segments of a dataset.

Metrics Configuration

Metric	Calculation Basis
Start	The first timestamp in the index.
End	The last timestamp in the index.
Period	The total duration/length of the index.
Total Symbols	Number of unique symbols in the data.
Null Counts	Count of NaN values per symbol.

Sources: vectorbt/data/base.py:614-635, vectorbt/generic/stats_builder.py:48-65

Processing Logic for Stats

Application Example: Candlestick Pattern Detection

The technical implementation of data fetching is visible in specialized applications such as the Candlestick Patterns app. This application demonstrates the flow from a data source (Yahoo! Finance) to pattern recognition using TA-Lib.

Data Processing Pipeline

1. Fetch: Data is retrieved via yfinance based on user-defined symbols and intervals.
2. Filter: The data is sliced using a date_slider.
3. Recognize: TA-Lib functions are mapped over the OHLC arrays to detect specific patterns.
4. Signal Generation: Bullish and Bearish patterns are converted into entry/exit signals for the Portfolio class.

# Example of data fetching in Candlestick Patterns app
def fetch_data(symbol, period, interval, auto_adjust, back_adjust):
    df = yf.Ticker(symbol).history(
        period=period,
        interval=interval,
        auto_adjust=auto_adjust,
        back_adjust=back_adjust
    )
    return df

Sources: apps/candlestick-patterns/app.py:440-455, apps/candlestick-patterns/app.py:518-540

Data Updating & Persistence

The system supports incremental updates to existing datasets, allowing users to keep their analysis current without re-downloading entire histories.

Update Mechanism

The update method uses the original download_kwargs to fetch only the missing data points starting from the last available timestamp. The old and new data are then concatenated, with duplicates handled based on the latest index entry.

Conclusion

Data Fetching & Processing in vectorbt provides a robust foundation for high-performance trading research. By combining strictly aligned data structures with vectorized accessors and flexible metric builders, the system allows for the seamless transition from raw data retrieval to complex strategy backtesting. This integrated approach ensures that data integrity is maintained across thousands of concurrent configurations, making it suitable for both manual exploration and automated AI-driven workflows.