Data Feeds Management

Relevant source files

The following files were used as context for generating this wiki page:

• backtrader/feeds/btcsv.py
• backtrader/feeds/yahoo.py
• backtrader/feeds/vchart.py
• backtrader/feeds/csvgeneric.py
• backtrader/filters/datafiller.py
• backtrader/feed.py

Introduction

Data Feeds Management is a core component of the backtrader framework, responsible for ingesting, parsing, and providing a standardized stream of market data to strategies. The system is designed to handle diverse data sources, including local CSV files, binary formats like Visual Chart, and live data through various brokers or online providers such as Yahoo Finance.

The management system abstracts the underlying data source into "lines"—primarily consisting of Open, High, Low, Close, Volume, and OpenInterest—allowing strategies to remain agnostic of the data's origin. It supports advanced features such as data filtering, filling missing gaps, and multi-timeframe synchronization. Sources: README.md, backtrader/feeds/btcsv.py, backtrader/feeds/yahoo.py

Core Architecture and Data Flow

The architecture relies on a hierarchy of classes starting from a base feed and DataBase abstraction. Every data feed must implement methods to initialize the connection (start), fetch new lines (_load), and tear down resources (stop).

Data Feed Component Hierarchy

The following diagram illustrates the relationship between various data feed implementations and their base classes.

This hierarchy allows for specialized parsing logic while maintaining a uniform interface for the Cerebro engine. Sources: backtrader/feeds/csvgeneric.py:34-40, backtrader/feeds/yahoo.py:38-42, backtrader/feeds/vchart.py:32-35

General Logic Flow

The data flow typically follows this sequence:

1. Initialization: Parameters such as dataname, fromdate, and todate are passed to the feed.
2. Setup: The start() method opens files or establishes network connections.
3. Iteration: The engine calls _load() (or _loadline() for CSVs) repeatedly to populate the current bar.
4. Filtering: If active, filters like DataFiller intercept the data to address gaps.

CSV Data Feeds

CSV handling is highly flexible, supporting generic formats, Yahoo-specific structures, and internal testing formats.

Generic CSV Parameters

The GenericCSVData class allows mapping CSV columns to specific data lines using indices.

Parameter	Type	Default	Description
datetime	int	0	Column index for Date/Time
open	int	1	Column index for Open price
high	int	2	Column index for High price
low	int	3	Column index for Low price
close	int	4	Column index for Close price
volume	int	5	Column index for Volume
openinterest	int	6	Column index for Open Interest
dtformat	str	%Y-%m-%d %H:%M:%S	Format string for parsing dates

Sources: backtrader/feeds/csvgeneric.py:36-48

Yahoo Finance Data

The Yahoo Finance implementation includes specialized logic for handling dividend/split adjustments. If adjclose is set to True, the feed automatically adjusts the Open, High, Low, and Close prices based on the ratio between the close price and the adjusted close price. Sources: backtrader/feeds/yahoo.py:108-135

Specialized Binary and Store Feeds

Beyond text-based files, backtrader supports binary formats and store-based (broker) data integration.

Visual Chart (VChart)

VChartData parses binary .fd (daily) or .min (intraday) files. It uses the struct module to unpack binary bars according to specific formats:

• Daily: 28 bytes per bar, format IffffII.
• Intraday: 32 bytes per bar, format IIffffII.

Sources: backtrader/feeds/vchart.py:72-85

Interactive Brokers (IB) Integration

IBCSVData facilitates a hybrid approach where data is derived from CSV files but linked to an IBStore. It parses ticker strings into IB Contract objects using sectype, symbol, currency, and exchange tokens. Sources: backtrader/feeds/btcsv.py:145-185

Data Management and Filtering

Filters modify the data stream before it reaches the strategy. A prominent example is the DataFiller, which ensures continuity in time-series data.

Data Filling Logic

The DataFiller identifies gaps between the previous bar's timestamp and the current bar's timestamp. If a gap exists, it generates synthetic bars using the closing price of the last known bar and user-defined volume/open interest values.

• Boundary Checking: It calculates session starts and ends to avoid filling across non-trading hours.
• Time Deltas: It uses timedelta objects mapped to the timeframe (e.g., 60 seconds for Minutes) and compression factor.

Sources: backtrader/filters/datafiller.py:118-166

Conclusion

Data Feeds Management in backtrader provides a robust framework for normalizing diverse market data into a standardized interface. By utilizing a common DataBase abstraction, the system supports everything from simple CSV parsing to complex binary decoders and live broker integrations, while filters like DataFiller ensure the data integrity required for accurate backtesting and live execution. Sources: backtrader/feed.py, backtrader/feeds/vchart.py, backtrader/filters/datafiller.py