Historical Data Backfills

Relevant source files

The following files were used as context for generating this wiki page:

• hyperdrive/DataSource.py
• hyperdrive/History.py
• scripts/update_dividends.py
• scripts/upload_data.py
• test/test_DataSource.py
• README.md

Historical Data Backfills in the Hyperdrive library refer to the systematic process of retrieving, standardizing, and persisting long-term market data from various providers like IEXCloud and Polygon.io. This system is designed to provide a robust foundation for quantitative research and backtesting by maintaining local or cloud-based (S3) caches of Open-High-Low-Close (OHLC), intraday movements, dividends, and corporate splits.

The backfill mechanism leverages a multi-provider strategy to ensure data redundancy and completeness. It specifically focuses on "storing data" rather than just transient retrieval, checking for existing cached data to correct inconsistencies in values and column names before updating files.

Sources: README.md:8-13, hyperdrive/DataSource.py:12-20, scripts/update_dividends.py:16-20

Architecture and Data Flow

The backfill system is built around the MarketData base class and its provider-specific subclasses (IEXCloud, Polygon). These classes coordinate with FileWriter, FileReader, and PathFinder components to manage the lifecycle of historical data.

Component Relationship

The following diagram illustrates how the MarketData orchestration layer interacts with external providers and internal storage utilities to facilitate backfills.

Core Backfill Methods

Backfill operations are typically invoked via save_ methods that follow a standardized logic:

1. Path Resolution: Identify the storage path via PathFinder.
2. Redundancy Check: Remove or check existing files.
3. Data Retrieval: Fetch data from the API with built-in retry logic.
4. Standardization: Convert provider-specific schemas to Hyperdrive's internal format (e.g., mapping exDate to C.EX).
5. Persistence: Update the CSV file via FileWriter.

Sources: hyperdrive/DataSource.py:82-90, hyperdrive/DataSource.py:141-149

Data Standardization and Providers

Hyperdrive enforces a strict schema for all historical data to ensure compatibility across the History module. Standardization includes mapping timestamps, converting values to floats or integers, and handling null values with defaults.

Standardization Mapping

Data Type	Standardized Columns	Provider Mapping (Example)	Default Value
OHLC	TIME, OPEN, HIGH, LOW, CLOSE, VOL	date -> TIME, volume -> VOL	0
Dividends	EX, PAY, DEC, DIV	exDate -> EX, amount -> DIV	0
Splits	EX, PAY, DEC, RATIO	ratio -> RATIO	1
Sentiment	TIME, POS, NEG	bullish -> POS, bearish -> NEG	0

Sources: hyperdrive/DataSource.py:108-115, hyperdrive/DataSource.py:214-222, test/test_DataSource.py:128-135

Provider Implementation Details

• IEXCloud: Implements backfills for OHLC (chart), dividends, and splits. It includes a specific _get_prev_ohlc method to fetch the most recent data point for incremental updates.
• Polygon.io: Supports extensive intraday backfills. It features an obey_free_limit mechanism to prevent rate limiting during large backfill tasks (default 12-second delay for free tier).
• Redundancy: Scripts like update_dividends.py run IEX and Polygon updates in parallel using multiprocessing.Process to maximize data capture.

Sources: hyperdrive/DataSource.py:348-355, hyperdrive/DataSource.py:465-475, scripts/update_dividends.py:58-64

Intraday and Large-Scale Backfills

Intraday backfills are more complex as they involve daily CSV files stored in symbol-specific directories. The TimeTraveller component is used to generate a range of dates for which data must be retrieved.

Intraday Update Sequence

The sequence below shows how the system iterates through dates to build a complete intraday history.

Backfill Configurations

Historical updates can be configured via timeframe parameters, which support both relative (e.g., 5y, 30d, 1m) and absolute identifiers (max).

# Example: Large-scale backfill for 15+ years (approx 6300 days)
poly.save_intraday(
    symbol=symbol,
    timeframe='6300d',
    retries=1
)

Sources: scripts/upload_data.py:31-35, hyperdrive/DataSource.py:510-520

Integration with Research and Backtesting

Once historical data is backfilled and stored, it is consumed by the History module for quantitative analysis. The Historian class uses these CSV files to create vectorbt portfolios.

• Optimal Signals: The system can analyze historical price action to find "optimal" signals based on local extrema.
• Random Strategy Generation: Historically backfilled data allows the generation of thousands of random strategies to establish performance benchmarks (e.g., comparing a strategy against "Buy and Hold").
• Classification: Backfilled data is standardized and passed through PCA (Principal Component Analysis) and scalers before being used to train classifiers like SVMs or Neural Networks.

Sources: hyperdrive/History.py:59-72, hyperdrive/History.py:108-115, scripts/update_api.py:35-45

Conclusion

Historical Data Backfills in Hyperdrive provide the persistence layer necessary for reliable trading research. By abstracting provider-specific complexities behind a standardized MarketData interface and implementing automated synchronization scripts, the system ensures that high-quality historical data is always available for strategy development and validation.