DataSource Module

Relevant source files

The following files were used as context for generating this wiki page:

• hyperdrive/DataSource.py
• hyperdrive/Workflow.py
• README.md
• scripts/update_api.py
• test/test_DataSource.py

The DataSource module is the core data acquisition and standardization engine of the hyperdrive library. It provides a unified interface for fetching market data, social sentiment, and economic indicators from various third-party APIs including IEXCloud, Polygon.io, StockTwits, Glassnode, and the Bureau of Labor Statistics (BLS).

The module is designed to handle the complexities of financial data ingestion, including API rate limiting, retry logic, and data standardization into consistent pandas DataFrames. It integrates closely with the FileOps and Storage components to cache fetched data locally or in AWS S3, ensuring efficient data retrieval for quantitative research and backtesting.

Sources: hyperdrive/DataSource.py:1-24, README.md:1-25

Class Architecture

The module utilizes an inheritance-based structure where a base MarketData class provides common utilities, and specialized provider classes implement specific API logic.

This diagram illustrates the inheritance hierarchy and key methods for data providers.
Sources: hyperdrive/DataSource.py:15-560, test/test_DataSource.py:38-45

Core Components and Logic

1. MarketData Base Class

The MarketData class serves as the foundation for all data providers. It initializes essential helpers for file operations (FileWriter, FileReader), path management (PathFinder), and time manipulation (TimeTraveller).

• Retry Logic: The try_again method implements a configurable retry mechanism (defaulting to C.DEFAULT_RETRIES) to handle transient network errors or API timeouts.
• Standardization: The standardize method is a generic utility that maps provider-specific API response keys to internal library constants (e.g., mapping "t" or "date" to C.TIME).

Sources: hyperdrive/DataSource.py:15-84

2. Data Standardization Mapping

Each data type (OHLC, Dividends, Splits, etc.) has a dedicated standardization routine to ensure uniform column names across different providers.

Data Type	Key Mapping (API -> Internal)	Target File Path Example
OHLC	date, open, high, low, close, volume	data/ohlc/iexcloud/AAPL.csv
Dividends	exDate, paymentDate, amount	data/dividends/polygon/TSLA.csv
Sentiment	timestamp, bullish, bearish	data/sentiment/MSFT.csv
S2F Ratio	t, o.daysTillHalving, o.ratio	data/indicators/s2f.csv

Sources: hyperdrive/DataSource.py:91-105, hyperdrive/DataSource.py:176-189, test/test_DataSource.py:115-130

Data Flow and Persistence

The following sequence diagram represents the typical flow when a user requests data that needs to be fetched and cached.

The flow ensures that data is processed and stored locally to avoid redundant API calls.
Sources: hyperdrive/DataSource.py:202-230, README.md:65-90

Provider Specific Implementations

Polygon.io and Rate Limiting

The Polygon class includes specific logic to handle the "Free" tier constraints. The obey_free_limit method ensures that calls do not exceed the allowed threshold by calculating time since the last call and sleeping if necessary.
Sources: hyperdrive/DataSource.py:444-454

Workflow Integration

The Flow class in Workflow.py uses MarketData to check if automated data collection jobs are currently active. It calculates expected durations based on the number of symbols retrieved via md.get_symbols().

# From hyperdrive/Workflow.py
def is_workflow_running(self, workflow_name, buffer_min=30):
    md = MarketData()
    start_time = self.get_workflow_start_time(workflow_name)
    num_stock = len(md.get_symbols())
    # ... logic to calculate duration based on symbols and API delays

Sources: hyperdrive/Workflow.py:26-30

API Summary Table

Class	Primary Methods	Data Sources
IEXCloud	get_ohlc, get_dividends, get_splits	Stock market fundamentals and prices
Polygon	get_ohlc, get_intraday, get_splits	High-frequency and historical aggregates
StockTwits	get_social_volume, get_social_sentiment	Social media sentiment metrics
LaborStats	get_unemployment_rate	US Bureau of Labor Statistics (BLS)
Glassnode	get_s2f_ratio, get_diff_ribbon, get_sopr	Blockchain and Bitcoin indicators

Sources: hyperdrive/DataSource.py:355-560, test/test_DataSource.py:216-300

Summary

The DataSource module acts as the ingestion layer for the hyperdrive library. By abstracting the specificities of various APIs into a unified class structure and providing robust standardization and retry logic, it enables the rest of the library to work with clean, consistent financial datasets. Its integration with automated workflows ensures that the local data cache remains up-to-date for real-time and historical analysis.