Fetching Market Data (OHLC & Intraday)
Relevant source files
The following files were used as context for generating this wiki page:
Fetching market data is a core capability of the hyperdrive library, specializing in data collection for quantitative research. The system supports retrieving Open-High-Low-Close (OHLC) and intraday time-series data from multiple providers, primarily IEXCloud and Polygon.io. Data is typically standardized into consistent formats and cached as CSV files for local or cloud storage.
Sources: README.md:1-25, hyperdrive/DataSource.py:11-20
Architecture and Data Flow
The data fetching architecture is built around the MarketData base class, which provides shared logic for standardization and retry mechanisms. Concrete provider classes like IEXCloud and Polygon inherit from this base to implement API-specific requests.
Component Relationship
The following diagram illustrates how the various components interact to fetch and store market data.
Data Standardization
A critical feature of the system is the standardize_ohlc method. It ensures that regardless of the source API (which may use different keys like t for time or v for volume), the resulting DataFrame uses a unified set of column names defined in the Constants module.
| Original Key (Polygon) | Original Key (IEX) | Standardized Key | Description |
|---|---|---|---|
t | date | C.TIME | Timestamp of the record |
o | open | C.OPEN | Opening price |
h | high | C.HIGH | Highest price |
l | low | C.LOW | Lowest price |
c | close | C.CLOSE | Closing price |
v | volume | C.VOL | Trading volume |
vw | average | C.AVG | Volume weighted average price |
Sources: hyperdrive/DataSource.py:129-158, hyperdrive/DataSource.py:440-450
Fetching Strategies
OHLC (Historical)
OHLC data is fetched using specific timeframes (e.g., '1d', '1m', 'max'). The system uses a try_again wrapper to handle transient network or API failures with configurable retries and delays.
# Example of saving OHLC data
# hyperdrive/DataSource.py:165-173
def save_ohlc(self, **kwargs):
symbol = kwargs['symbol']
filename = self.finder.get_ohlc_path(symbol, self.provider)
if os.path.exists(filename):
os.remove(filename)
df = self.reader.update_df(
filename, self.get_ohlc(**kwargs), C.TIME, C.DATE_FMT)
self.writer.update_csv(filename, df)
Sources: hyperdrive/DataSource.py:22-38, hyperdrive/DataSource.py:165-173
Intraday Data
Intraday data retrieval often involves iterating through specific dates. For IEXCloud, this involves hitting the chart/date/{date} endpoint. For Polygon, it uses the aggregates endpoint with a 'minute' span.
Rate Limiting and Workflow Management
The Polygon class includes an obey_free_limit method to handle API constraints for free-tier users, ensuring a mandatory delay between calls. Additionally, the Workflow module provides utilities to check if automated data collection jobs (cron) are currently running to avoid resource contention.
Key Methods for Flow Control:
obey_free_limit(): Sleeps for a duration defined byC.POLY_FREE_DELAYminus the time since the last call.is_workflow_running(workflow_name): Calculates the expected duration of a data job based on the number of symbols and API delays.
Sources: hyperdrive/DataSource.py:408-418, hyperdrive/Workflow.py:26-44
Automated Updates
Production environments utilize scripts to keep the data repository current. The update_ohlc.py script demonstrates a parallel approach using multiprocessing to update both stock and crypto symbols across multiple providers simultaneously.
# Parallel update example
# scripts/update_ohlc.py:53-58
p1 = Process(target=update_iex_ohlc)
p2 = Process(target=update_poly_ohlc)
p1.start()
p2.start()
p1.join()
p2.join()
Sources: scripts/update_ohlc.py:20-61
Summary
The "Fetching Market Data" module provides a robust, standardized interface for ingesting financial time-series data. By abstracting provider-specific logic into a common hierarchy and implementing automated retries, rate-limiting, and parallelized update scripts, hyperdrive maintains a reliable local cache of OHLC and intraday data suitable for complex quantitative backtesting.