System Architecture Overview
Relevant source files
The following files were used as context for generating this wiki page:
Hyperdrive is an algorithmic trading platform designed for quantitative research, data collection, and strategy execution. Unlike typical backtesting libraries, it emphasizes high-performance data pipelines and integration with external market data providers and exchanges.
The system is structured into four primary functional layers: Data Acquisition, Persistence & Storage, Quantitative Analysis (History/Calculus), and Execution. It utilizes cloud storage (AWS S3) for data persistence and GitHub Actions for automated workflow orchestration.
Sources: README.md:1-25, hyperdrive/DataSource.py:15-25
Component Architecture
The architecture relies on specialized modules that handle specific parts of the trading lifecycle. The DataSource module serves as the primary entry point for fetching and standardizing market data from various providers.
Core Modules and Classes
| Component | Responsibility | Key Classes/Files |
|---|---|---|
| Data Acquisition | Interfaces with external APIs (IEXCloud, Polygon, etc.) to fetch OHLC, dividends, and social sentiment. | MarketData, IEXCloud, Polygon, StockTwits |
| File Operations | Manages local and remote file I/O, supporting JSON and CSV formats. | FileReader, FileWriter |
| Storage Engine | Interfaces with AWS S3 for persistent data caching and retrieval. | Store |
| Orchestration | Manages the timing and state of data update workflows. | Flow |
| Execution | Handles order placement and portfolio tracking on exchanges. | Binance, Exchange |
Sources: hyperdrive/DataSource.py:15-30, test/test_FileOps.py:10-15, README.md:180-200, hyperdrive/Workflow.py:6-7
High-Level System Flow
The following diagram illustrates the data flow from external providers through the system to final order execution.
The system retrieves data, standardizes it into a common format, and persists it to S3. Analysis modules then use this historical data to generate signals for the execution module.
Sources: hyperdrive/DataSource.py:50-100, scripts/execute_order.py:15-30, README.md:40-60
Data Acquisition & Standardization
The MarketData base class and its provider-specific subclasses (IEXCloud, Polygon, StockTwits, LaborStats, Glassnode) implement the logic for fetching and normalizing financial data.
Standardization Logic
Data from different providers is mapped to internal constants (e.g., C.TIME, C.OPEN, C.CLOSE) to ensure consistency across the library. This allows the backtesting and execution components to work independently of the original data source.
# Standardizing OHLC data to internal format
def standardize_ohlc(self, symbol, df, filename=None):
full_mapping = dict(zip(
['date', 'open', 'high', 'low', 'close', 'volume'],
[C.TIME, C.OPEN, C.HIGH, C.LOW, C.CLOSE, C.VOL]
))
# ... logic to rename and format columns ...
Sources: hyperdrive/DataSource.py:130-150, test/test_DataSource.py:157-175
Data Providers Support
| Provider | Supported Data Types |
|---|---|
| IEXCloud | OHLC (Daily/Intraday), Dividends, Splits |
| Polygon | OHLC, Intraday Aggregates, Dividends, Splits |
| StockTwits | Social Sentiment, Social Volume |
| BLS | Unemployment Rates |
| Glassnode | BTC Stock-to-Flow, Difficulty Ribbon, SOPR |
Sources: hyperdrive/DataSource.py:383-700, test/test_DataSource.py:270-350
Storage and File Operations
The system uses a combination of local caching and S3 storage. FileWriter and FileReader (utilizing Store) manage these operations, ensuring that data is updated rather than overwritten when new records are available.
Workflow Orchestration
The Flow class in Workflow.py manages the state of automated tasks. It prevents concurrent execution of data-heavy workflows (like OHLC or Intraday updates) by checking GitHub Action cron schedules and estimated durations.
Workflow Management Parameters
- Buffer Time: 30-minute window added to workflow start/end times.
- Duration Calculation: Based on the number of symbols and a
POLY_FREE_DELAYconstant to account for API rate limiting. - Active Workflows: Managed categories include
ohlc,intraday,dividends, andsplits.
Sources: hyperdrive/Workflow.py:1-40, test/test_Workflow.py:10-20
Strategy Execution
Execution is handled by the Exchange module (specifically the Binance class). The system compares current signals against recent history to determine if a trade is required.
Order Logic Flow
- Signal Retrieval: Load signals generated by the
Historian. - Change Detection: Compare the most recent signal with the previous one.
- Execution: If a signal change occurs (e.g., False to True), a 'BUY' or 'SELL' order is dispatched to the exchange.
- Logging: Order details (ClientOrderID, executed quantity, etc.) are saved back to the storage layer for portfolio tracking.
Sources: scripts/execute_order.py:15-40, README.md:150-175
Conclusion
The architecture of Hyperdrive is built for scalability and data integrity. By decoupling data acquisition (via DataSource) from storage (via Store) and analysis, the system allows for flexible quantitative research that can be seamlessly transitioned into live trading through the Exchange and Workflow modules.