Introduction & Getting Started
Relevant source files
The following files were used as context for generating this wiki page:
hyperdrive is an algorithmic trading library designed to facilitate quantitative research and data collection. Unlike standard backtesting libraries, it places a heavy emphasis on robust data acquisition from multiple providers, market data storage in cloud environments (AWS S3), and the creation of reproducible research pipelines. It serves as the underlying technology for the quant research firm forcepu.sh.
The library supports a wide range of data types including OHLC (Open, High, Low, Close), intraday ticks, dividends, stock splits, social sentiment from StockTwits, and macroeconomic indicators like unemployment rates from the Bureau of Labor Statistics. It integrates with external providers such as IEXCloud, Polygon.io, and Glassnode to provide a comprehensive data suite for algorithmic strategy development.
Sources: README.md:1-25, hyperdrive/DataSource.py:10-30
Core Features and Architecture
The architecture of hyperdrive is modular, separating data sourcing, storage operations, and historical analysis. The MarketData class acts as a base orchestrator, utilizing helper components for file operations (FileReader, FileWriter), path management (PathFinder), and time manipulation (TimeTraveller).
Primary Components
| Component | Description |
|---|---|
| MarketData | Base class for data handling, standardization, and caching logic. |
| DataSource Providers | Specific implementations for IEXCloud, Polygon, StockTwits, LaborStats, and Glassnode. |
| FileOps | Handles reading and writing of CSV and JSON data, including S3 synchronization. |
| Historian | Manages backtesting, portfolio creation, and signal optimization. |
| Storage (Store) | Manages interaction with AWS S3 buckets for data persistence. |
Sources: hyperdrive/DataSource.py:15-25, test/test_FileOps.py:10-15, README.md:40-60
System Data Flow
The following diagram illustrates how data flows from external API providers through the hyperdrive standardization layer into local or cloud storage.
The diagram shows the ingestion process where raw provider data is normalized by the standardization layer before being committed to storage via FileOps.
Sources: hyperdrive/DataSource.py:40-150, test/test_Storage.py:15-30
Installation and Environment Setup
Prerequisites
- Python 3.8+ (setup.py requires >= 3.7)
- AWS Account and S3 Bucket for data persistence
- API Keys for chosen providers (IEXCloud, Polygon, etc.)
Sources: README.md:28-32, setup.py:42
Installation
The library can be installed via pip:
python -m pip install hyperdrive -U
Sources: README.md:34-38
Environment Configuration
Most configuration is handled via environment variables, typically stored in a config.env file. These variables are critical for both API authentication and cloud storage access.
| Variable | Description |
|---|---|
IEXCLOUD / POLYGON | API tokens for market data providers. |
AWS_ACCESS_KEY_ID | AWS credentials for S3 access. |
AWS_SECRET_ACCESS_KEY | AWS credentials for S3 access. |
S3_BUCKET | The primary S3 bucket for data storage. |
BINANCE | API token for order execution on Binance.US. |
Sources: README.md:45-55, hyperdrive/DataSource.py:17-20
Getting Started: Data Collection
The library follows a pattern of "Get, Standardize, and Save." The MarketData and provider-specific classes (like IEXCloud) provide methods to fetch and cache data.
Example: Storing Market Data
from hyperdrive import DataSource
from DataSource import IEXCloud, MarketData
# Initialize
md = MarketData()
iex = IEXCloud()
# Save OHLC data to local/S3 cache
iex.save_ohlc(symbol='TSLA', timeframe='7d')
# Retrieve cached data
df = md.get_ohlc(symbol='TSLA', timeframe='7d')
Sources: README.md:85-110, hyperdrive/DataSource.py:130-150
Internal Standardization Process
When data is fetched, hyperdrive maps provider-specific keys (e.g., date, open, uHigh) to a set of internal constants (e.g., C.TIME, C.OPEN, C.HIGH). This ensures that backtesting logic remains agnostic of the data source.
This sequence demonstrates how the IEXCloud class utilizes the MarketData base class's standardization routines before persisting data.
Sources: hyperdrive/DataSource.py:118-140, test/test_DataSource.py:165-180
Backtesting and Analysis
hyperdrive integrates with vectorbt for strategy evaluation. The Historian class is used to generate signals and create portfolios for performance comparison.
Key Analysis Metrics
The library focuses on several quantitative metrics during backtesting:
- Total Return [%]
- Max Drawdown [%]
- Sharpe/Sortino Ratios
- Win Rate [%]
- Profit Factor
Sources: README.md:140-160, scripts/update_api.py:35-55
Workflow Example
from hyperdrive import History, DataSource, Constants as C
from History import Historian
from DataSource import MarketData
hist = Historian()
md = MarketData()
# Load data and generate signals
df = md.get_ohlc(symbol='TSLA', timeframe='1y')
signals = hist.get_optimal_signals(df[C.CLOSE])
# Create and evaluate portfolio
my_strat = hist.create_portfolio(df[C.CLOSE], signals)
print(my_strat.stats())
Sources: README.md:130-150, test/test_History.py:40-55
Summary
hyperdrive provides a structured framework for moving from raw financial data to actionable quantitative strategies. By enforcing strict data standardization and leveraging cloud storage for caching, it allows researchers to build pipelines that are resilient to API changes and data inconsistencies. Developers should begin by configuring their environment variables and using the DataSource module to build a local cache of the instruments they intend to analyze.