Automation Utilities
Relevant source files
The following files were used as context for generating this wiki page:
Automation Utilities in the hyperdrive project encompass the systems and scripts responsible for managing background workflows, maintaining environment dependencies, and automating data synchronization for the platform's API and research modules. These utilities ensure that data collection from various providers (IEXCloud, Polygon, StockTwits) remains synchronized and that the system's operational state is protected during high-load periods.
The utilities are divided between high-level workflow management, which prevents resource contention during data ingestion, and maintenance scripts that automate repository updates and security tasks.
Sources: hyperdrive/Workflow.py:1-40, util/update.py:1-40, README.md:200-240
Workflow Management
The Flow class provides logic to parse GitHub Actions configuration files and determine the operational status of scheduled data pipelines. This is used to prevent concurrent execution of intensive tasks that might exceed API rate limits or cause data corruption.
Operational Logic
The system calculates the expected duration of workflows based on the number of symbols being tracked and known API delays (e.g., POLY_FREE_DELAY).
The duration calculation varies by workflow type. For instance, ohlc and intraday workflows include multipliers for both stock and crypto symbols, while dividends and splits scale linearly with the stock symbol count.
Sources: hyperdrive/Workflow.py:6-38, hyperdrive/DataSource.py:530-550
Key Workflow Attributes
| Workflow Name | Data Source Dependencies | Scaling Factor |
|---|---|---|
ohlc | IEXCloud / Polygon | Stock Count + Crypto Count |
intraday | IEXCloud / Polygon | Stock Count + Crypto Count |
dividends | IEXCloud / Polygon | Stock Count |
splits | IEXCloud / Polygon | Stock Count |
Sources: hyperdrive/Workflow.py:30-40
Dependency and Repository Automation
The project utilizes automated scripts to maintain up-to-date environment configurations and package requirements.
Package Update Utility
The util/update.py script automates the synchronization of requirements.txt with the latest versions available on PyPI, with specific exclusions for sensitive machine learning libraries.
- Exclusion Logic: Packages like
scikit-learnandauto-sklearnare skipped to prevent unpredictable model behavior between versions. - Process: The script fetches JSON metadata from PyPI, identifies the latest numeric release, and performs a local upgrade before updating the requirement file.
Sources: util/update.py:7-35
Security and Versioning
Automation also extends to security through util/encrypt.sh, which uses GPG symmetric encryption with AES256 to protect sensitive configuration files. Repository versioning is automated within setup.py by querying the GitHub API for the latest release tag.
Sources: util/encrypt.sh:10-13, setup.py:11-20
API and Data Synchronization
The scripts/update_api.py utility serves as the bridge between the research core and the production API. It automates the generation of "preview" datasets by comparing algorithmic strategies against a "Buy and Hold" (HODL) benchmark.
Synchronization Flow
The script generates portfolio performance metrics including Total Return, Max Drawdown, Win Rate, and Profit Factor for both USD and BTC pairs.
Sources: scripts/update_api.py:20-100
Metadata Transformation
Metrics are standardized for the API via the transform_stats function, which handles rounding and converts NaN values to None for JSON compatibility.
Sources: scripts/update_api.py:10-16
The Automation Utilities provide the operational backbone of the hyperdrive project, ensuring that data pipelines are managed intelligently, dependencies are kept current, and API-facing data remains synchronized with the latest quantitative research results.