Environment Variables & Secrets
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
The hyperdrive library relies heavily on environment variables and secrets to manage configuration, authenticate with third-party market data providers, and secure sensitive files. Most secrets are required to be passed as environment variables, which the library loads via python-dotenv. These variables control everything from cloud storage targets (S3) to API authentication for platforms like IEXCloud, Polygon, and Binance.
Sources: README.md:46-50, hyperdrive/DataSource.py:16
Configuration Management
Configuration is primarily handled by loading a config.env file. This is performed across various modules and scripts to ensure global access to necessary credentials and flags.
Environment Flags
The system uses boolean flags to determine the execution context, which affects behavior such as data storage targets and API usage.
| Variable | Description | Impact |
|---|---|---|
DEV | Development flag | Used to toggle development-specific logic. |
CI | Continuous Integration flag | Changes behavior in scripts (e.g., removing local files after upload). |
TEST | Testing flag | Used to reduce retries or use sandbox environments. |
Sources: hyperdrive/Constants.py:7-17, scripts/update_splits.py:27
File Encryption and Security
For sensitive data files that need to be stored securely, the project provides utilities to encrypt and decrypt files using AES256 or the cryptography library.
The encryption flow uses a key derived from a password (RH_PASSWORD) and a salt (SALT) via the Scrypt Key Derivation Function (KDF).
The diagram shows the logic used in util/encrypt.py to secure local files before storage or transmission.
Sources: util/encrypt.py:12-23, util/decrypt.py:12-23
External Service Authentication
Hyperdrive integrates with multiple financial and infrastructure providers. Each provider requires specific environment variables for authentication.
Market Data Providers
| Provider | Environment Variable | Purpose |
|---|---|---|
| IEXCloud | IEXCLOUD | Production API token. |
| IEXCloud | IEXCLOUD_SANDBOX | Token for testing/sandbox environment. |
| Polygon | POLYGON | API key for market data. |
| StockTwits | STOCKTWITS | OAuth token for social sentiment volume. |
| Glassnode | GLASSNODE | API key for blockchain/on-chain metrics. |
| BLS | BLS | Registration key for Labor Statistics. |
Sources: hyperdrive/DataSource.py:347-353, hyperdrive/DataSource.py:444, hyperdrive/DataSource.py:534, hyperdrive/DataSource.py:598
Infrastructure & Deployment
The library utilizes environment variables to interact with AWS, GitHub, and PyPI during research and release cycles.
- AWS/S3: Uses
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_DEFAULT_REGION, andS3_BUCKET(orS3_DEV_BUCKETfor development) to manage data persistence. - GitHub: Uses
GITHUBtoken to fetch release information during installation. - PyPI: Uses
PYPIandPYPI_TESTtokens to generate.pypircfor package distribution.
Sources: README.md:52-56, setup.py:12-14, util/pypi.py:7-8
Data Flow & Authentication
When a DataSource object (like IEXCloud or Binance) is initialized, it automatically retrieves credentials from the environment.
This flow ensures that no secrets are hardcoded within the library's logic.
Sources: hyperdrive/DataSource.py:353, README.md:126-128
Summary
Environment variables and secrets are the backbone of the hyperdrive configuration system. By utilizing .env files and standard environment variables, the library maintains security while providing the flexibility to switch between testing, development, and production environments across various market data providers and cloud infrastructure services.
Sources: README.md:46-47, hyperdrive/Constants.py:7-10