AWS S3 Data Storage
Relevant source files
The following files were used as context for generating this wiki page:
The AWS S3 Data Storage system in hyperdrive provides a centralized, cloud-based repository for market data, quantitative research models, and technical indicators. It serves as the primary persistence layer, enabling the library to collect data from various providers (IEXCloud, Polygon, Glassnode) and cache it for quantitative research and backtesting. Sources: README.md:1-20, hyperdrive/Storage.py:10-15
The system is designed to handle both production and development environments through environment-variable-driven bucket selection. It integrates closely with the library's file operations to ensure that data is synchronized between local storage and the cloud, facilitating consistent data access across different execution environments. Sources: hyperdrive/Storage.py:16-20, hyperdrive/FileOps.py:15-20
Core Architecture
The storage architecture is built around the Store class, which wraps boto3 to provide high-level abstractions for S3 operations. It manages bucket connectivity, file transfers, and object metadata.
The Store Class
The Store class is the primary interface for AWS S3. It dynamically selects the target bucket based on the DEV flag and manages the lifecycle of S3 objects. Sources: hyperdrive/Storage.py:10-25
The diagram above illustrates the relationship between the low-level Store class and the higher-level file operation classes that utilize it for data persistence. Sources: hyperdrive/Storage.py:10-40, hyperdrive/FileOps.py:15-25, hyperdrive/FileOps.py:90-95
Environment Configuration
The system relies on specific environment variables for authentication and configuration. These are typically loaded from a config.env file. Sources: hyperdrive/Storage.py:11-12, README.md:85-95
| Variable | Description |
|---|---|
AWS_ACCESS_KEY_ID | AWS credentials for S3 access. |
AWS_SECRET_ACCESS_KEY | AWS secret key for authentication. |
S3_BUCKET | Production bucket for storing market data. |
S3_DEV_BUCKET | Development bucket for testing and sandboxing. |
DEV | Boolean flag to toggle between production and dev buckets. |
Sources: hyperdrive/Storage.py:12-20, README.md:88-93
Data Synchronization Flow
Data management in hyperdrive follows a "Local-First, Cloud-Synced" pattern. The FileReader checks if a local file exists and is up-to-date; if not, it pulls the latest version from S3. The FileWriter ensures that any local changes are immediately reflected in the S3 bucket. Sources: hyperdrive/FileOps.py:20-40, hyperdrive/FileOps.py:96-105
Download Sequence
When a data request is made, the system follows this sequence to ensure data availability:
The logic for determining if a file is stale is based on a 24-hour threshold or if the file is completely missing from the local filesystem. Sources: hyperdrive/FileOps.py:19-33, hyperdrive/Storage.py:59-70
Upload and Maintenance
Data is uploaded either individually using upload_file or in bulk using upload_dir. The latter utilizes a multiprocessing.Pool for concurrent uploads, improving performance for large datasets like intraday market prices. Sources: hyperdrive/Storage.py:30-40, scripts/upload_data.py:20-35
Maintenance scripts like update_repo.py use the Store to clean up the bucket, preserving only the data/ and models/ prefixes while removing other artifacts. Sources: scripts/update_repo.py:7-13
Key Storage Operations
Object Management
- Key Path Normalization: All local paths are converted to S3 keys by replacing backslashes with forward slashes to maintain cross-platform compatibility. Sources: hyperdrive/Storage.py:28-29, hyperdrive/Storage.py:44
- Conditional Checks:
key_existscan verify an object's presence in S3 without downloading it by callingload()on the object, which triggers a HEAD request. Sources: hyperdrive/Storage.py:50-55 - Batch Deletion: The
delete_objectsmethod supports deleting multiple keys in a single request, which is used by maintenance scripts to prune data directories. Sources: hyperdrive/Storage.py:42-45, scripts/delete_data_dir.py:15-20
Metadata and Deltas
The system can track the freshness of data stored in S3 using the last_modified property of S3 objects. Sources: hyperdrive/Storage.py:87-90
last_modified(key): Retrieves the UTC timestamp of the object's last update in S3.modified_delta(key): Calculates the duration (timedelta) between the current time and the object's last modification.
Sources: hyperdrive/Storage.py:87-98
Implementation in Data Sources
The MarketData and its subclasses (IEXCloud, Polygon) use these storage utilities to cache API responses. For instance, when save_ohlc is called, the data is standardized into a CSV and passed to FileWriter.update_csv, which internally calls Store.upload_file to sync the new data to AWS. Sources: hyperdrive/DataSource.py:100-110, hyperdrive/FileOps.py:112-115
Data Directory Structure
While not explicitly defined in a single schema, the source files indicate a hierarchical structure:
data/ohlc/{provider}/{symbol}.csv: Historical daily bars.data/intraday/{provider}/{symbol}/{date}.csv: Minute-level data.data/dividends/{provider}/{symbol}.csv: Dividend history.models/: Serialized research models.
Sources: README.md:104-106, hyperdrive/DataSource.py:72, scripts/upload_data.py:12, scripts/update_repo.py:9
Summary
The AWS S3 Data Storage module is the backbone of hyperdrive's data persistence strategy. By abstracting raw Boto3 calls into a high-level Store class and integrating it directly into FileReader and FileWriter, the library ensures that large-scale financial datasets are easily accessible, versioned in the cloud, and synchronized across local and remote environments for quantitative analysis. Sources: hyperdrive/Storage.py:10-20, hyperdrive/FileOps.py:10-30, README.md:15-25