File Operations

Relevant source files

The following files were used as context for generating this wiki page:

• hyperdrive/FileOps.py
• hyperdrive/DataSource.py
• test/test_FileOps.py
• test/test_Storage.py
• README.md
• scripts/update_api.py

File Operations within the hyperdrive project encompass the management, synchronization, and persistence of market data and configuration files. The system is designed to handle local file system operations in tandem with remote cloud storage (AWS S3), ensuring that data is cached, updated, and available for quantitative research and trading execution.

The core logic is divided between reading operations (handled by FileReader) and writing operations (handled by FileWriter), both of which interface with a Store object for cloud synchronization. This module is critical for maintaining the integrity of historical OHLC data, dividend records, and social sentiment metrics used by other components like the MarketData provider.

Architecture and Components

The file operation system follows a layered approach where high-level classes use utility classes to manage data flow between the local disk and remote buckets.

Class Hierarchy and Relationships

The following diagram illustrates the relationship between the file operation classes and the underlying storage mechanism.

The interaction between components ensures that every write to the local disk is mirrored to the cloud, and every read checks for local freshness before fetching from the cloud.
Sources: hyperdrive/FileOps.py:12-113, hyperdrive/DataSource.py:16-25, test/test_FileOps.py:8-9

FileReader Logic

The FileReader class is responsible for retrieving data from the local file system while managing synchronization with remote storage. It implements a caching strategy where files are only downloaded if they are missing or considered stale (older than 24 hours).

Freshness Check

The should_be_updated method determines if a file needs to be synchronized from the Store.

• If the file does not exist locally, it returns True.
• If the file exists, it compares the current time with the file's last modification time. If the difference exceeds 86,400 seconds (one day), it returns True.
  Sources: hyperdrive/FileOps.py:18-28

CSV and JSON Loading

FileReader supports loading data into Python-native formats:

• JSON: Returns a dictionary via json.load.
• CSV: Returns a pandas.DataFrame. It includes error handling for EmptyDataError and FileNotFoundError.
  Sources: hyperdrive/FileOps.py:30-51

Dataframe Merging and Filtering

The update_df method provides a way to merge new data into an existing CSV file without duplicates, giving preference to new entries. The data_in_timeframe method allows for filtering records based on a specific timeframe (e.g., '1m', '7d', 'max') relative to the current time.
Sources: hyperdrive/FileOps.py:60-91

FileWriter Logic

The FileWriter class handles the persistence of data to the local disk and ensures that these changes are uploaded to the remote Store.

Storage Operations

Method	Description	Source
save_json	Serializes a dict to JSON, makes the local directory path, and uploads to S3.	hyperdrive/FileOps.py:101-107
save_csv	Saves a DataFrame to CSV and uploads to S3. Returns False if DataFrame is empty.	hyperdrive/FileOps.py:109-118
update_csv	Only performs a save if the new data contains at least as many rows as the existing file.	hyperdrive/FileOps.py:120-123
remove_files	Deletes files from both the local disk and the remote storage.	hyperdrive/FileOps.py:125-127

Data Flow for Writing

This sequence diagram shows the typical flow when the system saves market data.

Data Standardization and Persistence

File operations are heavily utilized by the MarketData class to standardize raw API responses from providers like IEXCloud or Polygon into a consistent local format.

# hyperdrive/DataSource.py:133-140
def save_ohlc(self, **kwargs):
    symbol = kwargs['symbol']
    filename = self.finder.get_ohlc_path(symbol, self.provider)
    if os.path.exists(filename):
        os.remove(filename)
    df = self.reader.update_df(
        filename, self.get_ohlc(**kwargs), C.TIME, C.DATE_FMT)
    self.writer.update_csv(filename, df)

In this workflow, the FileReader (self.reader) is used to load and merge data, while the FileWriter (self.writer) handles the final persistence and cloud synchronization.
Sources: hyperdrive/DataSource.py:133-140, README.md:92-104

Summary

File operations in hyperdrive serve as a robust abstraction layer over local and cloud storage. By centralizing read/write logic in FileReader and FileWriter, the project ensures consistent data handling across various market data types (OHLC, dividends, sentiment). The built-in 24-hour freshness check and duplicate-aware DataFrame updates facilitate efficient quantitative research by maintaining a high-quality local cache of market information.
Sources: hyperdrive/FileOps.py:1-140, hyperdrive/DataSource.py:1-350