Testing Data Systems

Relevant source files

The following files were used as context for generating this wiki page:

• test/test_DataSource.py
• test/test_FileOps.py
• test/test_Storage.py
• test/test_Constants.py
• test/test_Calculus.py
• test/test_History.py
• test/test_TimeMachine.py

Testing Data Systems within the hyperdrive project ensures the reliability of market data ingestion, storage operations, and quantitative calculations. The test suite validates that data retrieved from various external providers (e.g., IEXCloud, Polygon, StockTwits) is correctly standardized, cached, and retrieved, while also verifying the integrity of mathematical utilities used for financial modeling.

The testing infrastructure is built primarily using pytest and covers multiple layers of the system: from low-level S3 storage interactions and file operations to high-level data source providers and historical backtesting logic.

Sources: test/test_DataSource.py:10-20, test/test_Storage.py:10-15, test/test_Calculus.py:1-10

Data Source Testing Strategy

The TestMarketData and provider-specific classes (e.g., TestIEXCloud, TestPolygon) verify the ability to fetch and standardize various types of financial data. This includes OHLC (Open, High, Low, Close), dividends, splits, and social sentiment data. Tests often use a dedicated "dev bucket" for S3 operations to avoid polluting production data.

Data Flow for Market Data Validation

The following diagram illustrates how market data is tested from the initial request through standardization to storage validation.

The validation process ensures that:

1. API requests return valid data frames.
2. Standardization logic correctly maps provider-specific keys (e.g., exDate, bullish) to internal constants (e.g., C.EX, C.POS).
3. Cached files exist and have been modified recently.

Sources: test/test_DataSource.py:53-70, test/test_DataSource.py:136-150, test/test_DataSource.py:186-200

Supported Data Types and Verification Criteria

Data Type	Relevant Test Methods	Verification Fields
Dividends	test_get_dividends, test_save_dividends	C.EX, C.PAY, C.DEC, C.DIV
Splits	test_get_splits, test_save_splits	C.EX, C.DEC, C.RATIO
OHLC	test_get_ohlc, test_save_ohlc	C.TIME, C.OPEN, C.HIGH, C.LOW, C.CLOSE, C.VOL
Sentiment	test_get_social_sentiment, test_save_social_sentiment	C.TIME, C.POS, C.NEG, C.VOL, C.DELTA
Unemployment	test_get_unemployment_rate	C.TIME, C.UN_RATE

Sources: test/test_DataSource.py:53-150, test/test_DataSource.py:205-230

File and Storage Operations

Testing for FileOps and Storage focuses on the interaction between the local environment and AWS S3. These tests ensure that JSON and CSV files can be saved, updated, renamed, and deleted reliably across different environments.

Storage Interaction Flow

The interaction between FileReader, FileWriter, and the Store class is validated through sequence testing of file lifecycle events.

Key validations include:

• Update CSV: Testing writer.update_csv to ensure that data is appended or updated without losing existing records.
• Path Resolution: test_Constants.py verifies that PathFinder generates correct paths for different providers (e.g., data/ohlc/iexcloud/AAPL.csv vs data/ohlc/polygon/AAPL.csv).
• S3 Bucket Isolation: Tests check if they are running in a CI environment to decide whether to use a development bucket (S3_DEV_BUCKET).

Sources: test/test_FileOps.py:73-85, test/test_Storage.py:28-50, test/test_Constants.py:14-40

Quantitative and Historical Logic Testing

The system includes tests for the Calculator and Historian classes to ensure mathematical accuracy in trading signal generation and portfolio performance metrics.

Mathematical Utilities

The TestCalculator class validates core financial calculus functions:

• Delta: Percent change between consecutive elements.
• Smooth: Savitzky-Golay filtering for noise reduction.
• Derive: Calculating the derivative of a series.
• Fib: Generating Fibonacci sequences for technical analysis.

Sources: test/test_Calculus.py:11-40

Portfolio and Signal Verification

The TestHistorian validates backtesting logic and signal processing. It ensures that the system can generate optimal buy/sell signals based on historical data and calculate resulting performance stats.

Specific functions like undersample are tested to verify that they correctly balance datasets for machine learning classifiers (e.g., ensuring a 0.5 mean for target variables).

Sources: test/test_History.py:35-65

Time and Scheduling Utilities

TestTimeTraveller ensures that the system's time-related logic—critical for backtesting and scheduled data collection—functions as expected.

• Delta Conversion: Validates strings like '1d', '3w', '1m', '1y' are converted to correct timedelta objects.
• Date Range: Verifies dates_in_range produces the expected number of trading dates within a timeframe.
• Sleep Logic: test_sleep_until ensures the system correctly pauses execution until a specific UTC timestamp is reached.

Sources: test/test_TimeMachine.py:11-50

Conclusion

Testing Data Systems in hyperdrive provides a multi-layered verification of the platform's data integrity. By isolating storage, provider APIs, and mathematical logic into distinct test suites, the project ensures that quantitative research and algorithmic trading are performed on accurate, standardized data. The use of environmental checks for CI and dev buckets allows for safe, repeatable testing of cloud-based storage and external API integrations.

Sources: test/test_DataSource.py:1-40, test/test_FileOps.py:1-20, test/test_History.py:1-10