Corporate Actions Data

Relevant source files

The following files were used as context for generating this wiki page:

• hyperdrive/DataSource.py
• scripts/update_dividends.py
• scripts/update_splits.py
• scripts/update_hist_dividends.py
• scripts/update_hist_splits.py
• test/test_DataSource.py
• hyperdrive/Workflow.py

The Corporate Actions Data system in the hyperdrive library is responsible for fetching, standardizing, and persisting financial events—specifically dividends and stock splits—from various market data providers. This module ensures that quantitative research and backtesting have access to accurate historical adjustments and cash distribution information.

The system utilizes a provider-agnostic architecture where a base MarketData class defines the interface and standardization logic, while specialized subclasses like IEXCloud and Polygon implement the specific API interactions. Data is stored locally or in S3 buckets as CSV files, organized by symbol and provider.

Sources: hyperdrive/DataSource.py:12-25, README.md:15-20

Architecture and Data Flow

The architecture follows a tiered approach:

1. Provider Layer: Handles API requests to external services (IEX Cloud, Polygon.io).
2. Standardization Layer: Converts provider-specific JSON responses into a uniform DataFrame format using internal constants.
3. Persistence Layer: Uses FileWriter and FileReader to manage CSV storage and updates.

Data Acquisition Flow

The following diagram illustrates how the system fetches and saves corporate action data:

The system prioritizes data integrity by using an "update" mechanism that prevents overwriting existing historical data with potentially truncated API responses.
Sources: hyperdrive/DataSource.py:44-55, scripts/update_dividends.py:18-35

Dividend Management

Dividend data processing involves capturing the ex-dividend date, payment date, declaration date, and the actual cash amount.

Key Classes and Methods

• MarketData.get_dividends(symbol, timeframe): Retrieves cached dividend data from the local filesystem or S3.
• IEXCloud.get_dividends(**kwargs): Specifically fetches "Cash" dividends in "USD" from the IEX Cloud /stock/{symbol}/dividends/{timeframe} endpoint.
• Polygon.get_dividends(**kwargs): Fetches dividends using the Polygon REST client's reference_stock_dividends method.

Dividend Data Model

Standardized dividend dataframes use the following field mapping:

Original Field (IEX/Poly)	Standardized Constant	Description
exDate	C.EX	The date a stock trades without its dividend.
paymentDate	C.PAY	The date the dividend is actually paid.
declaredDate	C.DEC	The date the company announces the dividend.
amount	C.DIV	The cash amount per share.

Sources: hyperdrive/DataSource.py:65-80, hyperdrive/DataSource.py:345-375, test/test_DataSource.py:65-75

Stock Splits Management

Stock splits are managed similarly to dividends, but focus on the split ratio which affects historical price adjustments.

Split Standardization Logic

The standardize_splits method ensures that split ratios are treated as floats with a default value of 1 (no split) if data is missing.

def standardize_splits(self, symbol, df):
    full_mapping = dict(
        zip(
            ['exDate', 'paymentDate', 'declaredDate', 'ratio'],
            [C.EX, C.PAY, C.DEC, C.RATIO]
        )
    )
    filename = self.finder.get_splits_path(symbol, self.provider)
    return self.standardize(df, full_mapping, filename, [C.EX, C.RATIO], 1)

Sources: hyperdrive/DataSource.py:102-115

Data Fields for Splits

Constant	Usage
C.EX	Ex-Date of the split.
C.RATIO	The split ratio (new shares = old shares / ratio).
C.DEC	Declaration date of the split.

Sources: hyperdrive/DataSource.py:535-540, test/test_DataSource.py:100-110

Redundancy and Update Scripts

The project employs "Double Redundancy" for data updates, typically running processes for both IEX and Polygon simultaneously to ensure maximum data coverage.

Update Mechanism Sequence

The update_dividends.py and update_splits.py scripts utilize multiprocessing.Process to execute these updates in parallel.
Sources: scripts/update_dividends.py:58-70, scripts/update_splits.py:1-40

Workflow Integration

Corporate action updates are integrated into the automated execution pipeline. The Flow class in Workflow.py monitors these tasks.

• Duration Calculation: The system calculates expected runtime based on the number of symbols and a POLY_FREE_DELAY constant to avoid API rate limits.
• Workflow Monitoring: The is_workflow_running method checks if 'dividends' or 'splits' jobs are currently active by evaluating cron schedules and estimated durations.

Sources: hyperdrive/Workflow.py:24-42

Summary

The Corporate Actions Data module provides a robust framework for managing dividends and splits. By abstracting provider differences and enforcing a standardized schema through MarketData.standardize, the system creates a reliable historical record necessary for calculating total returns and adjusted prices in trading strategies. The use of redundant update scripts and automated workflow monitoring ensures the data remains current and accurate across multiple sources.