Workflow Orchestration

Introduction

Workflow Orchestration in the hyperdrive library serves to synchronize and manage the execution of various data collection and update tasks. It primarily handles the timing and status tracking of GitHub Actions workflows, ensuring that automated processes such as data ingestion for OHLC, dividends, splits, and social sentiment do not overlap destructively or exceed API rate limits.

The system utilizes a combination of cron schedule parsing from GitHub Workflow YAML files and real-time execution monitoring. This orchestration is critical for maintaining the integrity of the market data stored in S3 and local environments, providing a mechanism to check if specific data pipelines are currently active.
Sources: hyperdrive/Workflow.py:7-13, README.md:214-222

Workflow Monitoring and Status

The Flow class is the central component for managing workflow execution states. It determines if a workflow is "running" by calculating the expected duration based on the number of symbols and a pre-defined API delay, comparing this against the scheduled start time parsed from the project's .github/workflows/ directory.

Key Logic for Running Status

A workflow is considered running if the current time falls within a calculated window: start_time - buffer < now < start_time + duration + buffer. The duration is dynamically calculated based on the workload (e.g., number of stocks and cryptocurrencies) and the provider's free-tier delay.
Sources: hyperdrive/Workflow.py:27-41

The diagram above illustrates the logic used to determine the operational status of a specific workflow.
Sources: hyperdrive/Workflow.py:27-46

The Flow Class

Sources: hyperdrive/Workflow.py:7-48

Data Update Workflows

Orchestration is frequently applied to data maintenance scripts found in the scripts/ directory. These scripts often utilize Python's multiprocessing to handle concurrent updates across different data providers (e.g., IEX Cloud and Polygon) for redundancy.

Redundancy Strategy

The system implements a "Double Redundancy" pattern where updates are performed via one provider and then verified or supplemented by a second provider.
Sources: scripts/update_splits.py:11-13, scripts/update_hist_splits.py:13-15

This sequence shows the parallel or sequential execution of redundant data updates to ensure high data availability.
Sources: scripts/update_splits.py:16-52, scripts/update_hist_dividends.py:16-47

Implementation of Update Scripts

Scripts like update_splits.py use a multiprocessing.Value counter to track success rates across symbols. If the success rate falls below a threshold (e.g., 5%), the process exits with an error code to signal failure to the orchestration layer (GitHub Actions).
Sources: scripts/update_splits.py:54-55

# scripts/update_splits.py:54-55
if counter.value / (len(symbols) * 2) < 0.05:
    exit(1)

API Rate Limit Integration

Orchestration also involves respecting provider limits. The Polygon class, for instance, includes logic to obey_free_limit, which introduces delays between calls. The Flow.is_workflow_running method uses the POLY_FREE_DELAY constant (typically 12 seconds for free tier) to estimate how long a workflow will take to complete its symbol list.
Sources: hyperdrive/DataSource.py:504-511, hyperdrive/Workflow.py:31-33

Conclusion

Workflow orchestration in hyperdrive is a time-based management system designed for reliable data ingestion. By parsing cron schedules and estimating task duration based on asset counts and API constraints, the system prevents concurrent resource contention and provides a framework for redundant data updates across multiple financial data providers.