Live Market Fetchers
Relevant source files
The following files were used as context for generating this wiki page:
Live Market Fetchers constitute the data acquisition layer of the regime detection system. Their primary purpose is to ingest raw financial and macroeconomic data from diverse external sources—including the Federal Reserve, Bureau of Labor Statistics (BLS), European Central Bank (ECB), and commercial data providers like Alpaca and Investing.com—and normalize this data into structured formats (typically Parquet) suitable for the regime detection engine.
The system is designed to handle different acquisition cadences, ranging from unattended autonomous refreshes of market prices and FRED macro series to operator-assisted browser-based fetches for complex event calendars like corporate earnings. This layer also ensures data provenance by recording raw artifacts and fetch metadata into a specialized AcquisitionStore.
Sources: scripts/fetch_regime_engine_v1_data.py:27-45, src/regime_data_fetch/investing_live.py:17-25
Data Acquisition Architecture
The fetcher system is organized around a central registry of "Fetch Modes," categorized by their execution requirements. The architecture separates the high-level workflow orchestration from the source-specific implementation details.
Workflow Orchestration
The primary entry point for data acquisition is scripts/fetch_regime_engine_v1_data.py, which coordinates the execution of various fetchers based on the requested scope (v1, v2, or all).
The diagram shows the flow from CLI command to the planning phase, splitting into unattended and operator-assisted execution paths before persisting to the Acquisition Store.
Sources: scripts/fetch_regime_engine_v1_data.py:74-120, scripts/fetch_regime_engine_v1_data.py:304-324
Fetch Mode Categories
| Category | Modes | Description |
|---|---|---|
| Unattended | market, macro, events, sentiment, pmi, pit, fomc, powell | Fully automated; typically relies on APIs (FRED, Alpaca) or simple HTTP GET requests. |
| Operator-Assisted | eps, investing-live, daily-ohlcv-local-sqlite | Requires manual intervention, such as providing a local workbook or running a browser-based session. |
Sources: scripts/fetch_regime_engine_v1_data.py:74-96, scripts/fetch_regime_engine_v1_data.py:100-106
Specialized Fetcher Modules
FOMC Minutes Fetcher
This module scrapes the Federal Reserve website to retrieve meeting minutes. It handles both current calendars and historical archives dating back to 1993.
- Key Functions:
parse_fomc_minutes_listing(): Extracts meeting end dates and release dates from the main Fed calendar.parse_fomc_minutes_article(): Cleans HTML content to extract the meeting date text and minutes body.
- Data Structure:
FOMCMinutesArticlecontains meeting end dates, release timestamps, and the full body text.
Sources: src/regime_data_fetch/fomc_minutes.py:65-73, src/regime_data_fetch/fomc_minutes.py:126-150
BLS Schedule Fetcher
Retrieves release schedules for critical US economic indicators like Consumer Price Index (CPI) and Non-Farm Payrolls (NFP).
This sequence diagram illustrates the fetcher's ability to use a proxy service (TinyFish) if direct access to the BLS website is restricted.
Sources: src/regime_data_fetch/bls_schedule.py:80-94, src/regime_data_fetch/bls_schedule.py:105-120
Investing.com Live Fetcher
Used primarily for real-time earnings calendar data. Due to anti-bot measures, it utilizes a browser-based approach or consumes pre-loaded pages containing the __NEXT_DATA__ JSON payload.
- Components:
accessTokenmanagement for Investing.com API requests.- Browser capture via Playwright (when
earnings_browser_captureis enabled). - Data normalization from internal JSON schemas to the project's standard
EventCandidateformat.
Sources: src/regime_data_fetch/investing_live.py:897-902, scripts/fetch_regime_engine_v1_data.py:270-300
Official ECB Fetcher
Implements a specialized adapter for the European Central Bank's monetary policy decisions.
| Component | Target URL | Purpose |
|---|---|---|
| Archive Index | .../press/govcdec/mopo/html/index.en.html | Locates historical decision snippets by year. |
| Current Calendar | .../press/calendars/mgcgc/html/index.en.html | Identifies future scheduled monetary policy meetings. |
Sources: src/regime_data_fetch/event_sources/official_ecb.py:16-25, src/regime_data_fetch/event_sources/official_ecb.py:82-95
Macro and Market Data Ingestion
The run_macro_fetch and run_market_fetch functions serve as the primary conduits for bulk data.
Macro Ingestion (FRED)
The macro fetcher targets a specific set of FRED series IDs required for regime detection, including DTWEXBGS (Broad USD Index) and IORB (Interest on Reserve Balances). It supports ALFRED-style realtime observations to capture data vintages for historical "point-in-time" accuracy.
Sources: tests/test_fetch_workflow.py:180-205, scripts/_fetch_regime_engine_v1_args.py:65-75
Market Ingestion (Alpaca)
The market fetcher retrieves daily OHLCV bars. It manages a symbol universe that includes shared anchors (SPY, RSP), sector ETFs (XLF, XLK, etc.), and cross-asset symbols (GLD, USO, TLT).
- Incremental Updates: The system checks for existing Parquet files and merges incremental updates, ensuring that new data is appended while preserving historical records.
- VIX Proxying: If true VIX data is unavailable, the fetcher can be configured to use a proxy symbol like
VIXY.
Sources: tests/test_fetch_workflow.py:112-160, scripts/_fetch_regime_engine_v1_args.py:100-115
Summary
Live Market Fetchers provide the robust infrastructure necessary to maintain the system's data integrity. By combining automated API clients with fallback mechanisms like proxying and browser-based capture, the system ensures a continuous flow of macroeconomic and market data. Every fetch operation is tracked, and data is validated against specific contracts—such as symbol presence and date continuity—before being used in regime classification.