Macro Data Fetchers

Relevant source files

The following files were used as context for generating this wiki page:

• src/regime_data_fetch/fetch_workflow.py
• src/regime_data_fetch/fomc_minutes.py
• src/regime_data_fetch/event_sources/official_ecb.py
• src/regime_data_fetch/event_sources/official_boe.py
• scripts/fetch_regime_engine_v1_data.py
• src/regime_data_fetch/fred.py
• src/regime_data_fetch/cleveland_fed_nowcast.py
• src/regime_data_fetch/ism.py

Macro Data Fetchers represent the data acquisition layer responsible for gathering macroeconomic indicators, central bank communications, and nowcast data. These fetchers interface with external APIs (such as FRED) and scrape official central bank websites (FED, ECB, BoE) to provide the necessary evidence for the regime detection engine.

The system is designed to be incremental, merging new observations with existing Parquet-based local storage while maintaining data provenance through an AcquisitionStore (SQLite-backed). It categorizes fetches into "unattended" automated workflows and "operator-assisted" manual imports.

Sources: scripts/fetch_regime_engine_v1_data.py:65-72, src/regime_data_fetch/fetch_workflow.py:388-395

Core Macro Fetching Architecture

The macro fetching logic is orchestrated by run_macro_fetch, which iterates through a predefined set of Federal Reserve Economic Data (FRED) series. It handles the raw JSON acquisition, parsing into DataFrames, and merging with local partitioned Parquet files.

FRED Series Registry

The system tracks specific series IDs mapped to logical names used within the regime detection rules.

Logical Name	FRED Series ID	Description
2y_yield	DGS2	2-Year Treasury Constant Maturity Rate
10y_yield	DGS10	10-Year Treasury Constant Maturity Rate
broad_usd_index	DTWEXBGS	Nominal Broad Dollar Index
cpi_all_items	CPIAUCSL	Consumer Price Index for All Urban Consumers
hy_oas	BAMLH0A0HYM2	ICE BofA US High Yield Index OAS
gdp_nowcast	GDPNOW	Atlanta Fed GDPNow Nowcast

Sources: src/regime_data_fetch/fetch_workflow.py:27-66

Macro Data Flow

The following diagram illustrates the lifecycle of a macro data point from FRED to the local artifact store.

The flow ensures that raw responses are preserved for audit purposes before any normalization or cleaning occurs.
Sources: src/regime_data_fetch/fetch_workflow.py:440-520

Central Bank Communication Fetchers

The system includes specialized adapters for fetching and parsing minutes and decisions from major central banks. These fetchers are critical for the monetary_pressure axis.

FOMC Minutes (Federal Reserve)

The run_fomc_minutes_fetch workflow scrapes the Federal Reserve's calendar and historical pages. It extracts both HTML and PDF links, then parses the HTML article body into plain text for NLP scoring.

• Listing Scraper: Targets fomccalendars.htm for recent meetings and fomc_historical_year.htm for archives back to 1993.
• Article Parser: Extracts the meeting date text, title, and body from specific HTML containers (div#article or div#leftText).
• Release Timing: Timestamps are normalized to 14:00 ET on the release date.

Sources: src/regime_data_fetch/fomc_minutes.py:15-50, src/regime_data_fetch/fomc_minutes.py:230-260

European Central Bank (ECB)

The OfficialECBAdapter fetches monetary policy decisions. It utilizes an archive index that provides "data-snippets" for specific years.

Bank of England (BoE)

The OfficialBOEAdapter combines sitemap crawling for historical data and a POST-based News API for recent releases. It specifically targets "Monetary Policy Summary and Minutes".

Sources: src/regime_data_fetch/event_sources/official_boe.py:150-180

Specialized Macro Indicators

CPI Vintages and Nowcasts

The fetcher supports "Point-in-Time" (PIT) analysis by fetching CPI vintages using FRED's realtime start/end parameters. This allows the engine to replay what was known at a specific date rather than using revised data.
Sources: src/regime_data_fetch/fetch_workflow.py:470-500

ISM PMI

The ism.py module contains utilities to extract the Purchasing Managers' Index (PMI) from ISM reports. The extract_ism_pmi_value function uses regex to find numeric values associated with labels like "Manufacturing PMI".
Sources: [src/regime_data_fetch/ism.py:13-17] (Refrenced in tests/test_fetch_workflow.py:155-165)

Operational Workflow

Fetches are executed via the fetch_regime_engine_v1_data.py script, which organizes fetchers into a registry.

# scripts/fetch_regime_engine_v1_data.py:74-96
FETCH_MODE_REGISTRY = {
    "market": FetchModeSpec("market", UNATTENDED, invoke=_invoke_market_fetch),
    "macro": FetchModeSpec("macro", UNATTENDED, invoke=_invoke_macro_fetch),
    "fomc": FetchModeSpec("fomc", UNATTENDED, invoke=_invoke_fomc_fetch),
    "cleveland-fed-nowcast": FetchModeSpec(
        "cleveland-fed-nowcast", 
        UNATTENDED, 
        invoke=_invoke_cleveland_fed_nowcast_fetch
    ),
}

Execution Groups and Concurrency

To optimize performance, the fetch execution planner groups unattended tasks. Conservative concurrency allows for thread-pool execution of independent macro sources (e.g., FRED and Sentiment) while avoiding rate-limiting on sensitive market data providers.
Sources: scripts/fetch_regime_engine_v1_data.py:500-530

Conclusion

Macro Data Fetchers provide the empirical foundation for regime detection. By combining robust API integration with FRED and resilient scraping of central bank records, the system ensures that macro evidence is both historically accurate (via PIT vintages) and operationally current. The architecture emphasizes provenance, recording raw artifacts in a persistent store before normalization.