Text & Sentiment Ingestion
Relevant source files
The following files were used as context for generating this wiki page:
The Text & Sentiment Ingestion system is responsible for collecting, normalizing, and scoring qualitative data from central bank communications and investor sentiment surveys. This system serves as a critical evidence layer for the regime detection engine, providing non-deterministic indicators that complement quantitative market data. The scope includes ingestion of FOMC minutes, ECB monetary policy decisions, and AAII investor sentiment data.
This module implements a deterministic lexicon-based approach for text scoring, ensuring that qualitative inputs can be replayed in a stateless, reproducible manner. By converting raw HTML and PDF sources into structured numerical scores (e.g., hawkish vs. dovish indicators), the system allows the regime engine to incorporate monetary policy shifts and behavioral sentiment into its classification logic.
Sources: src/regime_detection/_config_layer2.py:73-86, scripts/validate_central_bank_text_lexicon.py:7-25
Central Bank Text Ingestion Architecture
The ingestion of central bank text follows a pipeline from raw web scraping to structured Parquet storage. The system targets specific official sources such as the Federal Reserve (FOMC) and the European Central Bank (ECB).
Data Flow for Central Bank Communications
The following diagram illustrates the lifecycle of central bank text data, from remote source discovery to the generation of a daily score series used by the engine.
This flow ensures that raw HTML is persisted for provenance before being cleaned and converted into numerical evidence.
Sources: src/regime_data_fetch/fomc_minutes.py:202-270, src/regime_data_fetch/event_sources/official_ecb.py:46-77
Federal Reserve (FOMC) Minutes
The FOMC ingestion module crawls federalreserve.gov to identify meeting minutes. It handles both current calendars and historical archives dating back to 1993. The parser extracts the meeting end date, release date, and the full body text of the minutes.
| Component | Description |
|---|---|
parse_fomc_minutes_listing | Scans the FOMC calendar for PDF/HTML links and release dates. |
parse_fomc_minutes_article | Extracts the core "Minutes of the Federal Open Market Committee" from HTML. |
_clean_html_text | Removes HTML tags, superscripts, and links while normalizing whitespace. |
fetch_release_timestamp | Standardizes release times to 14:00 America/New_York. |
Sources: src/regime_data_fetch/fomc_minutes.py:77-101, src/regime_data_fetch/fomc_minutes.py:126-150, src/regime_data_fetch/fomc_minutes.py:317-328
European Central Bank (ECB) Decisions
The OfficialECBAdapter scrapes monetary policy decisions and Governing Council calendars. It identifies candidates based on high-confidence keywords like "monetary policy decisions" in titles.
Sources: src/regime_data_fetch/event_sources/official_ecb.py:21-44, src/regime_data_fetch/event_sources/official_ecb.py:108-129
Text Scoring Logic & Lexicon Validation
Qualitative text is converted to a net_score using a deterministic hawkish/dovish lexicon. This avoids the non-determinism of LLM calls, which is a requirement for V1 §2.2 stateless replay.
Lexicon Scoring Mechanism
The scorer counts whole-word hits for two disjoint vocabularies: HAWKISH_TERMS and DOVISH_TERMS. The net_score is derived based on the frequency of these terms within the document.
| Net Score Result | Meaning | Mapping |
|---|---|---|
> 0 | More hawkish hits than dovish | Hawkish (1) |
< 0 | More dovish hits than hawkish | Dovish (0) |
0 or NaN | Equal hits or no lexicon hits | Neutral (2) |
Sources: scripts/validate_central_bank_text_lexicon.py:20-25, tests/test_central_bank_text.py:84-100
Document-Level Aggregation
Because the lexicon is designed for full-document scoring, individual sentences might have low coverage. The engine pools sentences by release or year to improve accuracy. When multiple releases (e.g., FOMC minutes and a Powell speech) occur on the same day, the system applies a collision strategy such as pick_longer or token_weighted_average.
Sources: scripts/validate_central_bank_text_lexicon.py:100-115, src/regime_detection/_config_layer2.py:102-108
Investor Sentiment Ingestion (AAII)
The system ingests individual investor sentiment data from the American Association of Individual Investors (AAII). This data is primarily used to calculate the sentiment_score via an 8-week moving average of the bull-bear spread.
Sentiment Data Structure
The AAII fetcher parses historical .cfb (XLS) files and scrapes current HTML tables from aaii.com.
The bull_bear_spread_8w_ma is the primary feature used for detecting "euphoria" or "panic" regimes.
Sources: src/regime_data_fetch/aaii_sentiment.py:157-163, src/regime_detection/_config_layer2.py:61-70
Ingestion Logic
The update_aaii_parquet function handles the synchronization between existing local data and remote updates.
- Seed/Load: Loads existing Parquet or seeds from
aaii_sentiment_historical.cfb. - Fetch: Scrapes
aaii.com/sentimentsurvey/sent_resultsfor rows newer than the last known date. - Compute: Calculates the spread and the 8-week moving average.
- Persist: Saves the result back to Parquet and records lineage in the
AcquisitionStore.
Sources: src/regime_data_fetch/aaii_sentiment.py:171-210
Configuration Parameters
The behavior of the ingestion and scoring layers is controlled through Pydantic configuration models.
Central Bank Text Configuration
| Parameter | Default | Description |
|---|---|---|
smoothing_window_sessions | 30 | Window (NYSE sessions) for smoothing the net_score. |
max_release_age_days | 365 | Safety cap to drop extremely stale releases. |
same_date_aggregation | pick_longer | Strategy for handling multiple releases on one day. |
Sources: src/regime_detection/_config_layer2.py:92-108
News Sentiment Configuration
| Parameter | Default | Description |
|---|---|---|
smoothing_window_sessions | 21 | Window for the SF Fed Daily News Sentiment index. |
Sources: src/regime_detection/_config_layer2.py:67-70
The ingestion system ensures that high-latency qualitative data is transformed into a low-latency, daily evidence stream that the regime detection engine can consume alongside price and volume data. By utilizing deterministic lexicons and rigorous HTML cleaning, the system maintains the integrity of historical replays while providing a structured view of monetary policy and investor psychology.