High-Level Architecture
Relevant source files
The following files were used as context for generating this wiki page:
The regime-detection system is an integrated platform designed for financial market regime classification. It transitions from a deterministic V1 model to a more complex V2 architecture that incorporates machine learning (HMM/GMM), point-in-time (PIT) data, and natural language processing of central bank communications.
The architecture is organized into three coupled product areas:
- Runtime Classifier: Responsible for building market context, processing features, and generating timeline outputs via the
RegimeEngine. - Data Acquisition: Handles the collection of market, macro, and event data, maintaining a structured record of provenance and artifacts.
- Operational Support: Includes scripts for calibration, profiling, and walk-forward verification to ensure model integrity.
Sources: TECH_DEBT_AUDIT.md:27-33, AGENTS.md:120-140
System Component Overview
The following diagram illustrates the high-level data flow from external data sources through the acquisition layer into the runtime engine.
The system relies on a strictly defined data flow where external inputs are normalized into source-specific models before being used by the core engine.
Sources: TECH_DEBT_AUDIT.md:35-43, AGENTS.md:10-15
Core Components
1. Runtime Classifier (RegimeEngine)
The RegimeEngine is the primary orchestrator for classification tasks. It utilizes MarketContext to encapsulate all necessary inputs (OHLCV, macro series, event calendars) and processes them through a FeatureStore.
- MarketContext: A unified data structure holding price data (SPY index, Sector ETFs), macro indicators, and sentiment data.
- FeatureStore: Aggregates technical and economic features, including V2-specific components like Hidden Markov Models (HMM) and Gaussian Mixture Models (GMM) for clustering.
- Axis Builders: Logic modules that interpret features into specific regime dimensions such as trend direction, volatility state, and breadth.
Sources: scripts/profile_engine.py:330-360, TECH_DEBT_AUDIT.md:36-39, src/regime_detection/feature_store.py:630-640
2. Data Acquisition and Persistence
Data acquisition is managed by regime_data_fetch, which persists raw artifacts and derived outputs. The system uses a SQLite-based AcquisitionStore to track fetch runs and maintain lineage.
Database Schema (Key Entities)
The following Entity Relationship diagram outlines the primary persistence structure for tracking data provenance.
3. Operational and Verification Scripts
Operational integrity is maintained through a suite of scripts located in the scripts/ directory. These tools handle the lifecycle of the models:
- Calibration:
run_v2_calibration.pyfits ML models (HMM/GMM) and generates candidate mappings for manual operator review. - Profiling:
profile_engine.pymeasures performance and wall-clock time across different classification stages. - Shadow Gates:
run_v2_shadow_ab_gate.pycompares V1 and V2 outputs to ensure zero regressions in legacy fields while validating new V2 enrichments.
Sources: scripts/run_v2_calibration.py:7-20, scripts/run_v2_shadow_ab_gate.py:9-20, scripts/profile_engine.py:1-25
Data Flow Architecture
The classification process follows a linear sequence, instrumented for profiling and observability.
Technical Specifications and Constraints
| Component | Responsibility | Key Files |
|---|---|---|
| Configuration | Centralized Pydantic-based configuration for all engine parameters. | src/regime_detection/config.py |
| Calendar | Strict NYSE trading calendar enforcement; UTC timestamps for persistence. | AGENTS.md, scripts/build_walkforward_report.py |
| ML Models | HMM and GMM clustering for V2 regime identification. | scripts/run_v2_calibration.py |
| NLP | Deterministic lexicon for scoring Central Bank text (Hawkish/Dovish). | scripts/validate_central_bank_text_lexicon.py |
Sources: TECH_DEBT_AUDIT.md:120-130, AGENTS.md:50-55, scripts/run_v2_calibration.py:7-15
Artifact Contract
All persistent data is handled as StoredArtifacts with a fully qualified URI contract. Parquet files are canonicalized (sorted rows, stripped metadata, snappy compression) to ensure sha256 consistency across environments.
Sources: scripts/publish_canonical_snapshot.py:40-60, TECH_DEBT_AUDIT.md:70-75
Conclusion
The high-level architecture of the regime-detection project is a modular system that separates data acquisition from runtime classification. By utilizing a central RegimeEngine and a structured FeatureStore, it supports both deterministic rules and machine-learning-based transitions. The architecture emphasizes data provenance and operational verification, requiring manual operator intervention for critical ML-to-economic label mappings.
Sources: scripts/run_v2_calibration.py:15-25, TECH_DEBT_AUDIT.md:27-45