Introduction & Overview
Relevant source files
The following files were used as context for generating this wiki page:
The Regime Detection project is a sophisticated engine designed to identify and classify financial market regimes using a unified V1 and V2 architecture. It leverages a combination of market data, macro-economic series, and specialized event sources (such as FOMC minutes and ECB decisions) to generate a high-dimensional representation of market states. The system is structured into three primary coupled products: a runtime classifier, a data acquisition layer, and a suite of operational scripts and gates for validation and calibration.
Sources: TECH_DEBT_AUDIT.md:27-33, pyproject.toml:7-8
System Architecture
The project architecture is divided into distinct functional areas that handle the lifecycle of regime detection, from raw data ingestion to final state classification.
Core Functional Areas
- Runtime Classifier (
src/regime_detection): Responsible for buildingMarketContext, extracting features, and generating timeline outputs using Pydantic wire models. - Data Acquisition (
src/regime_data_fetch): Collects external market, macro, and event artifacts, ensuring provenance and materializing inputs for the engine. - Operational Gates (
scripts/): A suite of tools for engine profiling, HMM/GMM calibration, walk-forward gates, and shadow A/B testing.
Sources: TECH_DEBT_AUDIT.md:29-33
Data Flow Diagram
The following diagram illustrates the high-level flow from external data sources through the feature store to the final regime classification.
Key Components & Modules
Feature Store & Axis Builders
The Feature Store acts as the central repository for calculated metrics. It aggregates data into several functional "seams," including monetary pressure, credit funding, inflation/growth, and network fragility. These features are then consumed by Axis Builders to determine specific regime labels.
Sources: scripts/audit_layer2_30d.py:53-113, scripts/profile_engine.py:512-526
V2 Calibration System
The system utilizes unsupervised learning for advanced regime labeling. The V2 calibration process fits Hidden Markov Models (HMM) and Gaussian Mixture Models (GMM) on historical data to identify hidden states.
- HMM (§6.1): Identifies states based on temporal persistence and transition probabilities.
- GMM (§6.2): Performs clustering based on centroid distances and scatter within the feature space.
- Manual Review: The engine requires an operator to map identified clusters/states to economic labels (e.g.,
calm_bull,stress_crash) before finalizing the model.
Sources: scripts/run_v2_calibration.py:4-18, scripts/run_v2_calibration.py:104-148
Verification & Performance Gates
To maintain the integrity of classifications, the project employs several "gates":
- V2 Shadow A/B Gate: Classifies sessions using both V1 and V2 modes to ensure zero regressions in V1 fields while validating new V2 enrichments.
- Canonical Snapshots: Ensures that data used for backtesting and production is stable by recomputing SHA256 hashes and row counts for Parquet artifacts.
- Engine Profiler: Measures the wall-clock execution time of different classification stages to identify performance bottlenecks.
Sources: scripts/run_v2_shadow_ab_gate.py:4-15, scripts/publish_canonical_snapshot.py:4-11, scripts/profile_engine.py:4-10
Operating Discipline & Rules
The project enforces a strict "Operating Discipline" for developers and agents to ensure deterministic results and prevent future-data leakage.
Non-negotiable Rules
| Rule ID | Name | Description |
|---|---|---|
| Rule 5 | Layer Validation | Validate every layer boundary; one source-specific model per source before normalization. |
| Rule 6 | CI Verification | Full-suite verification belongs in GitHub Actions; local runs are for targeted debugging only. |
| Rule 10 | UTC Enforcement | All stored timestamps MUST be UTC with timezone. Naive datetimes are banned. |
| Rule J | Failure Modes | Bug fixes require a failing test first. Patching symptoms without addressing root causes is discouraged. |
Sources: AGENTS.md:7-38
Implementation Details
Technology Stack
The project is built on a modern Python stack with strict version pinning for data stability.
- Language: Python >= 3.11
- Data Processing: NumPy, Pandas (v2.3.0), PyArrow (v23.0.0)
- Modeling: hmmlearn, bayesian-changepoint-detection
- Validation: Pydantic (v2.13.4), Pyright (strict mode)
- Infrastructure: S3 (via boto3), Playwright (for browser-based fetching)
Sources: pyproject.toml:7-25, pyproject.toml:68-71
Execution Workflow (Sequence Diagram)
This diagram represents the typical sequence for a profiling or classification run.
Conclusion
The Regime Detection engine provides a robust framework for financial market state analysis by combining deterministic rule-based logic (V1) with advanced probabilistic modeling (V2). Its architecture emphasizes data provenance, performance observability, and strict adherence to NYSE trading calendars to ensure that detections are both accurate and reproducible across different environments.
Sources: AGENTS.md:120-130, scripts/run_v2_shadow_ab_gate.py:4-15