Feature Store Construction
Relevant source files
The following files were used as context for generating this wiki page:
The Feature Store is a centralized Pydantic-based data structure responsible for aggregating, calculating, and housing all input features required by the regime detection engine. It acts as a bridge between raw market data (encapsulated in the MarketContext) and the various classification models (HMM, GMM, etc.) used to identify market regimes. Sources: src/regime_detection/feature_store.py:160-162, TECH_DEBT_AUDIT.md:28-30
Construction of the feature store is a deterministic process that involves a registry of builders. Each builder is responsible for a specific subset of features, ranging from V1 technical indicators (SMA, volatility) to complex V2 macro-economic and network fragility metrics. The store supports "seams" that allow it to operate in different modes depending on data availability and configuration. Sources: src/regime_detection/feature_store.py:607-627, src/regime_detection/feature_store.py:165-240
Architecture and Data Flow
The Feature Store construction follows a "registry of builders" pattern. The build_feature_store function initializes a build state and executes a sequence of specialized builder functions.
Core Build Lifecycle
- State Initialization: A
_FeatureStoreBuildStateobject is created to hold theMarketContextand temporary results. - Builder Execution: A tuple of
_FeatureStoreBuilderobjects is iterated, each executing itsbuildcallable to mutate the state. - Final Assembly: The
FeatureStoreclass is instantiated using the accumulated state, enforcing strict schema validation via Pydantic.
Sources: src/regime_detection/feature_store.py:116-158, src/regime_detection/feature_store.py:630-685
Data Flow Diagram
The diagram below illustrates how raw data from the context is transformed into structured features.
Feature Seams and Optionality
The Feature Store implements "seams"—conditional population points that gate features based on configuration and data availability. This is critical for maintaining V1/V2 compatibility. Sources: src/regime_detection/feature_store.py:175-235
| Feature Group | Requirement | Source File |
|---|---|---|
| Network Fragility | sector_etf_closes presence | feature_store.py:442-468 |
| Monetary Pressure | macro_series (DGS2, DGS10, USD Index) | feature_store.py:503-524 |
| HMM Evidence | hmm config + volume_liquidity_v2 + network_fragility | feature_store.py:541-557 |
| Clustering | clustering config + breadth_state_v2 + network_fragility | feature_store.py:560-580 |
| Inflation/Growth | DBC, TLT, XLY, XLI, XLP, XLU closes | feature_store.py:604-627 |
Feature Calculation Logic
V1 Technical Indicators
Basic indicators like the 50-day Simple Moving Average (SMA) and Trend Direction are derived directly from SPY close prices.
- SMA 50: Calculated using a rolling window of 50 sessions. Sources: src/regime_detection/feature_store.py:439-440
- Trend Direction: Uses
compute_trend_direction_featureson the SPY closing series. Sources: src/regime_detection/feature_store.py:388-389
V2 Macro and Sentiment Features
- Central Bank Text: Converts FOMC minutes and speeches into a daily score series using a smoothing window (default 30 sessions). Sources: src/regime_detection/feature_store.py:512-518, scripts/run_v2_calibration.py:57-60
- Sentiment Alignment: AAII Bull-Bear spreads are forward-filled onto the SPY session calendar to prevent future-data leakage. Sources: src/regime_detection/feature_store.py:339-361
- Volume Liquidity: Primarily exposes
volume_zscore_20d, using a 20-day rolling mean and standard deviation of SPY volume. Sources: tests/test_volume_liquidity_v2_features.py:63-75, src/regime_detection/feature_store.py:490-500
Implementation Constraints and Tech Debt
The Feature Store is identified as a central "choke point" in the architecture. Current technical debt audits highlight several areas of concern:
- Typed Bus: The internal registry currently uses a
dict[str, Any]bus for state, which is identified as a P1 debt item. Callers depend on string keys rather than typed attributes during the build process. Sources: TECH_DEBT_AUDIT.md:104-106, TECH_DEBT_AUDIT.md:162-164 - Builder Decomposition:
build_feature_storeis a large function that requires further decomposition into stable instrumentation hooks. Sources: src/regime_detection/feature_store.py:644-646, scripts/profile_engine.py:1550-1565
Usage in Operations
The Feature Store is heavily utilized in calibration and profiling scripts to verify data quality and model performance.
Calibration Summary
The run_v2_calibration.py script uses the Feature Store to extract state probabilities for HMMs and centroids for GMM clustering. It surfaces these features to generate candidate label maps for manual operator review. Sources: scripts/run_v2_calibration.py:90-115, scripts/run_v2_calibration.py:157-185
Performance Profiling
The profile_engine.py script measures the wall-clock time required for each feature builder. This allows developers to identify bottlenecks in the feature construction pipeline. Sources: scripts/profile_engine.py:1180-1200, scripts/profile_engine.py:1225-1240
Summary
Feature Store Construction is a deterministic, registry-driven process that transforms market context into a structured dataset for regime detection. By utilizing optional seams and a centralized builder pattern, it supports both legacy V1 requirements and complex V2 macro-evidence signals while maintaining strict schema validation through Pydantic. Sources: src/regime_detection/feature_store.py:630-685, TECH_DEBT_AUDIT.md:28-35