State Estimation: HMM & Volatility
Relevant source files
The following files were used as context for generating this wiki page:
State estimation within the regime-detection project focuses on identifying latent market states through probabilistic models and volatility analysis. The system uses a Hidden Markov Model (HMM) to provide diagnostic evidence of regime transitions and a multi-tiered volatility analysis (V1 and V2) to categorize market environments based on price action and implied risk.
These components act as "evidence seams" within the broader feature store, feeding into the transition score calculation and structural causal analysis. The HMM provides a permutation-invariant "top state probability," while the volatility modules compute metrics like Average True Range (ATR) ratios, realized volatility ratios, and gap frequencies to detect stress or "vol crush" events.
Sources: src/regime_detection/hmm_state.py, src/regime_detection/feature_store.py:113-144, src/regime_detection/configs/core3-v2.0.0.yaml:495-538
Hidden Markov Model (HMM) Evidence Layer
The HMM evidence layer (V2 §6.1) implements a Gaussian HMM to classify market sessions into a configured number of states (typically 4). It is designed to be a "fit-on-the-fly" diagnostic tool that uses a multi-seed sweep to ensure deterministic and monotonic convergence.
HMM Architecture and Data Flow
The HMM consumes five primary market features:
- 1-day Returns: Daily percentage change in SPY.
- 21-day Realized Volatility: Annualized rolling standard deviation.
- 63-day Trailing Drawdown: Percentage drop from the 63-session peak.
- 20-day Volume Z-Score: Normalized share volume.
- 63-day Average Pairwise Correlation: Network fragility metric.
The HMM uses hmmlearn.GaussianHMM with a "full" covariance type. It parallelizes a sweep across 10 random seeds (e.g., 42, 101, 202, etc.) and selects the fit with the highest log-likelihood that satisfies monotonicity (no likelihood drops during EM).
Sources: src/regime_detection/hmm_state.py:1-40, src/regime_detection/feature_store.py:534-550, src/regime_detection/configs/core3-v2.0.0.yaml:495-515
Key Data Structures: HMM
| Component | Type | Description |
|---|---|---|
HMMConfig | Pydantic Model | Configuration for n_states (4), training_window_days (1260), and random_seeds. |
HMMFeatures | Pydantic Model | Container for state_probabilities (DataFrame) and top_state_prob (Series). |
HmmOutput | Pydantic Model | The final API output containing top_state, top_state_prob, and mapped_label. |
Sources: src/regime_detection/hmm_state.py:44-88, src/regime_detection/models.py:339-366
Volatility State Estimation (V1 & V2)
The project maintains two levels of volatility analysis. V1 focuses on realized volatility vs. a VIX proxy, while V2 introduces technical indicators like ATR ratios and event-driven logic like "vol crush" detection.
Volatility V1 Logic
The primary function compute_features in the V1 module calculates the 1-day log return and a normalized "volatility score." This score is based on the relative position of the SPY realized volatility against the VIX proxy.
Sources: src/regime_detection/volatility_state.py:20-55
Volatility V2 Logic
Volatility V2 (V2 §1C) adds sophistication by analyzing intraday ranges and price gaps. It introduces the vol_crush regime, which is triggered when realized volatility drops significantly following a macro event (e.g., FOMC or CPI releases).
Volatility V2 Rules and Thresholds
| Rule | Threshold | Lookback | Description |
|---|---|---|---|
atr_ratio | 1.15 | 14d vs 50d | Ratio of short-term to long-term Average True Range. |
rv_ratio | 1.25 | 10d vs 63d | Ratio of 10-day to 63-day realized volatility. |
gap_frequency | 0.005 | 20d | Frequency of price gaps exceeding 0.5%. |
vol_crush | 0.75 | 10d vs 21d | Triggered if rv_10d < rv_21d * 0.75 within a 3-day event window. |
Sources: src/regime_detection/volatility_state_v2.py:155-212, src/regime_detection/configs/core3-v2.0.0.yaml:138-164
Feature Store Integration
The FeatureStore acts as the central assembly point for HMM and Volatility features. It enforces "seam gating," where features are only populated if the required data and configurations are present.
Gating Logic
- HMM Seam: Populated only if
volume_liquidity_v2andnetwork_fragilityseams are also lit. - Volatility V2 Seam: Requires
VolatilityV2Config. It consumesimplied_vol_30d(FRED VIXCLS) if available, enabling IV-based rules. - Drawdown Shared Input: A 63-day trailing drawdown is computed once and shared between HMM and Clustering features.
Sources: src/regime_detection/feature_store.py:192-205, src/regime_detection/feature_store.py:518-531
Timeline and Persistence
The timeline module handles the alignment of these probabilistic outputs across time. A critical diagnostic for the HMM is state_persistence_days, which tracks how long the model has remained in its current dominant state.
Timeline Persistence Calculation
The system uses the full (un-windowed) HMM series to calculate persistence, ensuring that counts starting before the current output window are reported accurately to the user.
# Conceptual persistence logic from timeline.py
def _hmm_state_persistence_days(full_series, target_ts):
current_state = full_series.iloc[loc]
days = 1
for i in range(loc - 1, -1, -1):
if full_series.iloc[i] != current_state:
break
days += 1
return days
Sources: src/regime_detection/timeline.py:186-210
Conclusion
State estimation via HMM and Volatility analysis provides a multi-faceted view of market risk. While the Volatility modules offer deterministic, rule-based labels for immediate tactical response, the HMM provides a probabilistic, training-based perspective that identifies structural shifts in market behavior over 5-year windows. Together, they ensure the project can distinguish between transient price noise and genuine regime transitions.
Sources: src/regime_detection/configs/core3-v2.0.0.yaml:495-538, src/regime_detection/models.py:339-366