Rolling Statistics & Temporal Normalization
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Rolling Statistics and Temporal Normalization form the mathematical backbone of the regime detection engine, transforming raw financial and macro-economic data into normalized, signal-ready features. The primary objective is to account for varying historical contexts (temporal normalization) and volatility environments by applying rolling windows for z-score calculations and change detections.
This system ensures that signals like "Tightening Pressure" or "Rate Shocks" are relative to their recent historical distributions rather than absolute values. By using standardized rolling windows, the project maintains deterministic replay capabilities across different market environments. These statistics are heavily utilized in the Monetary Pressure module to evaluate yield changes and USD index movements.
Architecture and Components
The architecture relies on a "One home per concept" philosophy, centralizing mathematical logic in shared utility modules while allowing individual feature slices to define their specific window parameters.
Core Mathematical Logic
The engine utilizes a standardized rolling change z-score calculation. This process involves two distinct windows:
- Change Window: The period over which the raw change in a variable (e.g., a Treasury yield) is measured.
- Normalizer Window: A much longer lookback period (often 5 years or 1260 sessions) used to calculate the mean and standard deviation of those changes to produce a z-score.
Sources: [src/regime_detection/monetary_pressure.py:89-105], [src/regime_detection/monetary_pressure.py:157-160]
Data Flow for Temporal Normalization
Data undergoes a specific transformation pipeline before it is consumed by the rule engines. This pipeline ensures that sparse macro observations are aligned to the NYSE trading calendar without introducing future-data leakage (lookahead bias).
This diagram shows the transformation of raw observations into normalized z-scores using dual-window rolling statistics.
Sources: [src/regime_detection/monetary_pressure.py:116-125], [src/regime_detection/monetary_pressure.py:141-155]
Key Functions and Implementation
Yield and Change Normalization
The primary implementation for normalization is the rolling_change_zscore helper. In the context of monetary pressure, this is wrapped to handle specific macro series like the 2-year (DGS2) and 10-year (DGS10) constant-maturity Treasury yields.
| Function | Purpose | Input Parameters |
|---|---|---|
_rolling_change_zscore | Core logic for z-score calculation. | series, change_window, normalizer_window, output_name |
_yield_change_zscore | Specialized wrapper for yield data. | yield_series, lookback, normalizer_window, output_name |
_carry_forward_observations | Aligns sparse data to NYSE sessions. | series (e.g., FRED daily rates) |
Sources: [src/regime_detection/monetary_pressure.py:108-113], [src/regime_detection/monetary_pressure.py:116-125]
Temporal Alignment
Temporal alignment is critical for "Point-in-Time" (PIT) accuracy. The profile_engine and historical_walkforward scripts ensure that data is sliced up to a specific as_of_date before statistics are calculated.
def _carry_forward_observations(series: pd.Series) -> pd.Series:
"""Align sparse macro observations to NYSE sessions without lookahead.
FRED daily rates can miss NYSE sessions because of source publication
calendars. Rolling 5y normalizers should consume the latest observation
available as of the session; staleness remains a separate concern.
"""
return series.astype(float).ffill()
Sources: [src/regime_detection/monetary_pressure.py:116-125], [scripts/profile_engine.py:385-395]
Integration in Feature Stores
Normalized statistics are aggregated into a FeatureStore. The build_feature_store function acts as the central assembly point, applying configurations (such as MonetaryPressureV2FeaturesConfig) to raw data to produce the MonetaryPressureV2Features dataclass.
Monetary Pressure Feature Set
The following features are generated using the temporal normalization pipeline:
| Feature Name | Description | Base Series |
|---|---|---|
yield_change_zscore_2y_63d | 63d change z-score for 2y Treasury. | DGS2 |
yield_change_zscore_10y_63d | 63d change z-score for 10y Treasury. | DGS10 |
broad_usd_index_zscore_63d | 63d change z-score for USD index. | DTWEXBGS |
yield_change_zscore_21d_2y | 21d change z-score (Rate Shock). | DGS2 |
yield_change_zscore_21d_10y | 21d change z-score (Rate Shock). | DGS10 |
Sources: [src/regime_detection/monetary_pressure.py:65-87], [scripts/audit_layer2_30d.py:53-62]
Performance and Profiling
The engine includes specific instrumentation to measure the performance of these statistical calculations. The StageTimer class and its measure context manager are used within scripts/profile_engine.py to track the "wall clock" duration of building axis series and feature stores.
This sequence shows how the engine instruments the feature creation process to ensure rolling calculations remain performant.
Sources: [scripts/profile_engine.py:114-135], [scripts/profile_engine.py:465-485]
Conclusion
Rolling Statistics and Temporal Normalization are fundamental to the project's ability to interpret market conditions dynamically. By standardizing the way z-scores and rolling changes are calculated—and by strictly enforcing calendar alignment without lookahead—the system provides a robust, reproducible framework for detecting economic regimes. This approach ensures that all downstream rules are evaluated against statistically significant, normalized signals rather than volatile raw data.