Clustering & Time Series Change Points
Relevant source files
The following files were used as context for generating this wiki page:
The Clustering and Time Series Change Point detection systems serve as the secondary "V2 evidence layer" within the regime detection engine. Their primary purpose is to provide unsupervised statistical signals that complement the deterministic rules used for identifying market environments. Clustering groups similar market sessions based on multivariate return and volatility profiles, while change point detection identifies structural shifts in time series behavior.
These components do not automatically label market regimes. Instead, they emit raw statistical outputs—such as cluster IDs, probabilities, and change point scores—which are then reviewed by human operators or used as evidence components in the final transition risk assessment.
Sources: src/regime_detection/clustering.py:1-15, tests/test_clustering.py:7-12, scripts/run_v2_calibration.py:7-15
Gaussian Mixture Model (GMM) Clustering
The clustering system utilizes a Gaussian Mixture Model (GMM) to partition market sessions into distinct statistical groups. It operates on a multivariate input set representing price action, volatility, and breadth.
Architecture and Data Flow
The clustering process follows a Point-in-Time (PIT) semantic, meaning that for any given session, the model is trained only on data available up to that session's lookback window. The flow involves feature extraction, normalization via a StandardScaler, and fitting the GMM.
The system requires specific features to be present for the clustering seam to "light up," including 21-day and 63-day returns, realized volatility, ADX (Average Directional Index), pairwise correlations, and market breadth (percentage of stocks above 50 DMA).
Sources: src/regime_detection/clustering.py:50-80, tests/test_clustering.py:20-75
Key Components
compute_clustering_features: The main entry point that validates history, fits the GMM, and generates assignments.ClusteringFeatures: A dataclass holding the resultingcluster_id,cluster_probabilities, anddistance_to_centroid(Mahalanobis distance).ClusteringConfig: Defines parameters liken_clusters(default 8),training_window_days(default 1260), andcovariance_type.
Sources: src/regime_detection/clustering.py:18-45, src/regime_detection/config.py:1013-1030
Bayesian Change Point Detection
The change point system identifies structural breaks in market data using Bayesian online change point detection algorithms. It specifically focuses on detecting shifts in the mean and variance of price returns.
Detection Logic
The system utilizes an underlying adapter that implements online_changepoint_detection. It models the observation likelihood using a Student-T distribution and assumes a constant hazard function. The primary output is a "run length" probability distribution, where a high probability of a low run length indicates a recent structural change.
Configuration and Outputs
| Parameter | Type | Description |
|---|---|---|
training_window_days | int | The lookback period used for detection (typically 252 days). |
hazard_lambda | float | Parameter for the constant hazard function. |
min_history_days | int | Minimum data required before a score is emitted. |
Sources: src/regime_detection/config.py:1032-1050
Calibration and Operator Review
Because these systems are unsupervised, the integer outputs (e.g., Cluster 0, Cluster 1) have no inherent economic meaning. The project implements a calibration workflow where candidate mappings are generated for manual review.
V2 Calibration Workflow
The run_v2_calibration.py script fits the models on the full historical dataset and generates candidate artifacts.
- Fit Models: Run GMM and Change Point detection on the training window.
- Summarize States: Calculate per-state means, persistence diagonals (probability of staying in the same state), and representative dates.
- Generate YAML: Emit
cluster_label_map.candidate.yamlandhmm_state_label_map.candidate.yaml. - Manual Review: An operator inspects the "extreme distance" dates and mean profiles to assign economic labels like
calm_bullorstress_crash.
Sources: scripts/run_v2_calibration.py:165-240, scripts/run_v2_calibration.py:410-450
Performance Profiling
Given the computationally intensive nature of PIT (Point-in-Time) GMM fitting—which requires refitting the model for every session in a lookback window—the project includes a dedicated profiling engine.
The profile_engine.py script instruments the feature_store assembly, specifically measuring the wall-clock time spent on feature_store.gmm_clustering and feature_store.change_point. It uses a StageTimer to capture totals across window classifications. If a run exceeds a defined run_timeout_seconds (default 300s), it raises a RunTimeout to prevent clustering/HMM calculations from hanging the pipeline.
Sources: scripts/profile_engine.py:90-120, scripts/profile_engine.py:1300-1350
Summary
Clustering and Change Point detection provide the statistical "backbone" for the V2 regime engine. By leveraging GMMs for multivariate grouping and Bayesian methods for structural break detection, the system captures market nuances that deterministic rules might miss. The architectural insistence on PIT semantics ensures these signals are free from look-ahead bias, while the calibration workflow maintains human interpretability of the unsupervised results.