Shadow Running & Backtesting Gates

Relevant source files

The following files were used as context for generating this wiki page:

• scripts/run_shadow_regime.py
• scripts/run_v2_walkforward_gate.py
• scripts/run_v2_shadow_ab_gate.py
• scripts/build_walkforward_report.py
• scripts/run_shadow_replay_check.py
• tests/test_v2_gate_scripts.py
• tests/test_v2_comparison.py

Shadow running and backtesting gates are critical operational safeguards in the regime detection engine. These systems ensure that model updates—specifically the transition from V1 to V2—are validated against historical performance (Walk-forward Gates) and live consistency requirements (Shadow A/B Gates). Shadow running allows the engine to execute on real-time data while persisting inputs and outputs for replay verification without impacting production trading systems.

Sources: scripts/run_v2_walkforward_gate.py:12-25, scripts/run_v2_shadow_ab_gate.py:11-25, scripts/run_shadow_regime.py:166-180

V2 Performance & Walk-forward Gates

The Walk-forward Gate (V2 §9.1) is a performance validation mechanism that compares the unified V1+V2 engine's output against a baseline. It iterates engine.classify() over a minimum one-year out-of-sample window, measuring wire-level metrics and strategy performance indicators.

Performance Evaluation Logic

The gate evaluates four primary strategy metrics to determine if a V2 update is eligible for promotion. A "Pass" is granted if at least one winning metric shows improvement over the V1 baseline.

Gate Metric	Target Direction	Description
LOWER_DRAWDOWN	Lower (Less Negative)	Reduces the maximum peak-to-trough decline.
HIGHER_SHARPE	Higher	Improves risk-adjusted returns.
EARLIER_CRISIS_DETECTION	Lower (Days)	Reduces detection lag for "Crisis" regimes.
LOWER_FALSE_SWITCH_RATE	Lower	Minimizes whipsaws/reversions to previous labels within hysteresis windows.

Sources: scripts/run_v2_walkforward_gate.py:15-20, tests/test_v2_comparison.py:27-110

Walk-forward Execution Flow

This diagram illustrates the process of running a walk-forward gate, which involves dual-mode classification (V1 and V2) and metric tallying.

The system utilizes a RegimeEngine to classify each NYSE session. The V2 mode includes additional data such as sector ETF closes, cross-asset closes, and first-release CPI vintages.

Sources: scripts/run_v2_walkforward_gate.py:152-205, scripts/run_v2_walkforward_gate.py:269-325

Shadow A/B Gating

The Shadow A/B Gate (§9.3) focuses on consistency and regression testing over the most recent 60 NYSE sessions. It ensures that the addition of V2 enrichments does not negatively impact existing V1 wire fields.

Disagreement Classification

The gate distinguishes between "Regressions" (illegal changes in V1 fields) and "Wins" (expected activations of V2 fields).

1. V1 Fields (Zero-Tolerance): Must stay identical. Any change in trend_direction, trend_character, volatility_state, breadth_state, or transition_risk_state is flagged as a P1 regression.
2. V2 Fields (Expected Deltas): Fields like transition_risk_score, change_point, credit_funding_state, and network_fragility are expected to populate/change when V2 inputs are wired.

Sources: scripts/run_v2_shadow_ab_gate.py:11-25, scripts/run_v2_shadow_ab_gate.py:53-85

Shadow Execution Architecture

Shadow Run Persistence & Replay

The shadow running system persists the exact inputs and outputs of a classification run to allow for byte-identical replays. This is critical for auditing and debugging deterministic engine behavior.

Shadow Data Schema

The system uses a SQLite database (regime_shadow.db) to track runs. The layout includes:

• Input Archives: Parquet files for market data and YAML for event calendars, stored in date-partitioned directories.
• Outputs: JSON files containing the full Pydantic model dump of the RegimeOutput.
• Replay Checks: A verification table that records whether a re-run of the engine with archived inputs matches the original stored output.

Sources: scripts/run_shadow_regime.py:166-215, scripts/run_shadow_replay_check.py:75-125

Replay Verification Logic

The run_shadow_replay_check.py script performs a deep comparison of stored vs. replayed payloads. It uses a recursive diffing function to identify exactly which keys or values have drifted.

def _diff_values(replayed: Any, stored: Any) -> Any:
    if replayed == stored:
        return None
    if isinstance(replayed, dict) and isinstance(stored, dict):
        keys = sorted(set(replayed) | set(stored))
        nested = {}
        for key in keys:
            child = _diff_values(replayed.get(key), stored.get(key))
            if child is not None:
                nested[key] = child
        return nested or None
    return {"replayed": _jsonable(replayed), "stored": _jsonable(stored)}

Sources: scripts/run_shadow_replay_check.py:45-66

Summary of Gate Invariants

To maintain system integrity, several non-negotiable rules are enforced across all gate scripts:

• Fail-Closed on Errors: Unless the --allow-session-errors flag is explicitly passed, any RuntimeError during a classification session causes the gate script to exit with code 1, failing CI/CD pipelines.
• NYSE Calendar Discipline: All shadow runs and backtests require a valid NYSE trading day for the as_of_date.
• Version Coupling: Gate reports explicitly log the engine_version and config_version to ensure that performance metrics are tied to specific code checkpoints.

Sources: tests/test_v2_gate_scripts.py:43-60, scripts/run_shadow_regime.py:167, scripts/run_v2_walkforward_gate.py:230-245