Testing Strategy & Frozen Fixtures
Relevant source files
The following files were used as context for generating this wiki page:
The testing strategy for the regime-detection repository is centered on deterministic replay, strict version coupling, and the use of frozen output fixtures to ensure backward compatibility. The system distinguishes between V1 wire-contract stability and V2 feature enrichment through multi-layered validation gates, including shadow A/B testing and walk-forward analysis.
A critical component of this strategy is the "Frozen Fixture" methodology, where byte-identical JSON snapshots serve as regression anchors for the V1 baseline. This ensures that any changes to the engine do not silently alter existing wire formats or deterministic outputs unless explicitly authorized by specification changes.
Testing Architecture and CI/CD Integration
The testing suite is split into prioritized execution layers to balance developer velocity with comprehensive verification. This is managed through a tiered GitHub Actions configuration.
CI Workflow Layers
| Layer | Trigger | Scope |
|---|---|---|
| PR Fast CI | Pull Request | Runs default pytest scope; excludes slow and v2_shadow markers. |
| Full Verification | Push to main | Runs the full test suite, including all markers, using parallel execution (-n 8). |
| Release Workflow | Release Creation | Executes all tests to ensure production readiness. |
Sources: tests/test_readiness_contracts.py:21-65, scripts/validate_agents_md.py:12-20
Operational Discipline for Automated Agents
The project enforces strict non-negotiable rules for code changes, particularly those affecting regime classifiers. Agents must announce intent, validate at every layer boundary, and ensure that V1 wire fields are never renamed or removed without updating frozen replay fixtures.
The diagram shows the progression from intent to full verification, highlighting the mandatory validation at boundaries and CI gates.
Sources: AGENTS.md:7-35, AGENTS.md:126-136
Frozen Fixtures and V1 Replay Stability
The repository uses "frozen" JSON snapshots to anchor the V1 wire contract. These files are stored in tests/fixtures/v1_frozen_outputs/ and represent the output of RegimeOutput.model_dump_json().
Fixture Provenance and Integrity
Fixtures are byte-frozen and must not be modified in-place. They are used for round-trip equality testing. If a wire change is required, it must be accompanied by an update to the PROVENANCE.md file, which tracks the originating commit and authorization for the change.
| Snapshot File | Purpose | Baseline Commit |
|---|---|---|
2023-12-14.json | V1 Baseline (Dec 2023) | 482e44b |
2024-02-15.json | V1 Baseline (Feb 2024) | 482e44b |
2024-04-15.json | V1 Baseline (Apr 2024) | 482e44b |
Sources: tests/fixtures/v1_frozen_outputs/PROVENANCE.md:3-15
Canonicalization Process
To ensure stable SHA256 hashes for data fixtures, the system employs a canonicalization script. This script sorts Parquet rows ascending by all columns, strips schema metadata, and enforces specific compression (Snappy) and timestamp coercion (microseconds).
The flow represents the idempotent transformation of data files into a stable, verifiable format for the manifest system.
Sources: scripts/publish_canonical_snapshot.py:90-118
V2 Validation Gates
Validation of the V2 engine involves two primary mechanisms: the Shadow A/B Gate and the Walk-Forward Analysis.
Shadow A/B Gate
This gate simulates a 60-session live shadow period. It compares v1-mode (standard) against full-v2-mode (enriched).
- V1 Fields (Regression Check): Fields like
trend_directionandvolatility_statemust have zero disagreements. - V2 Fields (Activation Check): New enrichments like
transition_risk_scoreandclusterare expected to differ, representing the "wins" of the V2 implementation.
Sources: scripts/run_v2_shadow_ab_gate.py:12-45
Walk-Forward Analysis
The walk-forward analysis generates a historical performance report by iterating through NYSE sessions and comparing engine outputs against baseline metrics and "golden results."
This sequence illustrates how historical classification runs are aggregated into a final pass/fail qualification report.
Sources: scripts/build_walkforward_report.py:53-75, scripts/build_walkforward_report.py:270-295
Mandatory Testing Disciplines
As defined in the project's operational guidelines, the following disciplines are enforced for all technical changes:
- Mocking Policy: Internal code must never be mocked. External services must use captured real fixtures (redacted).
- Assertion Quality: Assertions must exercise behavior, not just structure. Exact values or counts are preferred over
is not None. - Failure Protocol: When an assertion fails, the code or fixture is considered wrong, not the assertion. Relaxing equality to inequality or widening ranges to pass a test is prohibited.
- V2 Slice Review: Any V2 component score or weight that deviates from the cited specification section results in a P1 failure.
Sources: AGENTS.md:73-86, AGENTS.md:139-146
The testing strategy ensures the regime-detection engine remains a high-integrity system by combining strict regression anchors (Frozen Fixtures) with dynamic validation (Shadow Gates and Walk-Forward Analysis), all governed by a rigorous set of automated and manual review disciplines.