Testing & QA
Relevant source files
The following files were used as context for generating this wiki page:
Testing and Quality Assurance (QA) in ZINC-Fusion-V15 is a multi-layered discipline designed to ensure the integrity of a complex commodity forecasting system. It spans from basic project structure verification and environment sanity checks to advanced machine learning (ML) ablation testing and data staleness monitoring. The system enforces strict "gates" that must be passed before code changes are committed or deployed, ensuring that the critical data pipeline and model architectures remain robust.
The strategy is divided into localized environment checks, automated integration gates, and specialized data audit frameworks. These processes protect against common failure modes such as silent data corruption, "ghost" files that exist locally but not in CI, and model performance degradation.
Sources: scripts/verify.sh:1-12, CLAUDE.md:1-20, scripts/preflight_check.py:1-12
Automated Verification Gates
The project utilizes a central verification gate, scripts/verify.sh, which acts as the single source of truth for quality checks. This script is mandatory for AI agents and developers to run before claiming a task is complete. It implements a blocking mechanism where any single failure halts the process.
Verification Categories
| Gate Type | Tool/Method | Description |
|---|---|---|
| Python Core | pytest, ruff | Validates code logic, linting (specifically F401/F403/F821), and venv integrity. |
| Frontend | npm lint, tsc | Ensures Next.js/TypeScript code is valid and compiles without errors. |
| Database | prisma validate | Checks the schema.prisma file for architectural validity. |
| Security | gitleaks | Scans the repository for hardcoded secrets. |
| Integrity | scripts/prisma_status.sh | Checks for unapplied database migrations. |
Sources: scripts/verify.sh:25-95
Verification Workflow
This diagram illustrates the sequential flow of the automated verification gate.
The verification script identifies "Ghost" files (untracked TypeScript files) and ensures that all modules exported from functions.ts are git-tracked to prevent "works locally, fails on CI" incidents.
Sources: scripts/verify.sh:100-130, scripts/check_git_integrity.sh:1-15
Preflight Integrity Checks
The scripts/preflight_check.py utility is designed for concrete verification of the project environment. It is used both before and after making changes to verify project structure, imports, and connectivity.
Key Preflight Modules
- Structure Verification: Checks for the presence of critical files like
AGENTS.md,CLAUDE.md, and theprisma/directory. - Environment Validation: Ensures
DATABASE_URLandFRED_API_KEYare present in the.envfile. - Import Resolution: Confirms that critical libraries like
psycopg2,pandas,sklearn, andsqlalchemyresolve correctly within the virtual environment. - Database Connectivity: Tests the actual connection to the PostgreSQL database and performs a count of tables in key schemas (
mkt,training,analytics).
Sources: scripts/preflight_check.py:38-160, scripts/preflight_check.py:200-245
Data Quality & Staleness Testing
Because the system relies on time-series data for ML, specific tests are dedicated to monitoring data health and the behavior of the data loaders.
Forward-Fill and Staleness Logic
The system uses a "Double-Defense" pattern for data integrity. Unit tests in tests/test_data_loaders_staleness.py verify that:
- Forward-Fill Limits: Data loaders stop filling gaps after a predefined number of periods (e.g., 10 days) to prevent stale data from corrupting model features.
- Staleness Tracking: Utilities compute the number of days since the last real observation, returning a "No Data Indicator" (value: 999) if the series is entirely empty or NaN.
- NaN Conversion: Before writing to the database,
NaNvalues are strictly converted toNoneto prevent SQL insertion errors.
Sources: tests/test_data_loaders_staleness.py:25-50, tests/test_data_loaders_staleness.py:80-120, tests/test_data_loaders_staleness.py:180-205
Data Freshness Thresholds
The project defines specific Time-To-Live (TTL) values for different data sources to ensure freshness.
| Source Table | TTL (Days) | Cadence Type |
|---|---|---|
supply.usda_wasde_1m | 45 | Monthly |
pos.cftc_1w | 10 | Weekly |
econ.rates_1d | 3 | Daily (Business) |
supply.epa_rin_1d | 75 | Irregular/Monthly |
Sources: tests/test_data_loaders_staleness.py:230-255
Machine Learning Ablation Testing
A specialized framework, scripts/ablation_signal_test.py, is used to test the contribution of individual "Specialist" signals to the Core model's performance. This ensures that new signals provide actual predictive value before being integrated.
The Ablation Process
The framework compares two models: a Baseline (Core trained without the specialist signal) and a Signal-Enabled model (Core trained with the signal).
Acceptance Criteria
Signals are evaluated based on:
- MAE/MASE: Mean Absolute Error must not increase.
- Quantile Coverage: p30-p70 coverage must not decrease.
- Regime Stability: The signal's correlation with the target must remain consistent across three defined eras:
pre_2020,pandemic, andpost_2022.
Sources: scripts/_deprecated/ablation_signal_test.py:1-40, scripts/_deprecated/ablation_signal_test.py:190-230, scripts/_deprecated/ablation_signal_test.py:235-270
Forensic Data Audits
For data sources like Databento, the system runs forensic audit queries (scripts/_deprecated/run_audit_queries.py) to detect anomalies that automated unit tests might miss. These audits check for:
- Price Discontinuities: Percent changes exceeding 5% in the 15m or 1d intervals.
- Source Conflicts: Different values for the same timestamp coming from different providers.
- Coverage Gaps: Time intervals exceeding 30 minutes in intraday data.
- Freshness Age: Calculating the age of the latest record in hours to detect pipeline stalls.
Sources: scripts/_deprecated/run_audit_queries.py:75-150, scripts/_deprecated/run_audit_queries.py:200-240
In conclusion, the Testing & QA system in ZINC-Fusion-V15 is designed to be preventative rather than reactive. By enforcing mandatory verification gates, validating data staleness at the loader level, and performing ablation tests on ML signals, the project ensures that forecasting intelligence is built on a mathematically defensible and technically sound foundation.