Data Quality & Audits
Relevant source files
The following files were used as context for generating this wiki page:
The Data Quality & Audits system in ZINC-Fusion-V15 provides an institutional-grade validation framework designed to ensure the integrity of commodity forecasting data. It acts as a "gatekeeper" for the multi-layer ensemble ML pipeline, preventing "quiet corruption" or "blind spots" that could lead to multimillion-dollar procurement errors. The system encompasses automated validation gates, historical audit scripts, and real-time monitoring of data freshness and source distribution.
Sources: src/fusion/validation/data_quality.py:100-110, cubic.yaml:80-90
Core Validation Framework
The primary mechanism for data integrity is the DataQualityGatekeeper (also aliased as DataQualityValidator). This component performs schema integrity checks, null value analysis, and primary key uniqueness validation on incoming dataframes or Parquet files.
Validation Workflow
The system utilizes a structured reporting mechanism where individual checks generate ValidationResult objects, which are aggregated into a ValidationReport. If strict_mode is enabled, any "ERROR" level failure raises a DataQualityError, halting the ingestion pipeline.
The diagram above shows the sequential validation process applied to all incoming datasets to ensure they meet institutional standards before being consumed by ML models.
Sources: src/fusion/validation/data_quality.py:30-90, src/fusion/validation/data_quality.py:115-150
Key Validation Functions
| Function | Targeted Data Source | Key Validations |
|---|---|---|
validate_databento_futures_1d | Databento daily OHLCV | symbol, as_of_date, OHLCV columns |
validate_fred_economic | FRED Indicators | series_id, date, value |
validate_ml_matrix | Training Matrices | as_of_date, close, 30% null threshold |
validate_cftc_cot | Commitment of Traders | symbol, report_date, open_interest |
Sources: src/fusion/validation/data_quality.py:354-405
Ingestion Integrity Rules
ZINC-Fusion-V15 enforces surgical rules to prevent data contamination and preserve point-in-time (PIT) semantics. These rules are critical for ML training to avoid lookahead bias.
- Append-Only Landing Tables: Landing tables (schemas:
mkt,econ,supply, etc.) are append-only. StandaloneUPDATEorDELETEstatements are prohibited outside of specificON CONFLICTpatterns. - Deduplication Pattern: All new code must use the canonical pattern:
INSERT ... ON CONFLICT (natural_key) DO UPDATE SET ...utilizing arow_hashfor change detection. - Mandatory Logging: Ingestion functions are required to log to
ops.ingest_run, recordingrows_attempted,rows_inserted,rows_skipped, androws_quarantined. - Quarantine Mechanism: Invalid records (e.g., negative prices, NaN) must be moved to
ops.quarantined_recordrather than being silently dropped.
Sources: cubic.yaml:94-145, cubic.yaml:153-170
Audit and Staleness Monitoring
The project includes specific scripts and SQL queries for deep-tissue audits of historical data state and real-time freshness monitoring.
Data Freshness (TTL)
The system tracks "Time-To-Live" (TTL) for various data sources. If data exceeds its TTL, it is marked as "Stale" in monitoring dashboards.
| Data Source | TTL (Days) | Frequency Type |
|---|---|---|
econ.rates_1d | 3 | Daily (Business) |
pos.cftc_1w | 10 | Weekly |
supply.usda_wasde_1m | 45 | Monthly |
supply.epa_rin_1d | 75 | Irregular |
Sources: tests/test_data_loaders_staleness.py:127-148
Databento State Audit
The audit suite identifies price discontinuities (jumps >5%) and coverage gaps in high-frequency data. It checks for "Source Conflicts" where the same timestamp has overlapping data from different sources (e.g., databento vs databento_live).
The sequence diagram illustrates the comprehensive audit process performed by run_audit_queries.py to detect integration issues.
Sources: scripts/_deprecated/run_audit_queries.py:35-150, scripts/_deprecated/audit_databento_state.sql:5-90
Monitoring and Registry
The model.model_registry and ops.data_quality_metrics tables provide the backend for Grafana dashboards. These monitor:
- Hours Since Update: Calculated as
EXTRACT(EPOCH FROM (NOW() - last_update)) / 3600. - Model Performance: Tracking MASE (Mean Absolute Scaled Error) and 80% coverage rates.
- Dissent Index: Measuring disagreement between the 11 specialist models.
Sources: grafana/model_registry_queries.sql:35-60, scripts/generate_synthesis.py:168-185
Preflight Verification
Before deployment or major changes, the preflight_check.py script validates the environment. It performs the following checks:
- Structure: Verifies existence of
prisma/schema.prismaand critical directories. - Environment: Checks for
DATABASE_URLandFRED_API_KEY. - Database Connectivity: Tests connection and verifies that banned schemas (e.g.,
raw,gold,silver) are not present in the Prisma configuration.
Sources: scripts/preflight_check.py:48-200
In summary, the Data Quality & Audits system ensures that ZINC-Fusion-V15 operates on clean, timely, and mathematically defensible data. By enforcing strict ingestion rules, mandatory logging, and continuous staleness monitoring, the system provides the reliable data foundation necessary for institutional-grade commodity forecasting.