Grafana Monitoring

Relevant source files

The following files were used as context for generating this wiki page:

• grafana/model_registry_queries.sql
• grafana/dashboards/zinc-fusion-unified.json
• grafana/dashboards/zinc-fusion-training-monitor.json
• grafana/dashboards/zinc-fusion-model-registry.json
• grafana/dashboards/zinc-fusion-training-registry.json
• grafana/scripts/generate_specialist_dashboards.py

Grafana Monitoring in the ZINC-Fusion-V15 project provides a comprehensive observability layer for the multi-layer ensemble ML pipeline and the underlying data infrastructure. It serves as the primary interface for developers and software architects to monitor model performance, data freshness, and system health in real-time. The monitoring system is designed to track the lifecycle of 19 different models, including 4 Core ensembles and 11 Specialist signal generators, ensuring that procurement forecasting for soybean oil (ZL) remains accurate and reliable.

The monitoring stack utilizes PostgreSQL as its primary data source, querying metadata across various schemas such as model, training, ops, and analytics. This allows for high-granularity tracking of training runs, out-of-fold (OOF) prediction accuracy, and infrastructure metrics like CPU and memory usage during training cycles.

Monitoring Architecture and Data Flow

The monitoring architecture is centered around a series of specialized dashboards that pull data directly from the Prisma Postgres database. Data flows from the ingestion pipelines and ML training scripts into specific database schemas, which are then queried by Grafana to visualize system states.

Key Dashboard Modules

1. Model Registry and Lifecycle

The Model Registry dashboard provides a central view of all registered models within the system. It tracks model versions, algorithms used, and "Champion" status—identifying which model is currently active for production forecasts.

Panel Name	Description	Source Schema/Table
Champion Models	Displays MASE metrics and algorithms for active champions.	model.model_registry
Specialist Status	Tracks training time and success for the "Big 11" specialists.	model.model_registry
Training Success Rate	Visualizes the pass/fail ratio of runs in the last 30 days.	ops.training_runs
MASE by Horizon	Bar chart comparing Mean Absolute Scaled Error across 5d, 21d, 63d, and 126d.	model.model_registry

Sources: grafana/model_registry_queries.sql:18-40, grafana/dashboards/zinc-fusion-model-registry.json:1-200

2. Training Infrastructure Monitor

This module correlates hardware performance with training logs. It specifically monitors macOS node integrations, focusing on resource pressure during the execution of the AutoGluon model zoo.

• CPU Usage: Monitored via Prometheus (node_cpu_seconds_total) to detect training load peaks.
• Memory Pressure: Tracks "Wired," "Compressed," and "Internal" memory bytes to prevent Out-Of-Memory (OOM) kills during heavy ensemble training.
• Disk I/O: Monitors read/write speeds for model loading and checkpointing.
• Logs: Integrated with Loki to stream zinc-fusion/training job logs directly into the panel.

Sources: grafana/dashboards/zinc-fusion-training-monitor.json:34-118, grafana/dashboards/zinc-fusion-training-monitor.json:173-230

3. Data Quality and Freshness

Data freshness is critical for forecasting. The monitoring system tracks the "Hours Since Update" for every external data source (e.g., Market Futures, Economic indicators).

Specialist Model Monitoring

The system includes a script to dynamically generate dashboards for the "Big 11" specialist signal generators (Crush, China, FX, Fed, etc.). Each specialist dashboard tracks:

• Signal Direction: Visualized as quantile forecasts (P30, P50, P70).
• Coverage %: Calculates how often the actual target value falls within the P30-P70 band.
• Bias and MAE: Quantitative metrics for each horizon (5d to 126d).

# Specialist Configuration Example
SPECIALISTS = {
    "crush": {"title": "Crush Spread", "inputs": ["ZL", "ZS", "ZM", "CFTC"]},
    "tariff": {"title": "Tariff Policy", "inputs": ["EPU Trade", "China TPU", "Duties"]},
}

Sources: grafana/scripts/generate_specialist_dashboards.py:8-48, grafana/scripts/generate_specialist_dashboards.py:145-175

SQL Data Extraction Logic

The system relies on complex SQL queries to derive performance metrics from raw training data.

OOF Coverage Query

This query determines the reliability of specialist signals by checking if actual values fall within predicted quantile bands.

SELECT
    specialist as "Model",
    COUNT(*) as "Predictions",
    SUM(CASE WHEN actual >= pred_p30 AND actual <= pred_p70 THEN 1 ELSE 0 END) as "In Band",
    ROUND(100.0 * SUM(...) / COUNT(*), 1) as "P30-P70 Coverage %"
FROM model.v_oof_predictions
WHERE actual IS NOT NULL
GROUP BY specialist, horizon;

Sources: grafana/model_registry_queries.sql:108-125

Conclusion

Grafana Monitoring provides the necessary observability to maintain the ZINC-Fusion-V15 forecasting pipeline. By integrating system-level resource metrics with high-level ML performance data, the system allows for rapid identification of stale data, failing training runs, or model degradation. This integrated approach ensures the institutional-grade reliability required for high-stakes procurement decisions.