Python Compute Layer
Relevant source files
The following files were used as context for generating this wiki page:
The Python Compute Layer serves as the intelligence and heavy-forecasting core of ZINC-Fusion-V15. It is responsible for executing a multi-layered ensemble machine learning pipeline, managing real-time AI compute agents, and providing an API interface for the frontend dashboard. The layer translates raw market data into institutional-grade quantitative forecasts for Soybean Oil (ZL) futures across four horizons: 1 week, 1 month, 3 months, and 6 months.
Sources: README.md:1-15, AGENTS.md:3-8
Architecture Overview
The compute layer operates on a 4-layer hierarchical ensemble architecture (v3). It utilizes AutoGluon for core time-series forecasting and custom Python scripts for specialist signal generation and LLM-based intelligence synthesis.
Model Hierarchy (v3)
| Layer | Component | Description |
|---|---|---|
| L0 Core | AutoGluon TimeSeriesPredictor | 4 models (one per horizon) using a 25-model zoo. |
| L0 Specialists | 11 Signal Generators | Custom models (XGB, GBM, RF, etc.) producing horizon-agnostic signals. |
| L1 Meta | Stacked Ensembles | Combines Core out-of-fold (OOF) predictions with specialist signals. |
| L2 Calibration | Conformalized Quantile Regression | Produces outer risk envelopes (p10_cal/p90_cal). |
| L3 Risk | Monte Carlo Engine | Simulates 5,000+ paths for VaR, CVaR, and barrier probabilities. |
Sources: README.md:17-35, AGENTS.md:25-35
The following flowchart illustrates the high-level data flow through the Python Compute Layer:
The diagram shows the progression from raw data through specialist and core models to the final risk engine.
Sources: README.md:145-160
AI Compute Agents
The system features an "Agent Pool" framework that provides soft intelligence on top of heavy numerical forecasts. These agents utilize Large Language Models (LLMs) like Claude to analyze news, synthesize market posture, and correct traditional sentiment analysis models.
Pulse Engine and Synthesis
The Pulse Engine generates pre-quantified intelligence drops for the 11 specialist domains. These drops include direction, pressure (cents/lb), and confidence (edge). The Synthesis Engine then translates these quantitative outputs into natural language summaries for decision support.
Sources: src/fusion/ai_compute/agent_pool.py:1-25, scripts/generate_synthesis.py:15-30
Sentiment Intelligence
The SentimentScorerAgent provides news analysis with specific domain expertise in the Soybean Oil market. It includes "Inversion Awareness" to correctly interpret commodity-specific news (e.g., falling production being bullish).
This sequence diagram tracks the generation of an "Intel Drop" from data retrieval to database storage.
Sources: scripts/generate_pulses.py:45-65
API Layer
The API layer, built with FastAPI, exposes read-only endpoints for the frontend. It handles database queries via a unified abstraction layer and provides real-time pressure indicators for key market drivers.
Key API Endpoints
| Endpoint | Method | Description |
|---|---|---|
/api/dashboard/summary | GET | Latest ZL price, changes, and recommended procurement actions. |
/api/market-drivers | GET | Real-time pressure scores (0-100) for VIX, Crush, China, and Tariff. |
/api/forecast/quantiles | GET | Time-horizon forecasts (p10, p50, p90). |
/api/pulse/latest | GET | Recent AI-generated intelligence drops by domain. |
/api/db/query | POST | Secure DB explorer for SELECT/WITH queries (requires token). |
Sources: src/fusion/api/server.py:80-120, src/fusion/api/server.py:850-900
Database Interaction
The layer uses psycopg2 for Python-based runtime queries. It interacts with 12 distinct schemas in Prisma Postgres, including mkt, econ, training, forecasts, and analytics.
Sources: AGENTS.md:18-23, src/fusion/api/server.py:55-65
Development and Training Workflow
The Python Compute Layer enforces strict data contracts and validation gates to prevent data leakage and ensure forecast integrity.
Matrix Construction
The training.matrix_1d is assembled from over 213 features, including FRED macro data, FX rates, weather, and CFTC positioning. It utilizes point-in-time (PIT) correct staleness encoding and TTL-bounded forward fill for economic data.
Sources: src/fusion/core_training/build_matrix.py:35-65
Validation Gates
Training does not pass unless it satisfies specific gates:
- Quantile Monotonicity: Uncertainty ranges must widen with longer horizons.
- OOF Discipline: Meta-learners must only train on out-of-fold Core predictions.
- Coverage Checks: Empirically validated coverage for p10/p90 envelopes (~80%).
Sources: README.md:162-175, AGENTS.md:40-45
Conclusion
The Python Compute Layer represents the quantitative engine of ZINC-Fusion-V15. By combining traditional statistical models, deep learning ensembles via AutoGluon, and modern LLM-based agent intelligence, it provides a robust framework for commodity price forecasting. Its architecture ensures that heavy numerical work is handled by specialized models while soft intelligence and synthesis are provided by the AI compute agent pool, all served through a high-performance FastAPI interface.