Data Sources & Ingestion
Relevant source files
The following files were used as context for generating this wiki page:
ZINC Fusion V15 utilizes a high-frequency, multi-source ingestion pipeline to power its institutional-grade soybean oil (ZL) forecasting models. The system aggregates quantitative market data, economic indicators, and qualitative alternative data from over 847 global sources to feed a 19-model ensemble architecture.
Sources: README.md:1-20, Docs/zinc-fusion-v15-preview.html:1305-1315
The ingestion architecture is designed for point-in-time (PIT) correctness and strict data integrity. It employs a mix of serverless Inngest functions for scheduled jobs and Python-based scripts for complex batch processing. All data is consolidated into a Prisma-managed PostgreSQL warehouse across 12 specialized schemas, including mkt (market), econ (economic), supply (supply-chain), and alt (alternative/news).
Sources: AGENTS.md:25-35, cubic.yaml:10-30, README.md:140-155
Ingestion Architecture & Flow
The system employs a multi-step pipeline that transitions data from raw external APIs and scrapers into a structured "Elite" feature store. The flow prioritizes idempotency through row hashing and ensures data quality via automated quarantine of invalid records.
The diagram represents the flow from raw external data through the validation layer into schema-specific landing tables.
Sources: cubic.yaml:105-130, frontend/src/inngest/federal-register.ts:300-350, scripts/ingest_news_sources.py:400-450
Core Ingestion Principles
- Append-Only Semantics: Landing tables are append-only to preserve historical training integrity. Standalone
UPDATEorDELETEoperations are strictly forbidden on time-series tables. - PIT Correctness: Every row must represent what was known at the
event_dateto prevent lookahead bias in ML models. - Idempotency: Ingested records use
ON CONFLICT DO UPDATEpatterns based on arow_hash(typically a SHA256 hash of the natural key and values). - Quarantine Path: Invalid records (NaN, null, or domain-inappropriate values) are moved to
ops.quarantined_recordinstead of being silently dropped.
Sources: cubic.yaml:105-160, frontend/src/inngest/federal-register.ts:320-340
Quantitative Market & Economic Data
The project ingests daily and intraday OHLCV (Open, High, Low, Close, Volume) data for futures and macro indicators.
Primary Data Tables
| Schema | Table | Description | Data Frequency |
|---|---|---|---|
mkt | futures_1d | ZL, ZS, ZM, CL futures OHLCV | Daily |
mkt | fx_1d | Global currency pairs (CNY, BRL, DXY) | Daily |
econ | rates_1d | FRED interest rate series | Daily |
supply | epa_rin_1d | EPA RIN (Biofuel) prices | Daily |
alt | weather_1d | 215K+ observations from 57 stations | Daily |
Sources: README.md:140-155, src/fusion/api/server.py:250-300
API Integrations
- Databento: Provides high-fidelity futures data (ZL, ZS, ZM) including 1-hour and daily intervals.
- FRED (Federal Reserve Economic Data): Sources macro indicators across rates, inflation, labor, and monetary policy.
- Yahoo Finance: Used for EOD top-fill and supplemental market symbols.
Sources: README.md:140-155, scripts/_deprecated/run_audit_queries.py:50-100
Qualitative & Alternative Data
Alternative data is routed to the "Big-11" specialist buckets through a rule-based classifier that scans headlines and abstracts for specific domain keywords.
News & Policy Sources
The news ingestion pipeline pulls from over 25 agricultural, trade, and policy sources.
| Specialist Bucket | Primary Sources |
|---|---|
crush | Farm Policy News, FarmDoc Daily, USDA, DTN |
china | Reuters China, MOFCOM, Agrimoney China |
tariff | USTR, White House Briefing, Federal Register |
biofuel | EPA News, Biodiesel Magazine |
trump_effect | Truth Social, Federal Register (Exec Orders) |
Sources: scripts/ingest_news_sources.py:70-200
Automated Classification Logic
The system uses regex patterns to assign specialist_tags to incoming news. This ensures that a Section 301 tariff document from the Federal Register is correctly routed to both the tariff and china specialists for signal generation.
// Example Tag Rule logic from federal-register.ts
const TAG_RULES = [
{ pattern: /section[\s_-]?301/i, tags: ["tariff"] },
{ pattern: /\bchina\b|\bprc\b|chinese/i, tags: ["china", "tariff"] },
{ pattern: /renewable[\s_-]?fuel[\s_-]?standard|rfs/i, tags: ["biofuel"] }
];
Sources: frontend/src/inngest/federal-register.ts:50-100
Ingestion Operations & Scheduling
Ingestion is orchestrated via Inngest, running serverless functions that manage concurrency and retry logic.
Scheduling Hierarchy
- P0 Critical (Every 2 hours): Market-hours updates for prices and high-priority news.
- P1 High (Every 4-6 hours): Secondary news sources and policy updates.
- Daily Sweep (5 AM ET): Full data backfill, FRED macro indicators, and weather data consolidation.
Sources: scripts/ingest_news_sources.py:20-35, frontend/src/inngest/federal-register.ts:260-270
Operational Monitoring
Every ingestion job logs its status to ops.ingest_run, tracking:
rows_attempted: Total records found in source.rows_inserted: New records added to the database.rows_skipped: Duplicates identified byrow_hash.rows_quarantined: Invalid records rejected for quality issues.
Sources: cubic.yaml:105-120, frontend/src/inngest/federal-register.ts:360-380
Conclusion
Data sources and ingestion form the foundation of ZINC Fusion V15's forecasting capability. By enforcing strict point-in-time correctness, append-only semantics, and automated specialist routing, the system maintains a high-integrity data warehouse that powers complex ML ensemble training without temporal leakage.
Sources: cubic.yaml:160-175, AGENTS.md:85-100