Feature Engineering
Relevant source files
The following files were used as context for generating this wiki page:
Feature Engineering in ZINC-Fusion-V15 is a multi-layered process designed to transform raw market, economic, and alternative data into high-conviction signals for Soybean Oil (ZL) procurement forecasting. The system distinguishes between "Core" features—comprising 800+ indicators used by AutoGluon ensembles—and "Specialist" features, which are hand-picked, domain-specific indicators (20-50 per domain) curated for 11 specialized signal generators.
The architecture follows a strict "Model Reality, Don't Moralize It" doctrine, utilizing a 4-layer ensemble stack. Feature engineering spans from technical indicators and cross-asset correlations to advanced policy deadline tracking and AI-driven sentiment scoring. All data operations preserve Point-In-Time (PIT) semantics to prevent data leakage, ensuring that training matrices only contain information available at the time of the forecast.
Sources: README.md, AGENTS.md, scripts/generate_specialist_features.py:1-25
Feature Matrix Architecture
The system assembles a comprehensive training matrix (training.matrix_1d) from over 12 database schemas. This matrix serves as the foundation for both L0 Core models and L0 Specialist models.
Data Flow and Assembly
The construction of the feature matrix involves merging high-frequency market data with lower-frequency economic reports and alternative data.
The assembly process utilizes a merge_asof strategy to align weekly/monthly data (like CFTC COT or WASDE reports) to the daily trading calendar, ensuring that the first available trading day receives the new information without lookahead bias.
Sources: src/fusion/core_training/build_matrix.py:40-60, src/fusion/core_training/build_matrix.py:350-380
Feature Guardrails
To maintain model stability, the system enforces strict feature count guardrails.
| Parameter | Value | Description |
|---|---|---|
| MIN_FEATURES | 120 | Minimum required features for a valid matrix |
| MAX_FEATURES | 350 | Hard ceiling to prevent dimensionality explosion |
| TARGET_FEATURES | ~213 | Optimal feature set size after curation |
Sources: src/fusion/core_training/build_matrix.py:1150-1180
Specialized Feature Domains
Features are categorized into "Big-11" buckets, each handled by a domain specialist. Unlike the Core model which uses an "All Data" policy, specialists use expert-curated subsets.
Specialist Feature Sets
| Specialist | Primary Features | Purpose |
|---|---|---|
| CRUSH | Board crush, oil share, ZL/ZS/ZM ratios | Processing economics |
| CHINA | Copper (HG), USD/CNY, Export sales | Import demand signals |
| ENERGY | CL, HO, BOHO spread, Crack spreads | Biodiesel feedstock economics |
| TARIFF | Policy uncertainty, Trade war sentiment | Regulatory and trade risk |
| PALM | Palm/Soy ratio, MPOB inventory | Substitution pressure |
Sources: scripts/generate_specialist_features.py:45-150, AGENTS.md
AI-Compute Intelligence Layer
A soft intelligence layer generates features from unstructured data using the SentimentScorer agent. This agent processes news from 25+ sources and corrects "commodity-blind" sentiment scores (e.g., identifying that falling production is bullish due to supply shocks, even if NLP models label "fall" as bearish).
Sources: src/fusion/ai_compute/agent_pool.py:120-160, src/fusion/ai_compute/agent_pool.py:200-230
Advanced Feature Engineering Techniques
Policy Deadline Tracking
The TariffDeadlineFeatureEngine generates time-sensitive features based on known regulatory milestones. It uses a sigmoid function to calculate a deadline_risk_score, where risk accelerates as the expiry date approaches.
Forward Fill Policy (TTL-Bounded)
The project implements a strict Time-To-Live (TTL) bounded forward fill policy to handle sparse economic data while preventing the use of obsolete information.
- Daily Data: 3-day TTL (accommodates weekends).
- Weekly Data: 10-day TTL.
- Monthly Data: 45-day TTL.
- Market Prices: NEVER forward-filled.
Every forward-filled feature must be accompanied by an _age_days column, allowing the ML models to "see" the staleness of the data.
Sources: src/fusion/core_training/build_matrix.py:75-110, src/fusion/core_training/build_matrix.py:220-250
Missingness and Categorical Encoding
Instead of simple imputation, the system uses missingness encoding for daily features.
- Original Value: Filled with 0.0.
- Missing Flag:
{col}_is_missingset to 1. - Categorical Encoding: Ordinal mapping for regimes (e.g.,
hurst_regimemapped to 0=unknown, 1=random, 2=trending).
Sources: src/fusion/core_training/build_matrix.py:650-680, src/fusion/core_training/build_matrix.py:1000-1030
Technical Implementation Snippets
Position Proxy Calculation
Daily positioning flow signals are derived from Open Interest (OI) and Volume to provide intra-week dynamics between weekly CFTC reports.
# src/fusion/core_training/build_matrix.py:1255-1275
def compute_daily_positioning_proxies(df: pd.DataFrame) -> pd.DataFrame:
if "open_interest" in df.columns:
df["oi_delta_1d"] = df["open_interest"].diff(1)
df["oi_pct_change_1d"] = df["open_interest"].pct_change(1) * 100
# Z-score unusual activity (63d = 1 quarter)
oi_mean_63d = df["oi_delta_5d"].rolling(63).mean()
oi_std_63d = df["oi_delta_5d"].rolling(63).std()
df["oi_delta_zscore"] = (df["oi_delta_5d"] - oi_mean_63d) / oi_std_63d
return df
Sources: src/fusion/core_training/build_matrix.py:1255-1285
Conclusion
Feature engineering in ZINC-Fusion-V15 is a rigorous pipeline that combines institutional domain expertise with modern ML practices. By separating Core and Specialist feature policies, enforcing strict TTL-bounded data persistence, and utilizing AI to interpret complex market narratives, the system creates a robust, high-dimensional input space optimized for multi-horizon Soybean Oil price forecasting.