Sentiment Analysis Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Sentiment Analysis Engine is a multi-layered intelligence system within ZINC-Fusion-V15 designed to quantify the impact of news and alternative data on CBOT Soybean Oil (ZL) futures. It transforms qualitative text from global news sources, regulatory filings, and trade publications into quantitative signals that feed directly into the project's Specialist Models. The engine supports procurement decisions by classifying sentiment as "bullish" (prices likely rising) or "bearish" (prices likely falling).
The system employs a hybrid architecture combining high-speed keyword-based regex scoring for historical backfills and large-scale data ingestion with advanced AI agents (Claude 4.5/Sonnet) for real-time, context-aware "soft intelligence." It specifically addresses commodity-market nuances, such as "inversion awareness," where terms like "supply fall" are correctly interpreted as bullish catalysts rather than bearish indicators.
## Architecture and Scoring Logic
The engine operates through three primary layers: Keyword-based Scoring, AI Compute Agents, and Aggregation Services.
### Keyword-based Scoring Layer
This layer provides high-performance sentiment classification by scanning headlines and summaries for domain-specific weighted patterns. It is used primarily when structured sentiment data is missing from the database.
- Negation Detection: The system uses a 4-word lookahead/lookbehind window to detect negators (e.g., "no", "unlikely"). If a keyword falls within a negation span, its polarity is flipped at a 60% weight reduction.
- Weighted Pattern Accumulation: Keywords are categorized into "Bullish" and "Bearish" banks with calibrated weights (ranging from 0.5 to 2.5). A net score is calculated as
bullScore - bearScore. - Confidence Calculation: Confidence is derived as
min(1, |net| / 5), requiring a net difference of 5 points for full certainty.
Sources: frontend/src/lib/sentiment-scorer.ts:15-45, frontend/src/lib/sentiment-scorer.ts:160-185
### AI Intelligence Layer (Claude)
The AI Compute Layer, specifically the SentimentScorerAgent, uses Claude models to perform deep analysis that standard NLP models (like FinBERT) often misinterpret.
The AI Agent evaluates context to ensure supply shocks are marked bullish and demand destruction is marked bearish.
Key features of the AI layer include:
- Inversion Awareness: Corrects FinBERT errors where "Argentina production FELL" would be incorrectly marked bearish; the agent identifies this as a bullish supply shock.
- Specialist Routing: Maps articles to one or more of the "Big-11" domains (Crush, China, Tariff, Biofuel, etc.).
- SHAP Integration: Aligns sentiment with SHAP feature importance, such as weighting Biofuel legislation as a top positive driver (+0.12).
Sources: src/fusion/ai_compute/agent_pool.py:80-120, frontend/src/lib/ai-intelligence.ts:155-180
## Component Breakdown
### Scoring Metrics and Thresholds
The engine outputs specific quantitative values used for dashboard visualization and model training.
| Metric | Range / Threshold | Description |
|---|---|---|
| Impact Score | -1.0 to +1.0 | Quantitative direction: +0.7+ (Major Bullish), -0.7+ (Major Bearish). |
| Confidence | 0.0 to 1.0 | 0.9+ implies direct explicit impact; 0.5-0.6 implies indirect inference. |
| Sentiment Bias | +/- Sigma (σ) | Calculated as (bullish - bearish) / total across a 30-day window. |
| Agg. Sentiment | -1.0 to +1.0 | The mean impact score of articles within a 24h batch. |
Sources: src/fusion/ai_compute/agent_pool.py:125-150, frontend/src/app/sentiment/page.tsx:300-325
### Specialist Routing Matrix
Articles are automatically routed to specialists based on keyword triggers and AI analysis.
| Specialist | Trigger Keywords / Context |
|---|---|
| Crush | US processing, margins, NOPA crush, oil share. |
| China | GACC import flows, CNY/USD rates, trade relations. |
| Biofuel | RFS, RIN prices, 45Z credits, SAF demand. |
| Tariff | Section 301, USTR, trade war, retaliation. |
| Volatility | VIX, market stress, risk-on/risk-off sentiment. |
Sources: src/fusion/ai_compute/agent_pool.py:100-115, scripts/_deprecated/backfill_sentiment_scores.py:65-90
## Data Flow and API Integration
The Sentiment Engine exposes several endpoints via the FastAPI server to serve the frontend dashboard and monitoring tools.
Sequence showing the transformation of raw news rows into structured sentiment signals.
- Endpoint
/api/sentiment/series: Provides a historical time-series of sentiment scores for charting, typically scanning up to 5,000 articles to generate daily averages. - Endpoint
/api/sentiment/news: Returns the latest articles with their computedimpact_score,alert_buckets, andconfidence. - Database Schema: Sentiment data is persisted in the
features.news_sentiment_1dtable, including flags likeis_zl_relevantandaffects_trump_effect.
Sources: src/fusion/api/server.py:550-580, scripts/_deprecated/backfill_sentiment_scores.py:180-210
## Conclusion
The Sentiment Analysis Engine serves as the qualitative-to-quantitative bridge for ZINC-Fusion-V15. By combining deterministic keyword scoring with high-conviction AI analysis, the system ensures that market-moving news—from South American weather patterns to US trade policy shifts—is accurately quantified. Its ability to distinguish between different types of "price declines" and supply-chain disruptions provides a critical edge in forecasting ZL futures volatility and direction.