Macro & Economic Data
Relevant source files
The following files were used as context for generating this wiki page:
Macro & Economic Data represents a critical subsystem of the Rabid Raccoon platform, responsible for the ingestion, storage, and transformation of global economic indicators into actionable trading signals for Micro E-mini S&P 500 (MES) futures. The system primarily leverages the Federal Reserve Economic Data (FRED) API to track a wide range of variables including interest rates, yield curves, volatility indices, and labor statistics.
This data serves as the foundation for the project's machine learning pipelines and real-time technical analysis engines. By integrating point-in-time economic data with intraday price action, the system establishes a "macro context" that defines the current market regime, filters trade directions, and adjusts confidence levels for automated forecasts.
Sources: scripts/build-lean-dataset.ts:16-25, scripts/build-complete-dataset.ts:14-23, src/lib/instant-analysis.ts:25-32
Data Ingestion Architecture
The ingestion layer is designed to handle multi-domain economic series through specialized scripts that synchronize external API data with a local PostgreSQL database via Prisma.
Ingestion Pipeline Flow
The system utilizes a structured pipeline to fetch data, validate it, and distribute it into domain-specific tables.
The ingestion process involves generating a unique rowHash based on the series ID, event date, value, and source to prevent duplicate entries during retries or overlapping backfills.
Sources: scripts/ingest-macro-indicators.ts:50-54, scripts/ingest-fred-complete.ts:40-44
Economic Domains and Series
Data is categorized into specific domains, each mapped to its own database table for optimized querying and organization.
| Domain | Table | Examples |
|---|---|---|
| RATES | econ_rates_1d | Fed Funds (DFF), SOFR, Target Ranges |
| YIELDS | econ_yields_1d | 2Y, 10Y, 30Y Treasury Yields |
| VOLATILITY | econ_vol_indices_1d | VIX, VVIX, High Yield OAS, NFCI |
| FX | econ_fx_1d | DXY (Broad), EUR/USD, JPY/USD |
| INFLATION | econ_inflation_1d | TIPS Yields, CPI, PCE, PPI |
| LABOR | econ_labor_1d | Jobless Claims, NFP, Unemployment |
| ACTIVITY | econ_activity_1d | Real GDP, Retail Sales, Consumer Sentiment |
| MONEY | econ_money_1d | Fed Assets (WALCL), Reverse Repo (RRP) |
| COMMODITIES | econ_commodities_1d | WTI/Brent Crude, Copper Prices |
Sources: scripts/ingest-fred-complete.ts:51-115, scripts/ingest-macro-indicators.ts:145-190
Dataset Construction & Feature Engineering
The system transforms raw economic time series into features suitable for machine learning (AutoGluon) and technical indicators.
Anti-Leakage Policy
To prevent data leakage during model training, the system enforces a conservative "as-of" lookup policy. This ensures that a model at any specific timestamp only sees data that was officially available at that point in time.
- Daily Data: 1-day lag
- Weekly Data: 8-day lag
- Monthly Data: 35-day lag
- Quarterly Data: 100-day lag
Sources: scripts/build-complete-dataset.ts:133-138, scripts/build-15m-dataset.ts:133-138
Derived Macro Features
Beyond raw values, the system calculates complex relationship features:
- Yield Curve Slope: Difference between 10-year and 2-year Treasury yields (
DGS10 - DGS2). - Real Rates: Nominal yields minus inflation expectations or TIPS yields (e.g.,
DGS10 - DFII10). - Fed Liquidity Proxy: Fed Total Assets minus Overnight Reverse Repo (
WALCL - RRP). - Credit Spread Diff: Difference between High Yield OAS and Investment Grade OAS (
HY_OAS - IG_OAS). - Volatility Term Structure: Ratio of short-term to long-term volatility expectations (e.g.,
VVIX / VIX).
Sources: scripts/build-complete-dataset.ts:424-436, scripts/build-lean-dataset.ts:316-324
Real-Time Analysis & Regime Filtering
Economic data is used in real-time to bias technical signals and adjust trade confidence levels.
VIX Regime Filter
The VIX level is used as a "hard filter" for directional bias in the instant-analysis engine.
Sources: src/lib/instant-analysis.ts:446-475
Market Context Modeling
The MarketContext object aggregates live economic data to determine the current trading "Regime" (e.g., "Trending Bullish", "High Volatility Reversal"). This context includes:
- Yield Context: 10Y yield changes and spreads vs Fed Funds.
- Shock Reactions: Historical analysis of how MES reacts to sudden VIX or Yield spikes.
- Theme Scores: NLP-derived scores for topics like Tariffs, Rates, and AI/Tech.
Sources: src/lib/instant-analysis.ts:522-555, src/lib/analyse-data.ts:143-160
Implementation Details
Data Structure: Economic Series Registry
Every series is registered with metadata to define its source and characteristics.
await prisma.economicSeries.upsert({
where: { seriesId: spec.seriesId },
create: {
seriesId: spec.seriesId,
displayName: spec.displayName,
category: domainToCategory(spec.domain),
source: 'FRED',
sourceSymbol: spec.seriesId,
frequency: spec.frequency,
units: spec.units,
isActive: true,
}
})
Sources: scripts/ingest-fred-complete.ts:219-234
Data Structure: Value Observation
Individual data points are stored with high-precision values and a deterministic hash for deduplication.
| Field | Type | Description |
|---|---|---|
seriesId | String | Unique identifier (e.g., 'VIXCLS') |
eventDate | DateTime | The date the value was recorded (UTC) |
value | Double | The numerical observation |
rowHash | String | Unique SHA-256 hash for deduplication |
metadata | JSON | Provider-specific info (e.g., source symbol) |
Sources: scripts/ingest-macro-indicators.ts:63-69, scripts/ingest-fred-complete.ts:140-146
The Macro & Economic Data system provides the vital "contextual overlay" necessary for futures trading. By combining reliable FRED ingestion pipelines with strict anti-leakage feature engineering, the project ensures that its AI models and real-time analysis tools remain grounded in the broader economic reality, effectively filtering technical noise through macro signals.