Inngest Automation
Relevant source files
The following files were used as context for generating this wiki page:
Inngest Automation serves as the background job orchestration layer for the Rabid Raccoon trading platform. It manages complex, multi-step workflows including daily market data ingestion, macro indicator updates, and historical backfilling of Micro E-mini S&P 500 (MES) futures data. The system is designed for high reliability using deterministic replay, ensuring that side effects are only executed once even during retries.
The automation engine is isolated from other projects using a specific client ID and namespaced events to prevent cross-project interference. It interacts with various external data sources such as Databento, FRED, and institutional RSS feeds, persisting processed data into a PostgreSQL database via Prisma ORM.
Sources: cubic.yaml:22-29, README.md:103-111, src/inngest/backfill-mes.ts:18-25
System Architecture
The Inngest architecture consists of a central client, a serve handler route, and modular function definitions. All background operations are defined as Inngest functions that listen for specific events or follow a cron-based schedule.
Core Components
- Inngest Client: Configured with the unique ID
rabid-raccoonto ensure isolation within shared Vercel environments. - Serve Handler: The API route located at
/api/inngestwhich registers and exposes all background functions to the Inngest platform. - Step Execution: Functions utilize
step.run()to wrap all side effects (database writes, API calls, logging), which is critical for deterministic replay and preventing data corruption.
Sources: cubic.yaml:22-38, src/app/api/inngest/route.ts:5-13
Architecture Flow
The following diagram illustrates how the Inngest serve handler mediates between the Inngest Cloud and the local application logic.
The Inngest Cloud triggers functions via HTTP POST requests to the registered API route, which then invokes the defined logic.
Sources: src/app/api/inngest/route.ts:11-13, src/inngest/functions.ts:38-42
Daily Ingestion Pipeline
The dailyIngestionJob is the primary automated workflow, scheduled via cron to run daily at 07:00 UTC. It executes a series of sequential and parallel steps to refresh the system's data state.
Execution Steps
- Market Data: Pulls daily market prices for futures symbols in two distinct batches (e.g., MES, ES, NQ in batch one; ZN, ZB, CL in batch two).
- Macro Indicators: Updates FRED-sourced indicators including VIX, Federal Funds Rate, and 10-Year Treasury yields.
- Alternative News: Fetches institutional news from RSS feeds (Fed, SEC, ECB, etc.) and categorizes them into economic or policy news.
- Economic Calendar: Ingests tier-based release data and earnings calendars.
- News Scrape: Performs targeted scraping for specific layers like "trump_policy" or "volatility".
Sources: src/inngest/functions.ts:38-123, scripts/ingest-alt-news-feeds.ts:40-90, scripts/ingest-macro-indicators.ts:153-186
| Batch/Job | Frequency | Data Source | Files Reference |
|---|---|---|---|
| Market Daily | Daily (Cron) | Databento | functions.ts, ingest-market-prices-daily.ts |
| Macro Indicators | Daily (Cron) | FRED | ingest-macro-indicators.ts |
| Institutional News | Daily (Cron) | RSS Feeds | ingest-alt-news-feeds.ts |
| Measured Move | Daily (Cron) | Internal DB | ingest-mm-signals.ts |
Sources: src/inngest/functions.ts:43-60, scripts/ingest-alt-news-feeds.ts:251-260
Backfill Operations
The backfillMesAllTimeframes function is a specialized workflow for filling historical data gaps (2019 to present). It processes data in 7-day chunks to remain resumable and avoid timeout limits.
Backfill Logic
- Data Source: Fetches
ohlcv-1mdata from Databento. - Aggregation: Resamples 1-minute data into 15m, 1h, and 1d timeframes.
- Validation: Filters for candles where OHLC values are greater than zero.
- Batching: Inserts data using a batch size of 100 rows to comply with Prisma Accelerate limits.
Sources: src/inngest/backfill-mes.ts:18-45, src/inngest/backfill-mes.ts:108-144
The backfill process loops through day chunks, transforming raw 1m data into multiple timeframe candles before persistence.
Sources: src/inngest/backfill-mes.ts:145-212
Data Integrity and Safety
Inngest functions enforce strict data integrity rules defined in the project's architectural guidelines.
Critical Rules
- Idempotency: All side effects (database writes, external API calls) must be wrapped in
step.run()to prevent duplicate operations during deterministic replays. - Event Namespacing: Events must use prefixes like
rr/orbackfill/to avoid cross-project "fan-out" triggers. - BigInt Volume: Volume data for MES must be handled as
BigIntto avoid precision loss fromNumber.MAX_SAFE_INTEGERoverflows. - Batch Limits: Prisma
createManycalls are hard-limited to 100 rows per batch to ensure compatibility with Prisma Accelerate.
Sources: cubic.yaml:32-47, cubic.yaml:70-76, src/inngest/backfill-mes.ts:31
Data Persistence Schema
Automation jobs primarily populate the following tables:
mkt_futures_mes_15m,mkt_futures_mes_1h,mkt_futures_mes_1decon_news_1d,policy_news_1dingestionRun(for logging job status and results)
Sources: src/inngest/backfill-mes.ts:40-100, scripts/ingest-alt-news-feeds.ts:310-330, scripts/ingest-macro-indicators.ts:241-255
Summary
Inngest Automation provides the Rabid Raccoon platform with a resilient, multi-step background processing engine. By leveraging Inngest's deterministic replay and step.run() semantics, the system ensures high-integrity data ingestion for market prices, macro indicators, and news. The combination of scheduled daily jobs and on-demand backfill functions allows the platform to maintain a comprehensive and up-to-date dataset for trading signal generation.
Sources: src/inngest/functions.ts:38-42, src/inngest/backfill-mes.ts:18-25, cubic.yaml:32-38