Ingestion Scripts & CLI
Relevant source files
The following files were used as context for generating this wiki page:
The Ingestion Scripts and CLI module represents the data acquisition layer of the Rabid Raccoon platform. This system is responsible for the extraction, transformation, and loading (ETL) of market data, macroeconomic indicators, and institutional news from various external providers into a centralized PostgreSQL database via Prisma ORM. It supports both high-latency bulk backfills and low-latency live streaming updates.
The architecture is designed with a "zero-fake-data" policy, ensuring that all stored information is derived from verifiable institutional sources such as Databento, FRED (Federal Reserve Economic Data), and various regulatory RSS feeds. The scripts are primarily executed as standalone CLI tools during development and as automated background jobs via Inngest in production.
Sources: scripts/ingest-market-prices.ts:18-24, README.md:126-135
Core Ingestion Framework
The ingestion framework utilizes a shared set of utilities to handle common tasks such as date normalization, chunking large requests, and calculating cryptographic hashes for row-level deduplication.
Data Integrity and Validation
A critical component of the ingestion logic is the validation of OHLCV (Open, High, Low, Close, Volume) data. The system enforces strict checks to prevent the insertion of corrupt or "fake" candles.
- Deduplication: Every row is assigned a
rowHashcomputed from deterministic keys like symbol, timestamp, and price. - Validation: Price values must be positive and finite. Volume must be handled as
BigIntto prevent precision loss. - Batching: Database writes are batched (typically in sizes of 100 to 1000) to optimize performance and adhere to Prisma Accelerate limits.
Sources: scripts/ingest-market-prices.ts:167-175, scripts/ingest-mes-live-15m.ts:32-35, cubic.yaml:81-91
Orchestration and Monitoring
All ingestion attempts are logged in the ingestionRun table, tracking the job status, rows processed, rows inserted, and any failure details. This provides a clear audit trail for data provenance.
The diagram above shows the standard lifecycle of an ingestion execution.
Sources: scripts/ingest-macro-indicators.ts:117-124, scripts/ingest-alt-news-feeds.ts:275-285
Market Price Ingestion
The project distinguishes between primary trading instruments (MES Futures) and secondary market markers.
MES Live and Backfill
The system supports 15-minute, 1-hour, and 1-day granularities for MES futures. Live ingestion occurs in a polling loop (defaulting to every 45 seconds) to keep the active trading bar updated.
- Provider: Databento (GLBX.MDP3 dataset).
- Strategy: Upsert recent candles to "repair" partial bars in real-time.
| Script | Purpose | Timeframe |
|---|---|---|
ingest-mes-live-15m.ts | Streaming live updates | 15m |
backfill-mes.ts | Historical recovery (2019+) | 15m, 1h, 1d |
ingest-market-prices.ts | Daily canonical sync | 1h (MES), 1d (Others) |
Sources: scripts/ingest-mes-live-15m.ts:7-15, src/inngest/backfill-mes.ts:25-33, scripts/ingest-market-prices.ts:9-16
Macroeconomic & News Ingestion
Macroeconomic data is primarily sourced from FRED, while news is gathered via institutional RSS feeds and web scraping.
FRED Complete Sync
The ingest-fred-complete.ts script manages 47 keep-series spanning 9 domains. It supports a full fresh pull (defaulting to 2 years) or incremental updates.
| Domain | Examples | Frequency |
|---|---|---|
| RATES | DFF (Fed Funds), SOFR | Daily |
| YIELDS | DGS2, DGS10, DGS30 | Daily |
| VOL_INDICES | VIXCLS, VVIX, HY Spread | Daily |
| INFLATION | CPI, PCE, TIPS | Monthly/Daily |
| FX | DXY, USD/EUR, JPY/USD | Daily |
Sources: scripts/ingest-fred-complete.ts:38-115
Institutional News & Policy
The ingest-alt-news-feeds.ts script monitors RSS feeds from the Federal Reserve, SEC, ECB, BEA, and EIA.
- Categorization: News is automatically categorized into types such as CPI, GDP, EMPLOYMENT, or POLICY based on headline keywords.
- Source Tracking: Every item tracks its origin URL and publisher.
Visualizes the institutional news acquisition flow.
Sources: scripts/ingest-alt-news-feeds.ts:47-110, scripts/ingest-alt-news-feeds.ts:223-235
CLI Usage and Configuration
The following commands are defined in package.json for manual data management:
| Command | Script Target | Description |
|---|---|---|
npm run ingest:market | ingest-market-prices.ts | Full sync of canonical market symbols |
npm run ingest:macro | ingest-macro-indicators.ts | Daily refresh of FRED indicators |
npm run ingest:alt-news | ingest-alt-news-feeds.ts | Sync institutional news feeds |
npm run import:fusion:econ | import-fusion-econ.ts | Bulk import from legacy V15 database |
npm run ingest:mes:live | ingest-mes-live-15m.ts | Start live 15m MES poller |
Sources: package.json:20-31
Conclusion
The ingestion system ensures the Rabid Raccoon platform maintains a high-fidelity data environment. By combining automated Inngest jobs for daily updates with robust CLI scripts for historical backfilling and live polling for intraday trading, the system creates a resilient pipeline for technical and macroeconomic analysis. Accuracy is maintained through strict validation policies and cryptographic deduplication, preventing data corruption across the trading signal chain.