Historical Backfilling
Relevant source files
The following files were used as context for generating this wiki page:
Historical backfilling in the Rabid Raccoon project is the process of populating the local PostgreSQL database with extensive historical market and economic data. This data is critical for training AI models (such as AutoGluon), conducting technical analysis, and providing historical context for trading signals. The system supports backfilling for MES (Micro E-mini S&P 500) futures, other major futures contracts, and a wide array of economic indicators from FRED and legacy Fusion databases.
The backfilling architecture utilizes a combination of local TypeScript scripts for bulk initialization and Inngest background jobs for resilient, chunked cloud execution. The project enforces a "zero-fake" policy, ensuring all backfilled data originates from verifiable sources like Databento and FRED, with strict validation for price integrity and deduplication using deterministic row hashes.
Sources: README.md:104-122, cubic.yaml:61-68, src/inngest/backfill-mes.ts:13-22
Data Sources and Integration
The backfill system integrates multiple external APIs and legacy databases to construct a comprehensive historical record.
Market Data (Databento)
Market data for futures is primarily sourced from Databento. The system pulls continuous front-month (.c.0) data, specifically targeting the GLBX.MDP3 dataset. While the MES micro futures launched in May 2019, the backfill processes are often configured to pull from early 2019 or 2020 to ensure full coverage.
Sources: scripts/backfill-futures-all.ts:19-24, src/inngest/backfill-mes.ts:24-30
Economic Data (FRED & Fusion)
Economic indicators are backfilled via two primary paths:
- Direct FRED Ingestion: Pulling 47 key series across domains like Rates, Yields, and Volatility directly from the FRED API.
- Fusion Sync: Bulk importing historical data from a legacy Fusion V15 PostgreSQL database to leverage existing cleaned datasets.
Sources: scripts/ingest-fred-complete.ts:51-118, scripts/import-fusion-econ.ts:472-510
MES Backfill Architecture
The MES backfill is the most granular process, often involving resampling 1-minute (ohlcv-1m) raw data into higher timeframes.
Resampling and Aggregation Logic
The system fetches 1-minute candles and aggregates them into 15-minute, 1-hour, and 1-day intervals. This ensures consistency across different chart granularities used by the trading platform.
The diagram shows the data flow from raw 1m data fetching to multi-timeframe database insertion.
Sources: src/inngest/backfill-mes.ts:153-176, scripts/backfill-mes-15m.ts:101-115
Implementation Details
Backfills are implemented both as local scripts and Inngest functions. The Inngest implementation uses step.run() to process data in 7-day chunks, making the long-running process resumable and resistant to timeouts.
| Feature | Local Script (backfill-mes-15m.ts) | Inngest Function (backfill-mes.ts) |
|---|---|---|
| Chunking | 30-day windows | 7-day windows |
| Concurrency | Single-threaded | Limited to 1 (concurrency: [{ limit: 1 }]) |
| Batch Size | 150 rows | 100 rows (Prisma Accelerate safe) |
| Error Handling | Console logs, continues next chunk | Inngest retries, logs to ingestionRun table |
Sources: src/inngest/backfill-mes.ts:32-35, src/inngest/backfill-mes.ts:145-149, scripts/backfill-mes-15m.ts:24-27
Multi-Futures Backfill
For non-MES active symbols (e.g., ES, NQ, CL, GC), the backfill-futures-all.ts script handles bulk ingestion of daily and hourly data.
Supported Symbols
The system targets major liquid futures contracts:
- Equities: ES, NQ, YM, RTY, SOX
- Rates: ZN, ZB, ZF
- Commodities: CL, GC, SI
Sources: scripts/backfill-futures-all.ts:19
Chunked Execution
To avoid API timeouts, the script generates 3-month windows for each symbol and schema. It uses a Basic authentication header with the Databento API key and handles 422 status codes by adjusting the query end to the available data range.
Sources: scripts/backfill-futures-all.ts:40-76, scripts/backfill-futures-all.ts:114-127
Economic Series Reconcilliation and Sync
The backfill of economic data is managed through specialized scripts that ensure metadata and time-series data are correctly mapped to domain-specific tables.
Domain-Specific Mapping
Rather than a single massive table, economic data is split into specialized tables to improve query performance and data organization.
| Domain | Table Name | Key Series Examples |
|---|---|---|
| Rates | econ_rates_1d | DFF (Fed Funds), SOFR |
| Yields | econ_yields_1d | DGS2, DGS10, DGS30 |
| Volatility | econ_vol_indices_1d | VIXCLS, VXVCLS, BAMLH0A0HYM2 |
| Inflation | econ_inflation_1d | T10YIE, CPIAUCSL, DFII10 |
| FX | econ_fx_1d | DTWEXBGS (DXY), DEXUSEU |
Sources: scripts/ingest-fred-complete.ts:54-118, scripts/import-fusion-econ.ts:203-228
Critical Safeguards
Backfilling processes adhere to strict safety rules defined in the project configuration to prevent data corruption.
- Row Hashing: Every inserted row must have a
rowHashcomputed from a deterministic key (e.g.,symbol + timestamp + close price). This is used withskipDuplicates: trueto prevent double-counting. - Batch Limits: Prisma
createManycalls are capped at 100–150 rows to stay within Prisma Accelerate's hard limits. - Price Validation: Scripts filter for
open/high/low/close > 0to reject bad API data. - UTC Consistency: All timestamps are converted to UTC using helpers like
asUtcDateFromUnixSecondsto prevent timezone-shifted gaps in time series.
Sources: cubic.yaml:81-87, cubic.yaml:104-108, src/inngest/backfill-mes.ts:41-45, scripts/backfill-mes-15m.ts:24
Execution Workflow
The sequence shows the standard validation and deduplication loop used across all backfill scripts.
Sources: scripts/backfill-futures-all.ts:153-176, src/inngest/backfill-mes.ts:184-211
Historical backfilling provides the foundation for the Rabid Raccoon trading engine. By combining local bulk processing for initial setup with resilient cloud-based Inngest jobs for gap filling, the system maintains a high-fidelity dataset across market and macro domains while strictly enforcing data integrity through deduplication and batching.