Databento Market Data

Relevant source files

The following files were used as context for generating this wiki page:

• src/lib/databento.ts
• scripts/ingest-market-prices-daily.ts
• scripts/backfill-mes-1h-1d.ts
• scripts/backfill-mes-15m.ts
• scripts/backfill-futures-all.ts
• cubic.yaml
• README.md

Databento serves as the primary historical and real-time market data provider for the Rabid Raccoon platform, specifically focusing on CME Globex (GLBX.MDP3) futures. The system integrates Databento's API to ingest Open-High-Low-Close-Volume (OHLCV) candles for the Micro E-mini S&P 500 (MES) and other major futures contracts like NQ, ES, and CL. This data feeds the platform's technical analysis engines, AI forecasting models, and live charting interfaces.

The integration is designed with a "zero-fake" policy, ensuring that all market data is sourced directly from Databento without synthetic fillers. It employs rigorous validation rules to ensure data integrity, such as checking that price relationships (High >= Low, High >= Open, etc.) are maintained and using deterministic hashing for deduplication.

Sources: README.md:92-105, cubic.yaml:73-83

System Architecture and Data Flow

The Databento integration consists of a core library for API communication and a suite of specialized scripts for initial backfilling and daily incremental updates. Data is retrieved via HTTP POST requests to Databento's timeseries endpoints and stored in a local PostgreSQL database using Prisma ORM.

Ingestion Workflow

The following diagram illustrates the flow from Databento's API to the persistent storage layer:

This diagram shows the path of market data from the external provider through validation logic and into the database.
Sources: scripts/ingest-market-prices-daily.ts:133-170, cubic.yaml:85-115

Core Data Structures and Validation

Market data is strictly validated before insertion to prevent "silent data corruption," which could lead to incorrect trading signals.

OHLCV Validation Rules

Every candle must satisfy the following conditions:

• Open, High, Low, and Close must be greater than zero.
• High must be greater than or equal to Low.
• High must be greater than or equal to both Open and Close.
• Low must be less than or equal to both Open and Close.
• Volume must be handled as a BigInt to prevent overflow in aggregate calculations.

Sources: cubic.yaml:85-108, scripts/backfill-mes-15m.ts:46-49

Row Deduplication

To handle Inngest retries and overlapping backfills, each row is assigned a rowHash computed from a deterministic key consisting of the symbol, timestamp, and close price. Prisma's createMany with skipDuplicates: true is then used to ensure idempotency.

Sources: cubic.yaml:109-115, scripts/ingest-market-prices-daily.ts:31-35

Ingestion Components

Daily Updates

The ingest-market-prices-daily.ts script handles incremental updates. It calculates an overlap window (usually 72 hours) to ensure no gaps are left due to late-arriving data or processing delays.

Feature	Configuration	Source File
Default Lookback	72 Hours	scripts/ingest-market-prices-daily.ts:11
Batch Size	1000 rows	scripts/ingest-market-prices-daily.ts:12
MES Dataset	GLBX.MDP3	scripts/ingest-market-prices-daily.ts:167
MES Schema	ohlcv-1m (aggregated to 1h)	scripts/ingest-market-prices-daily.ts:13

Backfilling Modules

Backfill scripts are used for initial historical data population. These are chunked into monthly or 3-month windows to avoid Databento API timeouts.

• MES Specific: backfill-mes-15m.ts and backfill-mes-1h-1d.ts handle the primary trading instrument with higher granularity.
• Futures All: backfill-futures-all.ts pulls daily and hourly data for 11 secondary symbols including NQ, ES, CL, and GC.

Sources: scripts/backfill-futures-all.ts:25-40, scripts/backfill-mes-1h-1d.ts:60-70

Databento API Integration

The system interacts with Databento's /timeseries.get_range endpoint. Due to the volume of data in futures contracts, the platform uses specific scaling and schema parameters.

Technical Configuration

// Example of the scale and timeout constraints used in Databento calls
const FIXED_PRICE_SCALE = 1_000_000_000;
const REQUEST_TIMEOUT_MS = 120_000;

// Source: scripts/backfill-futures-all.ts:21-23

The system converts Databento's integer-scaled prices back to standard decimals by dividing by 1e9. Timestamps, provided by Databento in nanoseconds since the Unix epoch, are converted to standard JavaScript Date objects.

Parameter	Value	Description
Dataset	GLBX.MDP3	CME Globex MDP 3.0 Real-time Feed
Stype In	continuous	Continuous futures contract representation
Encoding	json	Data format for easy parsing in Node.js
Symbols	[SYMBOL].c.0	Target for the front-month continuous contract

Sources: scripts/backfill-futures-all.ts:21-23, scripts/backfill-mes-1h-1d.ts:38-48, src/lib/databento.ts:1-20

Reliability and Operational Constraints

As per the cubic.yaml surgical rules, the following operational constraints are strictly enforced:

1. Explicit Timeouts: Every Databento API call must include a timeout (typically ≤ 120,000ms) to prevent Inngest functions from hanging and wasting Vercel compute credits.
2. UTC Standard: All timestamps must be preserved in UTC. Local timezone conversion is prohibited as it breaks time-series alignment for charting.
3. Connection Management: Database connections used during ingestion must be released in finally blocks to prevent pool exhaustion during concurrent ingestion runs.

Sources: cubic.yaml:126-133, 190-195, 203-207

Conclusion

The Databento Market Data system provides the high-fidelity foundation required for the Rabid Raccoon trading platform. By combining strict architectural isolation, rigorous data validation, and automated daily ingestion, the system ensures that traders and AI models operate on accurate, deduplicated, and timely futures data.