Live Data Streaming
Relevant source files
The following files were used as context for generating this wiki page:
The Live Data Streaming system in Rabid Raccoon provides real-time Micro E-mini S&P 500 (MES) futures market data to the dashboard. It utilizes a bridge architecture where historical REST APIs are polled to simulate a live stream, as native Node.js SDKs for binary TCP protocols (DBN) are currently unavailable. The system ensures high-frequency data availability for technical analysis, charting, and automated forecasting.
Sources: scripts/ingest-mes-live-stream.ts:5-12, README.md:144-149
Ingestion Architecture
The ingestion process is handled by a background worker that polls the Databento Historical API for 1-minute OHLCV bars. These bars are then aggregated into 15-minute candles and persisted to a local PostgreSQL database using Prisma.
Data Flow Logic
- Polling Loop: The worker runs an infinite loop (unless
--once=trueis passed) with a default interval of 45 seconds. - Aggregation: 1-minute bars are deduplicated, sorted, and then resampled into 15-minute intervals.
- Persistence: Data is inserted using
createManywithskipDuplicates: true. To ensure the active/current candle does not go stale, a "refresh" logic updates the most recent 96 candles. - Logging: Each run is logged to the
ingestionRuntable to monitor pipeline health and staleness.
Sources: scripts/ingest-mes-live-15m.ts:77-124, cubic.yaml:126-134
Data Flow Diagram
The following diagram illustrates the path from the Databento API to the end-user's browser.
API Implementation (SSE)
The system exposes a Server-Sent Events (SSE) endpoint at /api/live/mes15m that facilitates a push-based model for the frontend.
Request Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
backfill | number | 96 | Number of historical 15m candles to return in the initial snapshot (capped at 20-500). |
Sources: src/app/api/live/mes15m/route.ts:46-49
Event Types
- Snapshot: Sent immediately upon connection. Contains the array of historical candles requested via
backfill. - Update: Sent whenever the poller detects a change in existing candles or a new candle is finalized.
- Ping: Sent every 15 seconds if no data has changed to prevent connection timeouts.
- Error: Sent if the database query or the Databento refresh fails.
Sources: src/app/api/live/mes15m/route.ts:28-31, src/app/api/live/mes15m/route.ts:98-130
Frontend Visualization
The LiveMesChart component utilizes the lightweight-charts library to render the stream. It maintains an internal pointsRef to manage the collection of candles and updates the chart in real-time as SSE messages arrive.
Technical Analysis Integration
The chart does not just show price action; it layers streaming data with:
- Measured Moves: Active "AB=CD" patterns derived from forecast data.
- Break-Hook-Go (BHG): Markers indicating structural trade phases.
- Session Stats: Live calculation of session high, low, and percentage change.
Sources: src/components/LiveMesChart.tsx:165-240, src/components/LiveMesChart.tsx:244-275
Visualization Sequence
Critical Safety and Integrity Rules
As a trading platform, the live streaming system adheres to strict data integrity rules defined in the project configuration.
- Deduplication: All inserts must use
rowHashcomputed fromsymbol + timestamp + closeto prevent duplicate candles during retries. - Volume Type: Volume must always be handled as
BigIntto prevent overflow errors. - Validation: OHLCV candles must be validated (e.g.,
high >= low,open > 0) before insertion. - Timezones: All timestamps are strictly UTC.
Sources: cubic.yaml:58-75, cubic.yaml:89-98, scripts/ingest-mes-live-15m.ts:32-35
Summary
The Live Data Streaming module provides the foundation for the Rabid Raccoon real-time experience. By combining a periodic background ingest worker with an SSE-based API and a specialized charting component, the system delivers high-integrity market data while overcoming current limitations in native Node.js streaming SDKs.