Data Flow: From Event to Visualization

Relevant source files

The following files were used as context for generating this wiki page:

• packages/analytics-web/core/event.js
• app/api/controller/event.controller.ts
• app/workers/eventWorker.ts
• packages/analytics-web/transport/sender.js
• packages/analytics-web/events/auto.js
• packages/shared/zod.schema.ts
• packages/db/prisma/schema.prisma
• PROJECT_OUTLINE.md

Introduction

The Glimpse data pipeline is designed for real-time ingestion, processing, and storage of user interaction data. It prioritizes session-based tracking over individual identity, capturing a wide array of automatic and manual events from the client-side SDK and funneling them through a distributed backend architecture.

The system utilizes a multi-stage pipeline: events are captured by the Web SDK, transmitted via an asynchronous transport layer to a Hono-based API, enqueued into BullMQ for background processing, enriched with geolocation data, and finally persisted to a PostgreSQL database via batching. This architecture ensures high throughput and low latency for real-time visualization of global traffic patterns.

Sources: PROJECT_OUTLINE.md:5-10, packages/analytics-web/core/event.js:4-43

Architecture Overview

The data flow follows a decoupled producer-consumer pattern utilizing Redis as an intermediary broker.

This diagram illustrates the lifecycle of an event from the client browser to the persistent database.
Sources: PROJECT_OUTLINE.md:144-177, app/workers/eventWorker.ts:70-155

Event Capture and Transport

The Web SDK is responsible for gathering environmental context and user actions. It supports automatic tracking for page views, scroll depth, element visibility, and errors.

Event Structure

Events are structured using a standardized schema that includes project identifiers, timestamps, and deep context regarding the user's environment.

Field	Type	Description
projectId	String	Unique identifier for the tracked site.
event	String	Name of the action (e.g., "Page View").
timestamp	Number	Unix timestamp of the event.
sessionId	String	Unique session identifier.
context	Object	Detailed data including URL, Viewport, and Screen resolution.

Sources: packages/shared/zod.schema.ts:3-35, packages/analytics-web/core/event.js:6-38

Transmission Logic

To optimize performance, the SDK employs a buffering strategy. Events are not sent immediately but are queued in an array and flushed either when the buffer size reaches 10 events or after a 5000ms timeout.

function enqueueEvent(event) {
    eventBuffer.push(event);
    if (eventBuffer.length >= BUFFER_SIZE) {
        flushBuffer();
    } else {
        // ... timer logic
        flushTimer = setTimeout(() => flushBuffer(), FLUSH_INTERVAL);
    }
}

Sources: packages/analytics-web/transport/sender.js:3-10, packages/analytics-web/transport/sender.js:45-56

Ingestion and Processing

The backend API receives events, validates them against Zod schemas, and prepares them for asynchronous processing.

API Ingestion

The IngestionEventController extracts the client's IP address and User-Agent from request headers, attaching them to the event as metadata before enqueuing the job into BullMQ.

Worker Processing and Enrichment

The eventWorker consumes jobs from the eventQueue. It performs three critical operations:

1. Session Management: Stores session data in Redis with a 300-second TTL to track "active" users.
2. Geolocation: Resolves the event's IP address into geographic coordinates (lat/lon), city, and country.
3. Buffering: Adds the enriched event to an in-memory buffer for batch database insertion.

Sources: app/workers/eventWorker.ts:80-125, PROJECT_OUTLINE.md:85-103

Data Persistence

Glimpse uses batching to reduce the overhead of database writes. Instead of inserting every event individually, the worker flushes an internal buffer to PostgreSQL every 1000ms or when the buffer hits 100 items.

Database Schema

Events are stored in the events table, which includes fields for both user-provided data and system-generated enrichment (GeoIP).

Column	Type	Description
id	BigInt	Primary key (Auto-increment).
projectId	String	Scopes the event to a specific project.
occurredAt	DateTime	When the event happened on the client.
properties	Json	Custom event metadata.
lat / lon	Float	Geographic coordinates for mapping.

Sources: packages/db/prisma/schema.prisma:18-45, app/workers/eventWorker.ts:25-55

Conclusion

The flow from event to visualization is a robust pipeline that balances real-time requirements with database efficiency. By leveraging client-side buffering, a queue-based backend, and GeoIP enrichment, Glimpse provides the necessary data for live mapping while maintaining session-based privacy and system performance.