Redis Integration

Relevant source files

The following files were used as context for generating this wiki page:

• packages/db/redis.ts
• app/workers/eventWorker.ts
• app/analytics-worker/analyticsWorker.ts
• PROJECT_OUTLINE.md
• docker/docker-compose.yml
• package.json

Introduction

Redis serves as a critical infrastructure component in the Glimpse project, functioning as both a high-performance key-value store and the backbone for asynchronous task processing. It is primarily utilized for managing real-time session state, tracking active user counts, and powering the BullMQ job queue system that decouples event ingestion from heavy processing tasks.

The integration ensures that the system can handle high-velocity analytics events by offloading tasks like geolocation lookups and database persistence to background workers. By utilizing Redis TTL (Time-To-Live) features, Glimpse automatically manages memory bloat for temporary session data.

Sources: PROJECT_OUTLINE.md:12-25, app/workers/eventWorker.ts:88-105

Architecture and Infrastructure

The Redis infrastructure is configured to run as a distinct service within the containerized environment. It is exposed to the internal network for various application services to consume.

Connection Configuration

The system uses the ioredis library to establish connections. Connection parameters are primarily driven by environment variables, with sensible defaults for local development.

Parameter	Environment Variable	Default Value	Description
Host	REDIS_HOST	redis	The hostname of the Redis server.
Port	REDIS_PORT	6379	The port number for the Redis server.

Sources: packages/db/redis.ts:3-6, docker/docker-compose.yml:68-71, app/workers/eventWorker.ts:16-18

Redis Role in Component Communication

The following diagram illustrates how Redis facilitates communication between the API, Event Workers, and Analytics Workers.

The diagram shows the dual-use of Redis as both a job broker (using BullMQ) and a data store for session management.
Sources: app/workers/eventWorker.ts:114-117, app/analytics-worker/analyticsWorker.ts:18-20, PROJECT_OUTLINE.md:144-170

Job Queue Management (BullMQ)

Glimpse leverages BullMQ for reliable, distributed job processing. Redis acts as the messaging broker for two primary queues:

1. eventQueue: Handles the ingest-event jobs. These jobs contain raw event data from the API, including metadata like IP addresses and User Agents.
2. analyticsQueue: Handles analytics-job tasks. Once the primary Event Worker has enriched the data (e.g., added geolocation), it pushes a secondary job to this queue for further downstream processing.

Sources: app/workers/eventWorker.ts:25-30, app/analytics-worker/analyticsWorker.ts:18-20, package.json:37

Real-time Data Storage

Beyond queuing, Redis is used as a fast-access layer for real-time analytics data.

Session Management

The system stores session information in Redis hashes. This allows for rapid updates to session metadata as new events arrive.

• Key Format: session:{projectId}:{sessionId}
• Data Stored: Country, region, city, timezone, latitude, longitude, and the last event timestamp.
• Expiration: Keys are set with a 300-second (5-minute) TTL, ensuring that inactive sessions are automatically purged.

Active User Tracking

To provide "Live" user metrics, Redis sets are used to track unique session IDs per project.

• Key Format: active_sessions:{projectId}
• Operations: Uses SADD to add session IDs and SCARD (planned/commented) to count active users.
• Expiration: The active session set also expires after 300 seconds of inactivity.

Sources: app/workers/eventWorker.ts:88-105, PROJECT_OUTLINE.md:160-165

Implementation Details

The Event Worker utilizes Redis pipelines to perform multiple operations in a single round-trip, optimizing performance when updating session states and active user lists.

// Example of Redis Pipeline usage in eventWorker.ts
const sessionKey = `session:${eventData.projectId}:${eventData.sessionId}`;
const pipeline = redis.pipeline();

pipeline.hset(sessionKey, {
    country: geoData?.country ?? "unknown",
    // ... other geo fields
    lastEventTimestamp: String(eventData.timestamp ?? Date.now()),
});

pipeline.expire(sessionKey, 300);

const activeSessionsKey = `active_sessions:${eventData.projectId}`;
pipeline.sadd(activeSessionsKey, eventData.sessionId);
pipeline.expire(activeSessionsKey, 300);

await pipeline.exec();

Sources: app/workers/eventWorker.ts:88-105

Conclusion

Redis Integration is the cornerstone of Glimpse's real-time capabilities. By combining the strengths of BullMQ for asynchronous job reliability and Redis's native data structures for session tracking, the architecture achieves a high level of responsiveness and scalability required for real-time user analytics.