Background Workers

Relevant source files

The following files were used as context for generating this wiki page:

• app/workers/eventWorker.ts
• app/analytics-worker/analyticsWorker.ts
• workers.ts
• PROJECT_OUTLINE.md
• packages/db/prisma/schema.prisma
• docker/docker-compose.yml

Introduction

The Background Workers system in Glimpse is responsible for asynchronous event processing, geolocation enrichment, and persistent storage of analytics data. By offloading these tasks from the main API, the system ensures high-performance event ingestion and real-time responsiveness. The architecture relies on BullMQ for job orchestration, backed by Redis for queue management and session state.

Sources: PROJECT_OUTLINE.md:111-133, app/workers/eventWorker.ts:1-10

The worker ecosystem is divided into specialized services: the Event Worker, which handles the initial ingestion and enrichment of raw events, and the Analytics Worker, designed for downstream processing of analytics data. This separation allows for independent scaling and failure isolation within the data pipeline.

Sources: app/workers/eventWorker.ts, app/analytics-worker/analyticsWorker.ts, docker/docker-compose.yml:29-67

Architecture and Data Flow

The worker architecture follows a producer-consumer pattern where the API server enqueues jobs into Redis, and the background workers process them concurrently.

Event Processing Pipeline

When an event is received, it undergoes a multi-stage enrichment and storage process:

1. Identification: Missing sessionId or anonymousId are generated using UUID v7.
2. Geolocation Enrichment: The worker performs an IP-based lookup to determine country, region, city, and coordinates.
3. Session Management: Session metadata and active status are updated in Redis with a 300-second TTL.
4. Buffering: Events are added to an in-memory buffer before being flushed to PostgreSQL in batches.
5. Downstream Queuing: Enriched events are forwarded to the analyticsQueue for further processing.

Sources: app/workers/eventWorker.ts:88-144, PROJECT_OUTLINE.md:135-163

Workflow Diagram

The following diagram illustrates the flow of an event through the Background Worker system.

The diagram shows the transition from raw job ingestion to database persistence and downstream analytics processing.
Sources: app/workers/eventWorker.ts:35-144, app/analytics-worker/analyticsWorker.ts:15-38

Core Components

Event Worker

The eventWorker is the primary entry point for raw event data. It processes jobs from the eventQueue with a default concurrency of 10. It utilizes a batching strategy to minimize database write operations, flushing a MAX_BUFFER_SIZE of 100 events or at a BUFFER_FLUSH_INTERVAL of 1000ms.

Sources: app/workers/eventWorker.ts:24-30, app/workers/eventWorker.ts:152-162

Analytics Worker

The analyticsWorker consumes from the analyticsQueue. While currently providing a framework for downstream processing, its role is to handle complex analytics aggregations and data pipeline tasks once the eventWorker has enriched the primary data.

Sources: app/analytics-worker/analyticsWorker.ts:15-38

Key Technical Specifications

Component	Technology	Configuration / Defaults
Job Queue	BullMQ	Redis-backed
Concurrency	eventWorker	10
Concurrency	analyticsWorker	5
Session TTL	Redis	300 seconds
Buffer Limit	Memory	100 events
Flush Timer	Interval	1000ms

Sources: app/workers/eventWorker.ts:25-26, app/workers/eventWorker.ts:130, app/workers/eventWorker.ts:161, app/analytics-worker/analyticsWorker.ts:40

Persistence and Storage

The workers interact with two primary data stores: Redis for transient state and PostgreSQL (via Prisma) for long-term storage.

Redis State Management

Workers use Redis pipelines to efficiently update session data and track active users per project.

• Session Keys: session:{projectId}:{sessionId} (Hash) stores geographic data and the last event timestamp.
• Active User Tracking: active_sessions:{projectId} (Set) tracks unique active session IDs.

Sources: app/workers/eventWorker.ts:117-133

Database Schema (Prisma)

The workers persist enriched events to the events table. Key fields populated during processing include:

model Event {
  id           BigInt   @id @default(autoincrement())
  projectId    String
  name         String
  occurredAt   DateTime
  sessionId    String
  ip           String?
  country      String?
  lat          Float?
  lon          Float?
  // ... traits, properties, context
}

Sources: packages/db/prisma/schema.prisma:20-50, app/workers/eventWorker.ts:48-73

Runtime Configuration

Background workers are managed as separate services within the container orchestration.

Variable	Description	Default (Docker)
REDIS_HOST	Hostname for Redis instance	redis
REDIS_PORT	Port for Redis instance	6379
DATABASE_URL	PostgreSQL connection string	postgresql://...

Sources: docker/docker-compose.yml:43-51, app/workers/eventWorker.ts:14-16

Summary

The Background Workers system provides the backbone for Glimpse's real-time analytics. By utilizing job queues, in-memory buffering, and GeoIP enrichment, it transforms raw HTTP events into structured, geographically-aware data while maintaining low latency for the client-facing API. The system ensures data integrity through graceful shutdowns and periodic buffer flushing.

Sources: workers.ts:5-18, app/workers/eventWorker.ts:35-80