Redis & Message Queue
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
In the Poneglyph architecture, Redis and Message Queuing systems facilitate asynchronous processing, state management, and real-time data synchronization. These components are primarily used to handle heavy background tasks such as dataset uploads, external data synchronization (CKAN), and AI-driven research processes without blocking the main application server.
The system leverages Google Cloud Pub/Sub for messaging and Upstash Redis for caching and session-related data storage. This decoupled architecture allows the server application to offload intensive "sync" and "upload" operations to dedicated worker services.
Sources: terraform/variables.tf:101-118, apps/server/package.json:20-33
Architecture Overview
The Poneglyph infrastructure utilizes a producer-consumer model for handling data ingestion and synchronization.
Messaging Components
- Google Cloud Pub/Sub: Acts as the primary message broker for upload jobs.
- Push Subscriptions: Used to trigger the upload worker via HTTP callbacks.
- Upstash Redis: Provides a serverless Redis interface for the Hono-based backend.
Data Flow Diagram
The following diagram illustrates how a user request triggers a background synchronization task via the message queue.
The diagram shows the asynchronous flow from the client to the background worker. Sources: terraform/variables.tf:46-60, apps/workers/sync-worker/src/main.rs:50-80
Redis Configuration
The server environment utilizes the @upstash/redis SDK to interact with Redis. It is primarily configured via environment variables to provide global state management for the Hono server.
| Variable | Description | Source |
|---|---|---|
upstash_redis_rest_url | REST URL for the Redis instance | terraform/variables.tf:112 |
upstash_redis_rest_token | Sensitive REST token for authentication | terraform/variables.tf:116 |
Redis is included in the apps/server dependencies to support lightweight data persistence and potential rate limiting or session storage.
Sources: apps/server/package.json:30, terraform/variables.tf:112-117
Message Queue Implementation
The project implements a specific queuing mechanism for dataset uploads and external data synchronization.
Pub/Sub Integration
The infrastructure defines a specific topic and subscription for upload jobs. This allows the system to scale the workers independently from the main API server.
- Topic:
poneglyph-upload - Subscription:
poneglyph-upload-worker-push - Max Scale: The worker service is configured with a
worker_max_scaleof 1, ensuring controlled processing of queued tasks.
Sources: terraform/variables.tf:46-55, terraform/variables.tf:65-74
Sync Worker Logic (Rust)
The sync-worker is a specialized service written in Rust that consumes synchronization requests. It provides an HTTP interface (compatible with Cloud Run or AWS Lambda) to receive triggers from the message queue.
| Endpoint | Method | Description |
|---|---|---|
/sync | POST | Triggers a new synchronization task based on the SyncRequest body. |
/health | GET/POST | Returns the health status of the worker. |
Sources: apps/workers/sync-worker/src/main.rs:71-100
Sequence of a Sync Task
The following sequence diagram represents the interaction between the worker and the database during a sync event triggered by the queue.
Sequence of events within the sync worker during a message processing cycle. Sources: apps/workers/sync-worker/src/main.rs:73-88, packages/db/src/migrations/meta/0005_snapshot.json:425-470
Database Logging for Queued Tasks
To track the progress of messages processed by the workers, the system maintains a sync_logs table in PostgreSQL. This table records the outcome of each background job initiated by the message queue.
| Field | Type | Description |
|---|---|---|
id | uuid | Unique identifier for the log entry |
sync_status | enum | Status: running, completed, or failed |
total_found | integer | Total items found in the external source |
error_count | integer | Number of errors encountered during the task |
started_at | timestamp | When the worker started the task |
Sources: packages/db/src/migrations/meta/0005_snapshot.json:425-475
Conclusion
The Redis and Message Queue systems in Poneglyph provide a robust framework for handling humanitarian data at scale. By utilizing Upstash Redis for fast state access and GCP Pub/Sub for reliable task distribution, the architecture ensures that long-running operations like CKAN synchronization and dataset processing remain isolated from user-facing API performance.