Background Workers & Job Queues
Relevant source files
The following files were used as context for generating this wiki page:
Background workers handle asynchronous tasks that are too resource-intensive or time-consuming for the main API thread. This system ensures the platform remains responsive by offloading operations such as message delivery, PDF generation, and third-party integrations to a dedicated service.
The implementation relies on BullMQ and Redis to manage job distribution and execution. Workers are organized as a standalone application within the monorepo, isolated from the core API logic.
Sources: README.md:20, apps/workers/package.json:1-15
System Architecture
The worker system follows a producer-consumer pattern. The core API acts as the producer, pushing jobs into specific Redis-backed queues. The apps/workers package contains the consumers (workers) that process these jobs.
The diagram shows the flow of tasks from the API to dedicated workers via a Redis message broker.
Sources: README.md:50-60, apps/workers/src/workers.test.ts:7-15
Queue Definitions and Processors
The system categorizes work into distinct queues to allow for independent scaling and monitoring. Each queue is mapped to a specific processor function that executes the business logic for that task type.
Supported Queues
The workers handle the following domains:
- Messages: Handling notifications or communications.
- PDFs: Generating clinical documents or prescriptions.
- ABDM: Integration with the Ayushman Bharat Digital Mission.
- Imports/Reconciliation: Processing bulk data uploads and financial syncing.
- Webhooks: Delivering events to external systems.
- System: Internal health checks and maintenance tasks.
Sources: apps/workers/package.json:4, apps/workers/src/workers.test.ts:18-24
System Processor
A specialized system processor exists to handle internal utility jobs. For example, it responds to ping jobs to verify worker availability.
// Example from system processor tests
await expect(processors.system!({ name: 'ping' } as Job)).resolves.toMatchObject({
pong: true,
});
Sources: apps/workers/src/workers.test.ts:25-30
Configuration
Workers are configured using environment variables. The system enforces strict validation on configuration values to prevent runtime failures.
| Variable | Description | Default / Requirement |
|---|---|---|
REDIS_URL | Connection string for the Redis instance. | redis://localhost:6379 |
WORKER_CONCURRENCY | Number of jobs a single worker can process in parallel. | Must be > 0 |
Sources: apps/workers/src/workers.test.ts:32-38
Operational Lifecycle
The lifecycle of a job involves several states: waiting, active, completed, or failed. Integration tests ensure that queue names remain unique and that all registered processors correspond to known queues.
The state machine represents the standard BullMQ job lifecycle managed within the service.
Sources: apps/workers/src/workers.test.ts:8-16, apps/e2e/tests/queue.spec.ts:1-20
Development and Testing
The worker service is built using tsup and run via tsx during development. It uses vitest for unit testing the logic within processors.
- Launch Command:
pnpm --filter @dhc/workers dev - Test Command:
pnpm test(withinapps/workers) - Redis Requirement: Redis 7 must be running (usually via
pnpm db:up).
Sources: apps/workers/package.json:7-13, README.md:55-58, package.json:20
Summary
The Background Worker system provides a scalable foundation for handling non-blocking tasks in the Digital Health Care platform. By using BullMQ and Redis, the system ensures high reliability for critical operations like PDF generation and ABDM integration while maintaining API performance. All workers are strictly typed and verified through a suite of unit and integration tests to ensure queue integrity and configuration validity.
Sources: README.md:20, apps/workers/src/workers.test.ts:5-10