Background Jobs & SQS Queues
Relevant source files
The following files were used as context for generating this wiki page:
Background Jobs and SQS Queues manage long-running asynchronous tasks in GoGovSG, specifically for high-volume operations like bulk QR code generation. The system offloads resource-intensive processing from the main web server to AWS Lambda via Amazon Simple Queue Service (SQS) to maintain application responsiveness.
Sources: README.md:143-145, src/server/modules/job/JobController.ts:40-60
Architecture Overview
The system consists of a producer (the Express server), a message broker (AWS SQS), and a consumer (AWS Lambda). The server chunks large requests into smaller batches, creates tracking records in the database, and dispatches messages to SQS. A serverless function then processes these messages independently.
The diagram shows the asynchronous flow from user request to serverless processing and asset storage.
Sources: src/server/modules/job/JobController.ts:45-65, src/server/serverless/bulk-qrcode-generation/index.js:15-50
Job Lifecycle Management
Job and JobItem Creation
When a user initiates a bulk operation, the JobController creates a single Job record associated with the user ID. It then splits the task into batches based on qrCodeJobBatchSize (default 1000). For each batch, a JobItem is created in the database and a corresponding message is sent to SQS.
Sources: src/server/modules/job/JobController.ts:46-60, src/server/config.ts:182
Status Computation
A Job's overall status is derived from its constituent JobItems. The JobManagementService computes the status using the following logic:
- Failure: If at least one JobItem has failed.
- InProgress: If no items have failed but at least one is still in progress.
- Success: If all JobItems have succeeded.
Sources: src/server/modules/job/services/JobManagementService.ts:85-103
Polling and Notifications
Clients track job progress via long-polling. The server waits for status changes for a set number of attempts (jobPollAttempts) at specific intervals (jobPollInterval). Upon completion (Success or Failure), the system sends a completion email to the user.
Sources: src/server/modules/job/services/JobManagementService.ts:135-155, src/server/config.ts:186-187
Serverless Consumer: Bulk QR Generation
The Lambda function processes batches of short URL mappings to generate QR codes in multiple formats.
- Input Validation: Parses the SQS record to extract mappings and
jobItemId. - CSV Generation: Creates a CSV manifest of the mappings and uploads it to S3.
- Image Generation: Generates QR codes in both SVG and PNG formats.
- Archiving: Zips the generated files and streams them to the S3 bucket.
- Completion Callback: Sends an HTTP message back to the main server to update the
JobItemstatus.
Sources: src/server/serverless/bulk-qrcode-generation/index.js:10-63
The sequence diagram illustrates the internal execution steps of the QR generation Lambda.
Sources: src/server/serverless/bulk-qrcode-generation/index.js:15-63, src/server/modules/job/JobController.ts:74-85
Configuration Parameters
The following environment variables control the behavior of the background job system:
| Variable | Description | Default |
|---|---|---|
SQS_BULK_QRCODE_GENERATE_START_URL | The URL of the SQS queue for QR generation. | '' |
SQS_REGION | AWS Region where the SQS queue resides. | ap-southeast-1 |
BULK_QR_CODE_BATCH_SIZE | Max number of QR codes processed per Lambda invocation. | 1000 |
JOB_POLL_ATTEMPTS | Maximum number of long-polling attempts. | 12 |
JOB_POLL_INTERVAL | Milliseconds between each polling attempt. | 5000 |
BULK_QR_CODE_BUCKET_URL | Base URL for downloading generated job assets. | '' |
Sources: src/server/config.ts:133-140, src/server/config.ts:182-187
Error Handling
The system implements multiple layers of error handling:
- Queue Timeouts: Managed via
SQS_TIMEOUTduring message dispatch. - Lambda Cleanup: The Lambda function uses a
try-catch-finallyblock to ensure temporary files in/tmpare deleted even if generation fails. - Status Reporting: If the Lambda fails, it invokes the callback with
isSuccess: falseand the error message to ensure the user is notified via the failure email.
Sources: src/server/serverless/bulk-qrcode-generation/index.js:52-60, src/server/modules/job/services/JobManagementService.ts:170-173, src/server/config.ts:140
Summary
The Background Job system provides a robust framework for handling bulk operations asynchronously. By combining SQS for reliable message delivery and Lambda for scalable compute, GoGovSG ensures that link shortener operations remain performant even during massive bulk creation events. The status tracking and polling mechanisms bridge the gap between asynchronous backend processing and the synchronous requirements of the user interface.