Background Workers & Queues

Relevant source files

The following files were used as context for generating this wiki page:

• services/api/src/worker.ts
• services/api/src/queue/emailQueue.ts
• services/api/src/retention.ts
• services/api/src/config/redis.ts
• blueprint.md
• services/api/package.json

The email platform leverages a background job processing system to handle asynchronous, long-running, or periodic tasks. This ensures that the main API remains responsive and that tasks like email ingestion and data cleanup are handled reliably and efficiently. The system is built using BullMQ, a robust queue system for Node.js that uses Redis as its backend.

This architecture is crucial for decoupling the immediate reception of an email by the SMTP server from the more intensive processing, validation, and storage steps. It also enables scheduled maintenance tasks, such as enforcing data retention policies, to run without direct user intervention.

Sources: blueprint.md, services/api/package.json:30-30

Architecture Overview

The background processing system consists of three main parts: job producers, the message queue (Redis), and job consumers (workers). When a task needs to be performed asynchronously, a "job" is created and placed onto a specific queue. Workers listen to these queues, pick up jobs, and execute the associated logic.

The following diagram illustrates the high-level data flow for inbound email processing, which is the primary use case for this system.

This diagram shows an external email being received by the SMTP server, which triggers the API to enqueue a processing job. A dedicated worker then processes this job, interacting with the database and file storage.

Sources: blueprint.md, services/api/src/worker.ts

Email Processing Worker

The core of the background system is the email processing worker, responsible for handling all inbound emails after they are received by the SMTP server.

Sources: services/api/src/worker.ts

Queue Definition

The email processing worker listens to a dedicated queue named email-queue.

Property	Value	Source File
Queue Name	email-queue	services/api/src/queue/emailQueue.ts:3-3
Job Data	EmailJobData	services/api/src/worker.ts:140-146

The EmailJobData interface defines the payload for each job:

interface EmailJobData {
    rawPath: string;
    envelope: {
        rcptTo: { address: string }[];
        remoteAddress?: string;
    };
}

Sources: services/api/src/worker.ts:140-146

Processing Workflow

When the worker picks up a job from the email-queue, it executes a series of steps to validate, process, and store the email. If any step fails, the job fails and an error is logged.

The sequence diagram below details the internal logic of the email worker.

Key Functions

Several helper functions support the main worker logic:

Function	Description	Source File
ensureDomainAndInbox	Finds or creates the necessary Domain and Inbox records for a recipient email address. Can auto-create domains if allowAutoDomainCreation is true.	services/api/src/worker.ts:28-54
persistAttachments	Validates attachments against size and MIME type allowlists, scans for viruses, saves them to storage, and creates Attachment records in the database.	services/api/src/worker.ts:74-124
enforceRateLimits	Checks if the number of messages from a source IP, to a domain, or to an inbox has exceeded configured limits within a time window.	services/api/src/worker.ts:129-138
checkSpam	Sends the raw email content to an external service (like Rspamd) to get a spam score.	services/api/src/worker.ts:210-216
scanBuffer	Scans a buffer (e.g., an attachment) for viruses.	services/api/src/worker.ts:86-90

Data Retention Worker

A separate worker and queue are responsible for enforcing data retention policies, automatically cleaning up old or expired data.

Sources: services/api/src/retention.ts

Queue and Job Definition

This worker uses the retention-queue and processes jobs of type cleanup. The job is configured to run on a repeatable cron schedule.

Property	Value	Source File
Queue Name	retention-queue	services/api/src/retention.ts:10-10
Job Type	cleanup	services/api/src/retention.ts:11-11
Cron Schedule	RETENTION_SWEEP_CRON env var	services/api/src/retention.ts:60-60

Cleanup Workflow

The retention worker performs a soft-delete on records that have passed their retention period or expiration date.

This flow shows the periodic, automated cleanup process managed by the retention worker.

Sources: services/api/src/retention.ts:23-56

Configuration

The behavior of the queues and workers is controlled by environment variables. The Redis connection, which powers the queues, is configured in a dedicated file.

Redis Connection

The connection to the Redis server is established using the following settings.

Variable	Description	Default	Source File
REDIS_HOST	The hostname of the Redis server.	localhost	services/api/src/config/redis.ts:6-6
REDIS_PORT	The port of the Redis server.	6379	services/api/src/config/redis.ts:7-7
REDIS_PASSWORD	The password for the Redis server.	undefined	services/api/src/config/redis.ts:8-8

Sources: services/api/src/config/redis.ts

Worker Configuration

These variables control the logic within the workers themselves.

Variable	Description	Source File
allowAutoDomainCreation	If true, the email worker will create a Domain record if one doesn't exist for a recipient.	services/api/src/worker.ts:34-36
maxAttachmentBytes	The maximum allowed size for an email attachment in bytes.	services/api/src/worker.ts:98-100
allowedAttachmentMimePrefixes	A list of allowed MIME type prefixes for attachments (e.g., image/).	services/api/src/worker.ts:102-104
allowedAttachmentExtensions	A list of allowed file extensions for attachments.	services/api/src/worker.ts:103-105
smtpRateLimit.perIp	Max messages per IP in the time window.	services/api/src/worker.ts:135-137
quotaMessagesPerInbox	Maximum number of messages allowed in a single inbox.	services/api/src/worker.ts:192-194
RETENTION_SWEEP_CRON	The cron expression for scheduling the retention job.	services/api/src/retention.ts:60-60
RETENTION_MESSAGES_DAYS	The number of days after which messages are soft-deleted.	services/api/src/retention.ts:25-25

Summary

The background worker and queue system is a fundamental component of the email platform's architecture. By using BullMQ and Redis, it provides a scalable and resilient way to process inbound emails and perform regular system maintenance. This design offloads heavy processing from the synchronous API flow, leading to a more robust and performant application.

Sources: blueprint.md, services/api/src/worker.ts, services/api/src/retention.ts