Background Send Job Processing

Relevant source files

The following files were used as context for generating this wiki page:

• src/lib/send-job-processor.ts
• src/app/api/send/route.ts
• src/app/api/send/status/route.ts
• src/lib/db.ts
• README.md

The Background Send Job Processing system is a robust, asynchronous mechanism designed to handle high-volume email campaigns without blocking user interactions or API responses. It leverages a job queue architecture where send requests are persisted to a database and subsequently processed by a dedicated worker or inline execution threads. This ensures reliable delivery, respect for rate limits, and accurate tracking of job progress.

The system is built to handle various execution environments, supporting both a "lazy" inline processing mode triggered by status polls and a "greedy" background processing mode using Next.js after() or external cron jobs. Sources: README.md:104-108, src/app/api/send/route.ts:252-271

System Architecture and Data Flow

The architecture follows a producer-consumer pattern. The API (producer) enqueues jobs and recipient lists into the database, while the Job Processor (consumer) claims these jobs, renders email templates, and transmits them via Amazon SES.

High-Level Workflow

1. Enqueueing: A POST request to /api/send validates the request, resolves the recipient list, and persists a send_job record along with its associated send_job_recipients.
2. Claiming: The processor queries for queued or stale jobs and updates their status to running to prevent duplicate processing.
3. Batching: Recipients are processed in batches (default size: 50) using configurable concurrency (default: 4 parallel sends).
4. Completion: Once all recipients are processed (as sent or failed), the job is marked as completed.

Key Components

1. Job Processor (send-job-processor.ts)

The core logic resides in processSendJobs. It manages the lifecycle of a job, including reclaiming stale jobs that may have crashed (indicated by a lack of heartbeats).

• Claiming Logic: Uses claimQueuedSendJob and claimRunningSendJob to atomically update job states.
• Template Rendering: Uses a regex-based renderTemplate function to inject contact fields (e.g., {{email}}, {{first_name}}) into the HTML body and subject line.
• Rate Limiting: Implements a wait(ms) function between batches to respect the rateLimit configured for the workspace.

Sources: src/lib/send-job-processor.ts:27-38, src/lib/send-job-processor.ts:121-140

2. Database Schema (db.ts)

The system relies on several tables to track state:

Table	Purpose
send_jobs	Stores job metadata, status, and the email payload (HTML, subject).
send_job_recipients	Tracks the delivery status (pending, sending, sent, failed) for every individual recipient.
contact_unsubscribes	Checked during processing to skip recipients who have opted out.

Sources: src/lib/db.ts:1280-1310, src/lib/db.ts:1438-1455

Execution Triggers

The project supports three ways to trigger the processing of send jobs:

Inline Processing (Poll-based)

When a user views a job's status via /api/send/status, the server can optionally process a small batch of recipients inline. This ensures the UI remains updated even if no dedicated worker is running.

• Configuration: SEND_JOB_STATUS_INLINE_PROCESS (Default: true).
• Limit: Defaults to 3 jobs and 50 recipients per poll.

Sources: src/app/api/send/status/route.ts:47-65

Background Processing (after())

In Next.js environments, the after() hook is used immediately after the send request returns a 200 OK to the user. This initiates a loop (default 20 iterations) to drain the queue.
Sources: src/app/api/send/route.ts:252-271

External Worker (Cron)

For production environments, it is recommended to call POST /api/send/process periodically. This endpoint requires a SEND_JOB_PROCESSOR_TOKEN for authentication.
Sources: README.md:104-108

Configuration Parameters

Processing behavior is controlled via environment variables:

Variable	Default	Description
SEND_JOB_BATCH_SIZE	50	Number of recipients claimed in a single DB transaction.
SEND_JOB_CONCURRENCY	4	Parallel SES requests within a single batch.
SEND_JOB_STALE_MS	180,000	Timeout (3 mins) before a 'running' job is considered crashed and reclaimed.
SEND_JOB_MAX_RECIPIENTS_PER_JOB	250	Max recipients processed per worker invocation.

Sources: README.md:135-141, src/lib/send-job-processor.ts:74-89

Error Handling and Recovery

The system includes built-in resilience:

1. Stale Recipient Recovery: If a worker dies while sending, recipients stuck in sending status for longer than SEND_JOB_STALE_RECIPIENT_MS are automatically reverted to pending by the next processor run.
2. Billing Enforcement: If BILLING_ENFORCED is true, the processor checks and consumes credits via the consume_workspace_credits RPC before each individual send.
3. Unsubscribe Filtering: Even if a job was created with a recipient, the processor does a final check against the contact_unsubscribes table right before transmission to prevent illegal sends.

Summary

The Background Send Job Processing module provides a reliable, scalable way to manage email delivery. By separating job creation from execution and implementing heartbeat-based recovery, it ensures that campaigns are delivered according to workspace rate limits while maintaining high availability for the end-user.