Background Jobs & pg-boss

Relevant source files

The following files were used as context for generating this wiki page:

• packages/jobs/src/types.ts
• packages/jobs/README.md
• README.md
• apps/worker/src/index.ts
• apps/worker/README.md
• apps/web/app/api/webhooks/github/route.ts
• packages/ingestion/src/process-webhook.ts

The background job system in this project is built on pg-boss, a job queue for PostgreSQL. It handles long-running tasks such as LLM-powered code reviews, repository indexing, and webhook processing outside of the main HTTP request/response cycle. This ensures that the web server remains responsive while the system performs heavy computation and external API interactions.

The system architecture follows a producer-consumer model where the web application (apps/web) enqueues tasks into the database, and a dedicated worker process (apps/worker) polls and executes those tasks. Using pg-boss allows for transactional enqueuing, meaning a job is only created if the surrounding database transaction succeeds.

Sources: README.md:54-68, packages/jobs/README.md:1-13

System Architecture

The project splits job handling into two distinct processes that communicate through a shared PostgreSQL database.

• Producer (apps/web): Receives GitHub webhooks and user requests. It validates inputs and writes job rows to the pgboss schema.
• Consumer (apps/worker): A long-running Node.js process that polls the database for new jobs, executes them, and updates their status.

Job Lifecycle Flow

The following diagram illustrates how a GitHub webhook triggers an asynchronous background review process.

The web server acknowledges webhooks within seconds, while the worker manages the 10–30 second LLM processing time.
Sources: README.md:12-42, apps/worker/README.md:8-16

pg-boss Implementation

The system uses pg-boss v10. A critical requirement of this version is that queues must be explicitly created using createQueue() before jobs can be sent. The project handles this during container boot for the web server and during handler registration for the worker.

Key Advantages

Feature	Description
Transactional Enqueue	Jobs can be enqueued within standard SQL transactions.
Single Database	No need for Redis or SQS; the application DB and queue share one backup story.
SQL-Based DLQ	Failed jobs can be inspected using standard SQL queries in the pgboss.archive table.
Serialization	Supports Postgres advisory locks to ensure only one job runs per installationId to respect rate limits.

Sources: packages/jobs/README.md:6-14, packages/jobs/README.md:32-35, README.md:62-68

Job Definitions and Payload Schemas

Jobs are defined using a constant enum JobNames and validated at runtime using Zod schemas. This ensures type safety between the producer enqueuing the job and the consumer processing it.

Core Job Names

Job Name	Type	Description
webhook.process	Infrastructure	Initial routing of incoming GitHub webhooks.
review.pr	Review	Orchestrates the full AI review for a Pull Request.
index.repository	Indexing	Performs Layer 1 codebase indexing (AST parsing/embeddings).
index.file	Indexing	Incremental indexing of a single file (triggered by push).
review.memory.store	Memory	Saves review decisions to the RAG memory layer.
comment.developer.process	Feedback	Processes developer replies to AI comments.

Sources: packages/jobs/src/types.ts:3-22

Payload Example: review.pr

The ReviewPr job requires specific identifiers to fetch the correct code and target the right PR.

[JobNames.ReviewPr]: z.object({
  repositoryId: z.string(),
  installationId: z.string(),
  pullRequestNumber: z.number().int().positive(),
  commitSha: z.string().min(7).max(40),
  aiReviewId: z.string(),
  correlationId: z.string().uuid().optional(),
})

Sources: packages/jobs/src/types.ts:31-38

Worker Configuration

The worker process registers handlers with specific batchSize configurations to control concurrency. It initializes clients for LLM providers (OpenAI/Anthropic) and embedding services before starting the job loop.

Handler Registration

Handlers are defined using defineHandler which wraps the business logic and configuration.

• ReviewPr: batchSize: 4 (Parallelized LLM calls).
• IndexRepository: batchSize: 2 (Heavy resource usage/cloning).
• WebhookProcess: batchSize: 8 (High throughput routing).

Sources: apps/worker/src/index.ts:51-74

Execution Flow: Webhook to Review

1. Ingestion: The Webhook route (/api/webhooks/github) verifies the HMAC signature and upserts a WebhookDelivery record to ensure idempotency. It then enqueues webhook.process.
2. Routing: The webhook.process handler in packages/ingestion parses the payload. For a pull_request event (actions: opened, synchronize, reopened), it creates an AiReview record in PENDING state and enqueues review.pr.
3. Orchestration: The review.pr handler in packages/review marks the review as RUNNING, fetches the diff, filters files, and calls the LLM orchestrator.
4. Completion: Once the LLM generates comments and a summary, the worker posts the review to GitHub and marks the AiReview as COMPLETED.

Sources: apps/web/app/api/webhooks/github/route.ts:80-99, packages/ingestion/src/process-webhook.ts:89-137, packages/review/src/orchestrator.ts:40-66

Summary

The background job system leverages pg-boss and PostgreSQL to provide a robust, transactional queue without additional infrastructure like Redis. By decoupling webhook intake from AI processing, the system maintains high availability and handles complex, multi-stage workflows—including repository indexing and review memory storage—through a type-safe Zod-validated architecture.