Adding New Background Jobs

Relevant source files

The following files were used as context for generating this wiki page:

• packages/jobs/src/types.ts
• apps/worker/src/index.ts
• packages/jobs/README.md
• apps/worker/README.md
• README.md
• apps/web/app/api/webhooks/github/route.ts

The AI Code Reviewer system utilizes a background job architecture to handle long-running tasks such as LLM reviews, repository indexing, and webhook processing. By offloading these tasks from the web server to a dedicated worker process, the system ensures that HTTP responses (especially GitHub webhooks) remain fast and reliable while heavy lifting occurs asynchronously.

The project uses pg-boss, a job queue built on top of PostgreSQL. This choice allows for transactional job enqueues—ensuring that a database update and its corresponding background task are committed together—and simplifies infrastructure by using the primary database as the queue provider.

Sources: README.md:58-69, packages/jobs/README.md:1-12

Job Infrastructure Architecture

The job system is split between producers (usually the web app) and consumers (the worker app). Both processes interact with the pgboss schema within the shared PostgreSQL database.

Core Components

• Producer: An application component (like a webhook route) that creates a job using boss.send().
• Consumer: The apps/worker process that polls the database, picks up jobs, and executes registered handlers.
• pg-boss: The underlying queue engine that manages job persistence, retries, and scheduling.
• Zod Schemas: Used to strictly type and validate job payloads at both the production and consumption stages.

Data Flow for a New Job

The following diagram illustrates the lifecycle of a background job from initiation to completion.

Step-by-Step: Adding a New Job

To add a new background job to the system, follow these three primary steps.

1. Define the Job Name and Payload Schema

All job definitions must reside in the @repo/jobs package. You must add the new job name to the JobNames enum and define a corresponding Zod schema in jobSchemas.

// packages/jobs/src/types.ts

export const JobNames = {
  // ... existing jobs
  NewFeatureTask: 'feature.task.do',
} as const;

export const jobSchemas = {
  // ... existing schemas
  [JobNames.NewFeatureTask]: z.object({
    id: z.string(),
    metadata: z.record(z.string()),
  }),
} as const;

Sources: packages/jobs/src/types.ts:3-74, apps/worker/README.md:32-35

2. Implement and Register the Handler

The worker process needs to know how to execute the job. This involves creating a handler function and registering it in the worker's entry point.

• Create Handler: Implement the business logic (usually in a specific service package like @repo/review or @repo/ingestion).
• Register in Worker: Use defineHandler in the worker's main() function to link the job name to the logic.

// apps/worker/src/index.ts

const newFeatureTask = createNewFeatureHandler({ prisma, logger });

await registerHandlers([
  defineHandler(JobNames.NewFeatureTask, { batchSize: 5 }, async (job) => {
    await newFeatureTask(job.data);
  }),
]);

Sources: apps/worker/src/index.ts:50-88, apps/worker/README.md:32-35

3. Produce (Enqueue) the Job

Trigger the job from elsewhere in the codebase (e.g., a webhook handler or a UI action) using the boss instance retrieved from the container.

// Example from apps/web/app/api/webhooks/github/route.ts
const c = await getContainer();
await c.boss.send(JobNames.NewFeatureTask, { id: "123", metadata: {} });

Sources: apps/web/app/api/webhooks/github/route.ts:98-100, packages/jobs/README.md:15-18

Job Configuration and Concurrency

The system supports various configuration options per job type to manage load and rate limits.

Configuration	Description	Usage Example
Batch Size	Number of jobs the worker pulls at once for this handler.	batchSize: 8 for webhooks
Serialization	Postgres advisory locks can be used to ensure only one job runs per entity (e.g., per installation).	Handled via advisory locks in packages/jobs
Queue Creation	Queues must be explicitly created using createQueue() before sending.	Called on web and worker boot

Sources: apps/worker/src/index.ts:51-87, packages/jobs/README.md:32-38, README.md:65-69

Current System Jobs

The following table summarizes the existing background jobs currently implemented in the platform.

Job Name	Payload Key Fields	Purpose
webhook.process	deliveryId	Initial routing of GitHub webhook events.
review.pr	repositoryId, pullRequestNumber, aiReviewId	Core AI review logic and posting to GitHub.
index.repository	repositoryId, installationId	Full AST-based indexing of a repository.
index.file	filePath, ref	Incremental re-indexing of a single changed file.
review.memory.store	aiReviewId	Persisting review outcomes for future RAG context.
comment.developer.process	githubCommentId, commentBody	Processing and classifying dev replies to AI.

Sources: packages/jobs/src/types.ts:3-74, apps/worker/src/index.ts:55-87

Conclusion

Adding background jobs involves a structured process of defining types in @repo/jobs, implementing logic in the relevant package, and registering the handler in apps/worker. This architecture ensures the AI Code Reviewer remains scalable and resilient by decoupling slow LLM and API operations from the user-facing web interface.

Sources: README.md:46-53, apps/worker/README.md:17-22