Background Jobs & Queues
Relevant source files
The following files were used as context for generating this wiki page:
Twenty utilizes a distributed task processing system to handle asynchronous operations, long-running processes, and deferred logic. This system ensures that the main API remains responsive by offloading heavy lifting to dedicated worker processes. The architecture is primarily built on NestJS, BullMQ, and Redis.
Sources: CLAUDE.md:143-145, packages/twenty-server/src/engine/core-modules/message-queue/message-queue.module.ts
System Architecture
The background job system consists of three main components: a message queue producer (the server), a broker (Redis), and a consumer (the worker).
Core Components
| Component | Technology | Description |
|---|---|---|
| Queue Manager | BullMQ | Manages job lifecycle, retries, and concurrency. |
| Data Store | Redis | Acts as the message broker and state store for queues. |
| Producer | NestJS Server | Dispatches jobs to the queue during API requests or events. |
| Consumer | Queue Worker | A dedicated process that executes the job logic. |
Sources: CLAUDE.md:139-145, packages/twenty-server/src/engine/core-modules/message-queue/message-queue.module.ts
Process Flow
The following diagram illustrates how a background job is initiated from a user request and processed by the worker:
This flow shows the decoupling of request handling from task execution.
Sources: CLAUDE.md:139-145, packages/twenty-server/src/queue-worker/queue-worker.ts
Implementation Details
Message Queue Module
The MessageQueueModule is a core module in the NestJS backend. It configures the connection to Redis and registers the necessary queues used across the application.
Sources: packages/twenty-server/src/engine/core-modules/message-queue/message-queue.module.ts
Queue Worker
The queue-worker.ts file defines the entry point for the background processing unit. In development and production environments, this runs as a separate process from the main server.
To start the background worker individually:
npx nx run twenty-server:worker
Sources: CLAUDE.md:21
Automation and Logic Functions
The background job system is heavily utilized by the Apps and Workflows systems. Logic functions often run as background jobs to handle side effects without blocking the UI.
Logic Function Execution
Logic functions can be triggered by record changes (e.g., record create) or through manual workflow triggers. These executions are tracked via logs and managed by the queue worker.
Sources: packages/twenty-codex-plugin/skills/develop-app/SKILL.md:4-15, packages/twenty-codex-plugin/CHANGELOG.md:88-100
Sequence of a logic function execution through the background queue.
Sources: packages/twenty-codex-plugin/skills/develop-app/SKILL.md:46-55, packages/twenty-codex-plugin/CHANGELOG.md:120-125
Operations and Maintenance
Synchronization and Sync Modes
When developers modify app entities, they must synchronize these changes. The system supports "one-shot" sync modes for automated environments to ensure consistency without the overhead of persistent watch modes.
| Command | Mode | Usage |
|---|---|---|
yarn twenty dev --once | One-shot | Synchronizes app changes once and exits. Preferred for agents. |
yarn twenty dev | Watch | Continuously monitors files and syncs. |
Sources: packages/twenty-codex-plugin/AGENTS.md:38-40, packages/twenty-codex-plugin/CHANGELOG.md:110-115
Logs and Troubleshooting
The CLI provides tools to inspect background job execution, specifically for logic functions:
yarn twenty dev:function:logs
Sources: packages/twenty-codex-plugin/CHANGELOG.md:120
Conclusion
The Background Jobs & Queues system is a fundamental part of Twenty's scalability. By leveraging BullMQ and Redis, the platform supports complex automations, logic functions, and high-volume data operations while maintaining high performance for end-users.