Scaling n8n (Queue Mode)
Relevant source files
The following files were used as context for generating this wiki page:
n8n can be configured to run in queue mode to support horizontal scaling. In this mode, the main n8n process is responsible for handling API requests, scheduling, and managing workflows, but it offloads the actual workflow executions to a pool of dedicated worker processes. This separation is achieved using Redis as a message broker, which allows for a distributed and scalable architecture.
This setup improves performance and reliability by distributing the execution load across multiple machines or processes. The main process enqueues jobs, and workers from the pool pick them up for execution, allowing the main process to remain responsive to user interactions and API calls.
Architecture
In queue mode, the n8n ecosystem is composed of three primary components: one or more main processes, one or more worker processes, and a Redis instance that acts as the message queue.
- Main Process: The user-facing part of n8n. It handles all UI interactions, API requests, and webhook triggers. When a workflow is triggered, the main process creates a job and pushes it onto a Redis queue instead of executing it directly.
- Worker Process: A headless n8n instance started with the
workercommand. Its sole purpose is to listen for jobs on the Redis queue, execute them, and manage their lifecycle. Multiple workers can run in parallel to process a high volume of executions. - Redis: The backbone of the queueing system. It uses BullMQ to manage job queues, ensuring that jobs are distributed among available workers.
The following diagram illustrates the high-level architecture of n8n in queue mode.
Configuration
To enable queue mode, specific environment variables must be set. These variables configure the execution mode and the connection to Redis.
Enabling Queue Mode
The primary setting to enable this architecture is EXECUTIONS_MODE.
| Variable | Description | Default | Source |
|---|---|---|---|
EXECUTIONS_MODE | Defines the execution mode. Must be set to queue to enable scaling. | regular | scaling-mode.config.ts:13-19 |
Redis Configuration
A running Redis instance is required for queue mode. The following environment variables configure the connection to Redis.
| Variable | Description | Default | Source |
|---|---|---|---|
DB_REDIS_HOST | The hostname of the Redis server. | localhost | redis.config.ts:14-17 |
DB_REDIS_PORT | The port of the Redis server. | 6379 | redis.config.ts:19-22 |
DB_REDIS_PASSWORD | The password for Redis authentication. | undefined | redis.config.ts:24-27 |
DB_REDIS_DB | The Redis database index to use. | 0 | redis.config.ts:29-32 |
DB_REDIS_USE_TLS | Whether to use TLS for the Redis connection. | false | redis.config.ts:34-37 |
DB_REDIS_PREFIX | A prefix for all keys created in Redis by n8n. | undefined | redis.config.ts:39-42 |
Multi-Main Process Configuration
For high-availability setups, you can run multiple main n8n processes. A specific configuration is needed to ensure that scheduled tasks (like cron jobs) are not duplicated.
| Variable | Description | Default | Source |
|---|---|---|---|
N8N_MULTI_MAIN_PROCESS | Set to true when running more than one main process. This ensures that only one QueueScheduler is active across all main instances. | false | multi-main-setup.ee.ts:21-24 |
When N8N_MULTI_MAIN_PROCESS is enabled, n8n uses a distributed lock in Redis to elect a single main process to run the BullMQ QueueScheduler. This prevents multiple instances from scheduling the same jobs.
Sources: packages/cli/src/scaling/multi-main-setup.ee.ts:33-51
Components
Worker Process
The worker is a dedicated process initiated via the CLI. It does not serve any API or UI.
To start a worker, run the following command:
n8n worker
This command initializes a JobProcessor instance, which connects to Redis and begins processing jobs from the queue. The worker process can be configured with options to control its behavior, such as the queues it listens to and its concurrency level.
Sources: packages/cli/src/commands/worker.ts:23-41
JobProcessor
The JobProcessor class is the core of the worker process. It is responsible for fetching jobs from the Redis queue and executing them.
Key responsibilities include:
- Connecting to the Redis instance.
- Creating a BullMQ
Workerto listen for and process jobs. - Handling job execution, including success, failure, and progress updates.
- Managing concurrency to control how many jobs are processed simultaneously.
The following sequence diagram shows the initialization and job processing loop of the JobProcessor.
The process method within JobProcessor is the entry point for executing a received job. It retrieves the execution ID from the job data and calls the internal executeWorkflow function to run the workflow.
Sources: packages/cli/src/scaling/job-processor.ts:111-120
Job Processing Flow
The end-to-end flow for a workflow execution in queue mode involves coordination between the main process, Redis, and a worker.
This flowchart details the lifecycle of a single workflow execution.
This distributed model ensures that the main process remains available for new requests while long-running executions are handled asynchronously by the scalable worker pool.
Sources: packages/cli/src/commands/worker.ts, packages/cli/src/scaling/job-processor.ts
Summary
n8n's queue mode provides a robust solution for scaling workflow executions horizontally. By leveraging a message queue like Redis, it decouples the main application from the execution logic, allowing for a highly available and performant system. This architecture is ideal for production environments with high throughput requirements, as it allows for the independent scaling of worker processes to meet demand without impacting the core application's responsiveness. Proper configuration of the execution mode, Redis connection, and multi-main process settings is crucial for a successful deployment.