Scaling & Worker Pools
Relevant source files
The following files were used as context for generating this wiki page:
The Scaling and Worker Pools system manages the distribution of workflow executions across multiple computing instances. It allows n8n to handle high-volume workloads by offloading execution tasks from the Main process to a pool of Worker processes. This architecture ensures the platform remains responsive under heavy load and provides high availability for critical automation tasks.
The system relies on a shared communication layer, typically Redis, to coordinate tasks between the Main instance and the Workers. This multi-process approach prevents long-running or resource-intensive workflows from blocking the user interface or API response times.
Sources: [packages/cli/src/scaling/scaling.service.ts], [CONTRIBUTING.md:278-290](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/CONTRIBUTING.md#L278-L290)
Architecture Overview
n8n uses a multi-instance architecture where a Main process coordinates work and Workers execute the actual workflow logic.
Scaling Architecture Flow
The following diagram illustrates the relationship between the Main process, the message broker (Redis), and the Worker pool.
The Main process acts as the producer, pushing execution jobs into a Redis-backed queue, while multiple Worker processes act as consumers to process these jobs. Sources: [packages/cli/src/scaling/scaling.service.ts], [CONTRIBUTING.md:278-290](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/CONTRIBUTING.md#L278-L290)
Component Descriptions
| Component | Description |
|---|---|
| Main Process | Handles the UI, API, triggers, and job scheduling. It delegates heavy executions to Workers. |
| Worker Process | Dedicated instances that pull jobs from the queue and run the n8n workflow engine. |
| Redis | Serves as the message broker, job queue, and distributed locking mechanism. |
| Scaling Service | The internal service managing job lifecycles and Worker coordination. |
Sources: [packages/cli/src/scaling/scaling.service.ts], [packages/cli/src/scaling/worker-server.ts]
Scaling Service Logic
The ScalingService class coordinates the distribution of executions. It initializes job queues and manages the lifecycle of workflow runs when scaling is enabled.
Job Distribution Sequence
When a workflow is triggered, the scaling service determines how to handle the execution.
The Scaling Service provides a layer of abstraction that allows the system to switch between local execution and distributed worker execution. Sources: [packages/cli/src/scaling/scaling.service.ts:30-70](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/packages/cli/src/scaling/scaling.service.ts#L30-L70)
Key Methods in ScalingService
init(): Sets up the Redis connections and initializes the Bull processing queues.addJob(jobData): Pushes a new execution task into the distributed queue.stop(): Gracefully shuts down queues and connections.
Sources: [packages/cli/src/scaling/scaling.service.ts:40-120](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/packages/cli/src/scaling/scaling.service.ts#L40-L120)
Worker Server Implementation
The WorkerServer represents the entry point for a worker process. It is responsible for listening to the queue and initializing the execution environment for each job.
Worker Initialization Flow
A Worker instance primarily runs a specialized server that stays connected to the shared Redis instance. Sources: [packages/cli/src/scaling/worker-server.ts:10-50](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/packages/cli/src/scaling/worker-server.ts#L10-L50)
Distributed Locking with Redis
To prevent race conditions—such as multiple workers attempting to process the same trigger or unique execution—n8n employs a distributed locking mechanism via the RedisLockService.
Locking Logic
- Lock Acquisition: Before a sensitive operation, a service requests a lock with a specific key.
- TTL (Time to Live): Locks have an expiration time to prevent permanent deadlocks if a process crashes.
- Release: The lock is explicitly released once the operation completes.
Sources: [packages/cli/src/scaling/redis-lock.service.ts:15-45](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/packages/cli/src/scaling/redis-lock.service.ts#L15-L45)
Resource Management and Batching
When handling large datasets in a scaled environment, n8n recommends specific node patterns to prevent memory exhaustion in worker processes.
- Split In Batches: Processes large datasets in chunks (e.g., 100 items) to prevent worker OOM (Out of Memory) errors.
- Wait Node: Introduces delays to respect external API rate limits during high-concurrency worker executions.
- Sub-workflows: Offloads complex logic into separate executions that can be distributed across different workers.
Sources: [packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts:98-115](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/packages/%40n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts#L98-L115), [AGENTS.md:120-140](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/AGENTS.md#L120-L140)
Configuration and Deployment
Scaling is typically configured via environment variables. The n8n-containers filter in the pnpm workspace allows for local testing of these configurations.
| Command | Purpose |
|---|---|
pnpm --filter n8n-containers stack:queue | Spins up a local stack with a Main process and Workers. |
pnpm --filter n8n-containers stack --mains 4 --workers 20 | Scales the local test environment to multiple instances. |
Sources: [CONTRIBUTING.md:280-305](https://github.com/thomasshunter716-cell/n8n/blob/cea974e770027a0834e1e7be55fbfb6c23169b03/CONTRIBUTING.md#L280-L305)
Summary
The Scaling and Worker Pools system provides n8n with the capability to process large-scale automation by distributing tasks through a Redis-backed queue. By separating the Main management process from the Worker execution nodes, the platform ensures stability, high availability, and efficient resource utilization across multi-instance deployments. Distributed locking and batching strategies further safeguard system integrity during high-concurrency operations.