Scaling & Concurrency
Relevant source files
The following files were used as context for generating this wiki page:
Scaling and concurrency management in n8n ensures that workflow executions remain stable and performant under high load. The system utilizes a combination of task batching, queue-based job processing, and dedicated concurrency control services to prevent resource exhaustion. This architecture allows n8n to handle thousands of items across parallel branches and distributed worker nodes while maintaining operational integrity.
The core of the scaling logic resides in the ScalingService and JobProcessor, which handle the distribution of tasks, while the ConcurrencyControlService manages the limits on simultaneous executions to prevent memory issues and service degradation.
Sources: packages/cli/src/scaling/scaling.service.ts, packages/cli/src/concurrency/concurrency-control.service.ts, packages/cli/src/scaling/job-processor.ts
Concurrency Control System
The ConcurrencyControlService is responsible for enforcing limits on how many workflows can run simultaneously. It monitors active executions and uses a queueing mechanism to hold new requests when the defined concurrency limits are reached. This is critical for preventing OOM (Out of Memory) errors during spikes in workflow triggers.
Logic and Components
- Limit Enforcement: The service checks the current number of active executions against a configurable maximum.
- Queue Management: If the limit is reached, incoming executions are placed into a "waiting" state.
- Resource Protection: By limiting concurrency, the system ensures that each active execution has enough CPU and memory to complete without affecting other processes.
Sources: packages/cli/src/concurrency/concurrency-control.service.ts, AGENTS.md:105-110
Execution Scaling and Batching
To handle large datasets without overwhelming the system, n8n employs specific scaling techniques such as "Split In Batches". This prevents a single node from attempting to process massive amounts of data in a single cycle, which would otherwise result in performance bottlenecks or timeouts.
Scaling Patterns
| Pattern | Description | Implementation Detail |
|---|---|---|
| Split In Batches | Processes large datasets in chunks (e.g., 100 items at a time). | Prevents memory overload and workflow timeouts. |
| Parallel Execution | Connects branches to specific merge inputs. | Uses merge nodes to combine parallel data streams. |
| Queue Mode | Offloads execution tasks to worker nodes. | Managed via Redis and ScalingService. |
Sources: packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts:10-15, packages/cli/src/scaling/scaling.service.ts
Scaling Workflow Diagram
The following diagram illustrates the flow of a workflow execution transitioning from a trigger to a scaled job processor.
Scaling Service and Job Processing
The ScalingService acts as the orchestrator for distributed execution. When n8n is configured in "Queue Mode", this service pushes workflow tasks to a shared message broker (typically Redis). The JobProcessor on worker nodes then picks up these jobs for execution.
Architecture and Data Flow
- Job Creation: When a workflow is triggered, the
ScalingServiceprepares a job payload containing the workflow definition and input data. - Distribution: Jobs are added to a queue.
- Worker Pickup: Independent worker instances subscribe to the queue and utilize
JobProcessorto run the nodes. - State Synchronization: Execution results are reported back to the main instance to update the database and UI.
Sources: packages/cli/src/scaling/scaling.service.ts, packages/cli/src/scaling/job-processor.ts
Job Processing Sequence
The sequence below shows the interaction between the Main instance and a Worker node during a scaled execution.
Performance Best Practices
To optimize scaling and concurrency, certain node-level strategies are recommended within the workflow design itself.
- Early Filtering: Use
Filternodes early in the workflow to reduce the item count before reaching heavy processing nodes or AI agents. - Batch Sizes: For datasets over 100 items, the
Split In Batchesnode should be configured with an appropriate batch size (usually 50-100) to maintain high performance. - Memory Management: Avoid using "Function" nodes for per-item operations if "Function Item" (or the newer "Code" node) can suffice, as "Function" nodes load the entire item array into memory.
Sources: packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts:130-145, packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts:180-190
Summary
Scaling and concurrency in n8n are managed through a multi-layered approach involving the ConcurrencyControlService for execution limits and the ScalingService for distributed job processing. By combining architectural safeguards with user-level design patterns like batching and filtering, the system maintains stability even when processing complex, high-volume workflows across distributed environments.