Scaling with Workers
Relevant source files
The following files were used as context for generating this wiki page:
- packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts
- packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/document-processing.ts
- packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/monitoring.ts
- packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/scraping-and-research.ts
- CONTRIBUTING.md
- AGENTS.md
Scaling with Workers refers to the architectural approach of distributing workflow execution across multiple processes to handle high-volume data, prevent memory exhaustion, and maintain system stability. This system utilizes a queue-based architecture where a main instance orchestrates tasks and dedicated worker instances perform the heavy lifting of node execution.
In n8n, scaling is essential when workflows process large datasets (e.g., 1000+ items), perform intensive document processing, or require high-frequency monitoring. By offloading these tasks to workers, the main instance remains responsive for UI interactions and trigger management. Sources: packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts:14-16, CONTRIBUTING.md:275-285
Worker Architecture and Infrastructure
The scaling infrastructure relies on a multi-process setup often orchestrated via Docker or similar containerization. In a typical scaled environment, n8n runs in a "Queue Mode" configuration.
Core Components
| Component | Description |
|---|---|
| Main Instance | Handles the Editor UI, API, and workflow triggers. It pushes execution jobs to the queue. |
| Redis | Acts as the message broker and state store for the execution queue. |
| Workers | Independent processes that pull jobs from Redis and execute workflow nodes. |
| Database | Shared persistence layer (PostgreSQL recommended) for workflow definitions and execution history. |
Sources: CONTRIBUTING.md:275-285, AGENTS.md:120-130
Execution Flow
The following diagram illustrates how an execution is handed off from the Main instance to a Worker.
The Main instance identifies a trigger and pushes the execution job into Redis. Available Workers pull jobs from the queue, execute the logic, and update the shared database with results. Sources: CONTRIBUTING.md:275-285, packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/monitoring.ts:50-60
Batch Processing and Memory Management
To scale effectively at the workflow level, you must manage how data is passed through the system. Large datasets can cause memory issues if processed as a single block.
The Split In Batches Node
The Split In Batches node is the primary tool for scaling large data operations. It breaks a large array of items into smaller chunks (e.g., 100 or 200 items per batch). This prevents the worker process from exceeding memory limits and avoids workflow timeouts. Sources: packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts:114-120, packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/scraping-and-research.ts:47-55
Sub-workflow Delegation
For heavy processing tasks, such as OCR or complex data transformations, delegating to sub-workflows is a recommended scaling pattern.
- Parent Workflow: Uses the
Execute Workflownode. - Sub-workflow: Starts with an
Execute Workflow Trigger. - Benefit: This modularity allows the system to manage memory more efficiently by isolating heavy tasks into their own execution context.
Sources: packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/document-processing.ts:165-180
Performance Considerations for Scaling
Scaling requires tuning both the infrastructure and the workflow design to avoid bottlenecks.
Infrastructure Tuning
- Concurrency: You can scale up the number of workers to handle more parallel executions.
- Database Choice: While SQLite is supported for single instances, PostgreSQL is required for multi-main or worker-based stacks to handle concurrent writes.
- Redis: Ensure Redis has sufficient memory to hold the job queue during peak loads.
Sources: CONTRIBUTING.md:275-285, AGENTS.md:120-130
Workflow Design Optimization
- Filter Early: Remove unnecessary data as early as possible in the workflow to reduce the payload size passed between nodes and workers. Sources: packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts:58-62
- Avoid Alert Storms: In monitoring workflows, use state management (e.g., checking against a database) to only alert on status changes rather than every execution. Sources: packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/monitoring.ts:40-45
- Wait Nodes: Use
Waitnodes to respect external API rate limits, preventing workers from being blocked by 429 errors. Sources: packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/scraping-and-research.ts:77-78
Summary
Scaling with Workers enables n8n to handle enterprise-grade workloads by decoupling the management of workflows from their execution. By utilizing Redis for queuing and implementing batching patterns within workflows, the system can process thousands of items across multiple workers without compromising the stability of the main application instance. Sources: CONTRIBUTING.md:275-285, packages/@n8n/workflow-sdk/src/prompts/best-practices/guides/data-analysis.ts:14-16