Scaling Architecture

The n8n scaling architecture, also referred to as "Queue Mode," provides a mechanism to distribute workflow executions across multiple, independent worker processes. This design offloads intensive processing from the main n8n instance, enabling horizontal scalability and improved throughput for handling a large volume of executions.

The architecture is composed of a main n8n instance that orchestrates tasks and a fleet of workers that execute them. Communication between the main instance and the workers is facilitated by a Pub/Sub messaging system, which uses Redis as its backend. This decoupled, event-driven approach ensures that the system is resilient and can scale efficiently.

Sources: CONTRIBUTING.md, packages/cli/src/scaling/pubsub/pubsub.eventbus.ts:10-12

Architecture Overview

The scaling system consists of three primary components: the Main n8n Instance, the Redis Pub/Sub Event Bus, and one or more Worker Instances. The main instance publishes execution jobs to the event bus, and workers subscribe to these jobs, execute them, and report back.

The following diagram illustrates the high-level interaction between these components.

Core Components

worker CLI Command

The entry point for launching a worker process is the n8n worker command. This command initializes a new worker instance that connects to the main n8n application's ecosystem via the event bus.

It accepts the following options:

When executed, this command instantiates and invokes the ScalingService to begin the worker's lifecycle.

Sources: packages/cli/src/commands/worker.ts:11-23

ScalingService

This service acts as the primary orchestrator for initializing a worker process. Its main responsibility is to create and manage the WorkerServer.

• startWorker(options): This method is called by the worker command. It creates a new WorkerServer with the provided concurrency options and starts it. It also sets up signal handlers for SIGTERM and SIGINT to ensure a graceful shutdown of the worker server.

Sources: packages/cli/src/scaling/scaling.service.ts:15-32

WorkerServer

The WorkerServer class encapsulates the core logic of a worker instance. It is responsible for connecting to the event bus and listening for incoming jobs.

• start(): This method initiates the worker's operation by subscribing to the event bus. Once subscribed, it is ready to receive and process jobs.
• stop(): This method handles the graceful shutdown of the worker. It unsubscribes from the event bus and disconnects, ensuring that no new jobs are accepted and allowing in-progress jobs to finish.

The lifecycle of a worker is managed by these two methods, controlled by the ScalingService.

Sources: packages/cli/src/scaling/worker-server.ts:13-34

PubSubEventBus

The PubSubEventBus is the communication backbone of the scaling architecture. It is an implementation of the IEventBus interface that uses ioredis to connect to a Redis server for publishing and subscribing to events.

Key implementation details:

• It maintains two separate Redis client connections: a publisher and a subscriber. This is a standard practice to prevent a client that is busy listening for messages from being blocked when trying to publish new messages.
• The publish(event) method serializes an event object and publishes it to a specific Redis channel derived from the event's name.
• The subscribe(eventName, listener) method subscribes to a channel and, upon receiving a message, deserializes it and passes it to the provided listener function for processing by the WorkerServer.

The class diagram below shows the structure of PubSubEventBus.

Communication Flow

The following sequence diagram illustrates how a job is published by the main instance and processed by a worker.

Frontend Awareness

The n8n frontend is aware of the scaling architecture, providing UI elements for monitoring and management, which are identified as an enterprise feature.

The following constants defined in packages/frontend/editor-ui/src/constants.ts indicate this integration:

These constants show that there is a dedicated UI (WorkerView) for users with the appropriate license to observe the status and performance of the worker instances.

Sources: packages/frontend/editor-ui/src/constants.ts:114, 461, 481