Execution Recovery & Crash Handling
Relevant source files
The following files were used as context for generating this wiki page:
- packages/cli/src/executions/execution-recovery.service.ts
- packages/cli/src/executions/execution-crash.service.ts
- packages/cli/src/crash-journal.ts
- packages/cli/src/executions/tests/execution-recovery.service.integration.test.ts
- packages/cli/src/modules/mcp/tools/workflow-builder/mcp-instructions.ts
- packages/cli/src/active-workflow-manager.ts (Inferred from execution-recovery.service.integration.test.ts:39)
Execution Recovery & Crash Handling provides mechanisms to detect, report, and recover from workflow execution failures caused by process crashes or unexpected terminations. This system ensures that executions left in a "running" state after a crash are correctly transitioned to a "crashed" state and that users are notified of these events.
The system uses a combination of filesystem-based journals and database state analysis to identify interrupted executions upon service restart or during active monitoring.
Sources: packages/cli/src/executions/execution-recovery.service.ts, packages/cli/src/crash-journal.ts
Architecture and Components
The recovery system consists of several specialized services that coordinate detection and state updates.
ExecutionRecoveryService
The ExecutionRecoveryService acts as the primary orchestrator for identifying and repairing "zombie" executions. When n8n starts, this service checks for executions that the database still marks as "running" but which no longer have an active process.
- Startup Recovery: Scans the database for any execution with a
runningstatus and transitions them tocrashed. - Log Recovery: Reconstructs the final state of an execution by analyzing logs or event messages to determine the last executed node and failure timestamp.
- Workflow Deactivation: Automatically deactivates workflows that crash repeatedly to prevent infinite crash loops.
Sources: packages/cli/src/executions/execution-recovery.service.ts:51-80, packages/cli/src/executions/tests/execution-recovery.service.integration.test.ts:98-115
ExecutionCrashService
The ExecutionCrashService handles the logic for transitioning a specific execution to a crashed state. It performs the database updates to set the status to crashed and logs the execution-crashed event to the internal event bus.
Sources: packages/cli/src/executions/execution-crash.service.ts:18-35
CrashJournal
The CrashJournal provides a filesystem-based persistence layer to track active executions. If the n8n process is killed abruptly (e.g., OOM or power failure), the existence of an entry in the crash journal allows the next process instance to identify exactly which execution was running at the time of the crash.
Sources: packages/cli/src/crash-journal.ts:14-25
Execution Recovery Flow
The following diagram illustrates the sequence of events when n8n detects a crashed execution during startup or via log analysis.
The diagram shows the recovery process triggered during application initialization.
Sources: packages/cli/src/executions/execution-recovery.service.ts:60-95, packages/cli/src/executions/execution-crash.service.ts:25-45
Log-Based Recovery Logic
When detailed event logs are available, the system can "amend" a crashed execution with more precise data. This is crucial for executions that crashed while processing high volumes of data.
| Logic Step | Condition | Action |
|---|---|---|
| Status Check | Execution is already success, error, or canceled | Return null (no recovery needed) |
| Dataless Recovery | Execution has null data but was successful | Transition to crashed and reconstruct result data |
| Node Trace | Logs contain n8n.node.started but no finished | Identify the last node as the cause of the crash |
| Timestamping | Logs contain node event timestamps | Set execution stoppedAt to the timestamp of the last node event |
Sources: packages/cli/src/executions/execution-recovery.service.ts:130-170, packages/cli/src/executions/tests/execution-recovery.service.integration.test.ts:230-270
Workflow Auto-Deactivation
The system includes a safeguard to prevent workflows from repeatedly crashing a server. If a workflow's most recent executions result in crashes, the ExecutionRecoveryService deactivates the workflow.
Deactivation Criteria
- The service identifies a set of workflows associated with recovered crashes.
- For each workflow, it checks the status of the last $N$ executions (defined by configuration).
- If all $N$ executions are
crashed, the workflow is deactivated.
Deactivation Process
- The workflow
activeVersionIdis set tonullin the database. - Active triggers (e.g., Webhooks, Cron) are removed via the
ActiveWorkflowManager. - If the Workflow Publication Service is enabled, it records a
deactivatedevent in the history and notifies the publication outbox.
Sources: packages/cli/src/executions/execution-recovery.service.ts:285-330, packages/cli/src/executions/tests/execution-recovery.service.integration.test.ts:470-520
The flowchart summarizes the logic used to decide whether to automatically disable a failing workflow.
Sources: packages/cli/src/executions/execution-recovery.service.ts:310-350, packages/cli/src/executions/tests/execution-recovery.service.integration.test.ts:500-530
Configuration Options
The recovery behavior is controlled via global execution settings.
| Configuration Key | Type | Description |
|---|---|---|
executions.recovery.workflowDeactivationEnabled | boolean | Enables or disables the auto-deactivation logic. |
executions.recovery.workflowDeactivationThreshold | number | The number of consecutive crashes required to trigger deactivation. |
N8N_DEV_RELOAD | boolean | If enabled, the crash journal is disabled to allow rapid development cycles. |
Sources: packages/cli/src/executions/execution-recovery.service.ts:55, packages/cli/src/crash-journal.ts:20, packages/cli/src/tests/execution-recovery.service.integration.test.ts:95
Error Handling in Recovery
If a recovery attempt for a specific execution fails (e.g., due to database locks), the service catches the error and continues to the next execution. This ensures that one corrupted execution record does not block the recovery of the rest of the system.
When deactivation fails for a specific workflow, the system still marks the associated executions as crashed before proceeding to handle the remaining workflows in the queue.
Sources: packages/cli/src/executions/execution-recovery.service.ts:98-110, packages/cli/src/executions/tests/execution-recovery.service.integration.test.ts:590-620
Summary
Execution Recovery & Crash Handling ensures data integrity by preventing executions from remaining in a permanent "running" state after a system failure. By combining startup database scans, log reconstruction, and automatic workflow deactivation, the system provides a resilient environment for workflow automation.