Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• packages/testing/playwright/CONTRIBUTING.md
• CLAUDE.md
• CONTRIBUTING.md
• packages/testing/playwright/AI-TEST-CHEAT-SHEET.md
• packages/testing/playwright/tests/cli-workflows/README.md
• packages/@n8n/ai-workflow-builder.ee/evaluations/README.md
• packages/testing/playwright/tests/ui/39-projects.spec.ts
• cypress/e2e/group5/29-templates.cy.ts
• packages/nodes-base/nodes/Notion/test/GenericFunctions.test.ts
• packages/frontend/editor-ui/src/components/WhatsNewModal.test.ts
• packages/nodes-base/nodes/Evaluation/test/metricHandlers.test.ts
• packages/@n8n/ai-workflow-builder.ee/test/test-utils.ts

The n8n project employs a multi-layered testing strategy to ensure code quality, prevent regressions, and maintain stability across its frontend, backend, and node ecosystem. The strategy leverages several modern testing frameworks, each tailored to a specific part of the application. The primary tools include Playwright for end-to-end testing, Jest for backend unit testing, and Vitest for frontend unit testing.

A significant emphasis is placed on structured, maintainable tests, particularly within the Playwright E2E suite, which follows a well-defined architectural pattern. The project is also in the process of migrating legacy Cypress tests to Playwright. In addition to traditional testing methods, the project includes a sophisticated evaluation framework for its AI-powered features, demonstrating a commitment to quality in both conventional and cutting-edge components.

Sources: CLAUDE.md:104-108, CONTRIBUTING.md:214-216, packages/testing/playwright/CONTRIBUTING.md:23-25, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:3-6

Testing Landscape

The project's testing landscape is divided into several distinct categories, each with its own set of tools and purposes.

This diagram provides a high-level overview of the different testing types and the primary technologies used for each.

Test Type	Scope	Primary Tool(s)	Description
End-to-End (E2E)	UI, API, full user flows	Playwright (current), Cypress (legacy)	Simulates real user interactions from the browser to the backend. The project is standardizing on Playwright.
Unit Testing	Backend, Nodes, Frontend Components	Jest (backend), Vitest (frontend)	Tests individual functions, classes, or components in isolation.
Workflow Testing	Node correctness, API compatibility	Playwright	A specialized framework for executing n8n workflows and validating their output, often using schema validation.
AI Evaluation	AI Workflow Builder	LangChain, Langsmith	A framework to evaluate the correctness and quality of AI-generated workflows from natural language prompts.

Sources: CLAUDE.md:104-108, CONTRIBUTING.md:214-232, packages/testing/playwright/tests/cli-workflows/README.md:3-6, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:3-6

End-to-End (E2E) Testing

E2E testing is a cornerstone of the quality assurance process, with a clear migration path from legacy Cypress tests to a more robust Playwright implementation.

Sources: CLAUDE.md:107-108

Playwright Architecture

The Playwright test suite is built on a "separation of concerns" principle, using layered architectures for both UI and API testing to ensure tests are readable, maintainable, and scalable.

Sources: packages/testing/playwright/CONTRIBUTING.md:23-33

UI Testing Architecture

UI tests follow a four-layer architecture that abstracts business logic from low-level UI interactions.

This diagram illustrates the flow of control from a high-level test script down to the base page utilities.

• Tests (*.spec.ts): Contain readable, high-level scenarios that describe a user's journey. They utilize Composables to perform actions.
• Composables (*Composer.ts): Orchestrate complex, multi-page business workflows. For example, createProject or executeWorkflowAndWaitForNotification.
• Page Objects (*Page.ts): Encapsulate the interactions for a single page or component. They contain element locators and simple action methods (e.g., clickSaveButton).
• BasePage: Provides generic, reusable interaction methods like clickByTestId that are extended by all Page Objects.

Sources: packages/testing/playwright/CONTRIBUTING.md:27-33, packages/testing/playwright/AI-TEST-CHEAT-SHEET.md:101-109

API Testing Architecture

API tests use a simpler, two-layer approach for direct interaction with the n8n backend.

• Tests (*.spec.ts): Define the test scenario and make calls to API services.
• API Services: A collection of helper classes that wrap the n8n REST API, providing methods for tasks like creating workflows (api.workflowApi.createWorkflow) or managing features (api.enableFeature).

Sources: packages/testing/playwright/CONTRIBUTING.md:35-38, packages/testing/playwright/tests/ui/39-projects.spec.ts:8-15

The following sequence diagram shows a typical combined API and UI test flow.

Workflow Testing Framework

A specialized testing framework built on Playwright is used to validate n8n workflows, ensuring node correctness and detecting breaking changes in external APIs.

Sources: packages/testing/playwright/tests/cli-workflows/README.md:3-6

Key features include:

• Two Test Modes:
  1. Basic Run: Executes workflows and verifies they run without errors.
  2. Schema Mode: Validates the structure of the workflow output against a saved JSON schema snapshot. This is crucial for detecting API contract changes (e.g., renamed fields, type changes) without being sensitive to dynamic data values.
• Configuration: A workflowConfig.json file controls which workflows are active or skipped.
• Snapshot Management: Playwright's snapshot functionality is used to generate and update schema files.

This flowchart shows the schema validation process.

An example entry in workflowConfig.json:

[
  {
    "workflowId": "123",
    "status": "ACTIVE",
    "enableSchemaValidation": true
  },
  {
    "workflowId": "456",
    "status": "SKIPPED",
    "skipReason": "Depends on external API that is currently down",
    "ticketReference": "JIRA-123"
  }
]

Sources: packages/testing/playwright/tests/cli-workflows/README.md:73-86

Legacy Cypress Tests

The repository contains E2E tests written in Cypress. However, project guidelines state that new E2E tests should be written using Playwright, as Cypress is being phased out.

Sources: CLAUDE.md:107-108, CONTRIBUTING.md:226-232

Unit and Integration Testing

Unit and integration tests are crucial for verifying the correctness of individual components in isolation. The project uses different tools for the backend and frontend.

Sources: CLAUDE.md:104-106

Backend Testing (Jest)

• Framework: Jest is used for unit testing backend components, including core logic and nodes.
• Mocking: nock is used for mocking HTTP requests to external services, and jest-mock-extended is used for creating mock objects.
• Location: Tests are typically located in test subdirectories or alongside the source files with a .test.ts extension.

An example of a Jest unit test for a Notion node helper function:

// packages/nodes-base/nodes/Notion/test/GenericFunctions.test.ts
describe('Test Notion, getPageId', () => {
	let mockExecuteFunctions: MockProxy<IExecuteFunctions>;
	const id = '3ab5bc794647496dac48feca926813fd';

	beforeEach(() => {
		mockExecuteFunctions = mock<IExecuteFunctions>();
	});

	it('should return page ID directly when mode is id', () => {
		const page = {
			mode: 'id',
			value: id,
		} as INodeParameterResourceLocator;

		mockExecuteFunctions.getNodeParameter.mockReturnValue(page);

		const result = getPageId.call(mockExecuteFunctions, 0);
		expect(result).toBe(id);
	});
});

Sources: packages/nodes-base/nodes/Notion/test/GenericFunctions.test.ts:70-88, CLAUDE.md:105

Frontend Testing (Vitest)

• Framework: Vitest is used for unit testing frontend Vue components.
• Utilities: Tests leverage @testing-library/vue for rendering components and @pinia/testing for mocking Pinia stores.
• Location: Frontend tests are co-located with the components they test.

An example of a Vitest test for a Vue component:

// packages/frontend/editor-ui/src/components/WhatsNewModal.test.ts
describe('WhatsNewModal', () => {
	beforeEach(() => {
		createTestingPinia();
		uiStore = mockedStore(useUIStore);
		uiStore.modalsById = {
			[WHATS_NEW_MODAL_KEY]: {
				open: true,
			},
		};
        // ... more store mocking
	});

	it('should not render update button when no version updates available', async () => {
		const { getByTestId, queryByTestId } = renderComponent({
			props: { data: { articleId: 1 } },
		});

		await waitFor(() => expect(getByTestId('whatsNew-modal')).toBeInTheDocument());
		expect(queryByTestId('whats-new-modal-update-button')).not.toBeInTheDocument();
	});
});

Sources: packages/frontend/editor-ui/src/components/WhatsNewModal.test.ts:50-78, CLAUDE.md:106

AI Evaluation Framework

The project includes a dedicated evaluation framework to test the AI Workflow Builder. This system assesses the AI's ability to generate correct and efficient n8n workflows from natural language prompts.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:3-6

Architecture

The framework supports two evaluation modes with a parallel architecture for performance.

This diagram shows the dual-mode architecture of the AI evaluation system.

• CLI Evaluation: Runs predefined test cases locally, providing immediate feedback and reports.
• Langsmith Evaluation: Integrates with Langsmith for dataset-based evaluation and experiment tracking.
• Workflow Evaluator: An orchestrator that runs multiple evaluation chains in parallel, each focusing on a specific metric.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:8-12, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:40-45

Evaluation Metrics

Workflows are scored across seven weighted categories, each assessed by a specialized LLM chain.

Metric	Weight	Description
Functionality	25%	Does the workflow achieve the intended goal?
Connections	15%	Are nodes connected logically and optimally?
Expressions	15%	Are n8n expressions syntactically correct and valid?
Node Configuration	15%	Are node parameters set correctly?
Efficiency	10%	Is the workflow free of redundancy and optimized?
Data Flow	10%	Is data transformed and validated correctly between nodes?
Maintainability	5%	Are nodes well-named and the workflow organized logically?

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:113-149

Running Tests

The project provides several pnpm scripts to execute the different test suites.

Command	Description
pnpm test	Runs all unit tests (Jest and Vitest).
pnpm test:e2e:all	Runs all E2E tests headlessly. (Legacy Cypress)
pnpm test:all	Runs all Playwright tests (UI, CLI workflow, performance).
pnpm test:workflows	Runs the specialized workflow testing framework.
pnpm eval	Runs the AI evaluation framework using the CLI.
pnpm eval:langsmith	Runs the AI evaluation framework using Langsmith.

Sources: CLAUDE.md:33-37, CONTRIBUTING.md:218-220, packages/testing/playwright/README.md:11-12, packages/testing/playwright/tests/cli-workflows/README.md:22-25, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:162-165