The AI Workflow Builder Evaluation module is an enterprise framework designed to test and measure the quality of n8n workflows generated by the AI Workflow Builder from natural language prompts. It provides a structured way to assess various aspects of the generated workflows, from functional correctness to maintainability, ensuring they are robust and efficient.

The system is built with a parallel evaluation architecture to optimize performance and supports two primary execution modes: a local Command Line Interface (CLI) for running predefined test cases and an integration with Langsmith for more extensive, dataset-based evaluations and experiment tracking.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:1-5

Architecture Overview

The evaluation framework is organized into a modular directory structure, separating core logic, execution runners, and evaluation chains. This design supports two distinct but interconnected evaluation modes.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:7-10

The following diagram illustrates the high-level architecture of the evaluation system.

Generating architecture diagram...

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:7-10, 24-27

Directory Structure

The module's file system is organized to separate concerns, making it extensible and maintainable.

evaluations/
├── cli/                 # CLI evaluation implementation
├── langsmith/          # Langsmith integration
├── core/               # Shared evaluation logic
├── types/              # Type definitions
├── chains/             # LLM evaluation chains
│   ├── evaluators/     # Individual metric evaluators
└── utils/              # Utility functions

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:12-38

Core Components

Workflow Evaluator

The workflow-evaluator.ts is the central orchestrator that coordinates the parallel evaluation of a generated workflow. It invokes all individual metric evaluators concurrently using Promise.all() for maximum performance. After gathering results, it calculates a weighted overall score, generates a human-readable summary, and aggregates any critical issues.

This diagram shows the parallel evaluation process orchestrated by the Workflow Evaluator.

Generating architecture diagram...

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/chains/workflow-evaluator.ts:133-152, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:49-56

Individual Evaluators

Each metric category has its own specialized evaluator chain with tailored prompts and scoring logic. These evaluators run in parallel. Examples include:

functionality-evaluator.ts: Focuses on whether the workflow achieves the explicitly requested goals.
connections-evaluator.ts: Analyzes node connections and data flow paths.
data-flow-evaluator.ts: Analyzes data transformations and validation logic.
maintainability-evaluator.ts: Assesses naming, organization, and structural quality.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:58-69, packages/@n8n/ai-workflow-builder.ee/evaluations/chains/evaluators/data-flow-evaluator.ts, packages/@n8n/ai-workflow-builder.ee/evaluations/chains/evaluators/maintainability-evaluator.ts

Test Runner

The core/test-runner.ts is responsible for the execution of a single test case. Its key responsibilities include:

Generating a workflow using the WorkflowBuilderAgent.
Validating the generated workflow structure.
Invoking the evaluateWorkflow orchestrator.
Returning a structured test result, including any errors.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:40-45, packages/@n8n/ai-workflow-builder.ee/evaluations/core/test-runner.ts

Execution Runners

CLI Runner

The cli/runner.ts orchestrates local test execution. It manages test case generation, controls concurrency for parallel test runs, and generates detailed reports in Markdown and JSON formats.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/cli/runner.ts, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:82-86

Langsmith Integration

The Langsmith integration consists of two main parts:

evaluator.ts: A Langsmith-compatible function that adapts Langsmith Run objects into the evaluation system's input format, validates data, and returns structured results.
runner.ts: Orchestrates the evaluation against a Langsmith dataset, handles dataset verification, and reports errors.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/evaluator.ts, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:71-80

Data Structures

The evaluation framework uses Zod schemas to define its core data structures, ensuring type safety and validation.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/types/evaluation.ts

This class diagram shows the main data schemas and their relationships.

Generating architecture diagram...

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/types/evaluation.ts:4-65

Evaluation Metrics

Workflows are evaluated across seven weighted categories. The final score is a weighted average of each category's score.

Category	Weight	Description
Functionality	25%	Assesses if the workflow achieves the intended goal and uses the correct nodes.
Connections	15%	Evaluates if nodes are connected logically and optimally.
Expressions	15%	Checks for correct syntax and valid data path references in n8n expressions.
Node Configuration	15%	Verifies that node parameters are set correctly and required fields are populated.
Efficiency	10%	Scores redundancy, path optimization, and the number of nodes used.
Data Flow	10%	Analyzes the correctness of data transformations and flow between nodes.
Maintainability	5%	Assesses node naming, workflow organization, and modularity.
Structural Similarity	5%	(Optional) Compares the generated workflow structure to a reference workflow.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:96-140, packages/@n8n/ai-workflow-builder.ee/evaluations/chains/workflow-evaluator.ts:13-46.

Violations are categorized by severity:

Critical: Workflow-breaking issues.
Major: Significant problems affecting functionality.
Minor: Non-critical issues or inefficiencies.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:142-146

Test Cases

Test cases provide the prompts and optional reference workflows for the evaluation runs. A set of basic test cases is predefined, and new ones can be generated dynamically.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/chains/test-case-generator.ts:1-12.

Predefined Test Cases

The framework includes a set of standard test cases to cover a range of common scenarios.

ID	Name	Prompt Summary
`invoice-pipeline`	Invoice processing pipeline	Parse invoices, extract data with AI, validate, and store in Airtable.
`ai-news-digest`	Daily AI news digest	Fetch news, summarize with an LLM, and send to Telegram.
`rag-assistant`	RAG knowledge assistant	Build a RAG pipeline for uploaded documents with a chatbot interface.
`email-summary`	Summarize emails with AI	Retrieve emails, analyze with AI for action items, and send a summary to Slack.
`youtube-auto-chapters`	YouTube video chapters	Auto-generate YouTube chapter timestamps from video captions.
`google-sheets-processing`	Process large Google Sheets data	Read a large sheet, enrich each row via an external API, and update the sheet.
`extract-from-file`	Extract data from uploaded files	Accept file uploads, extract text/data, and save in a structured format.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/chains/test-case-generator.ts:89-160.

Running Evaluations

Configuration

nodes.json: A nodes.json file containing all n8n node type definitions is required in the evaluations/ root directory. It can be generated by fetching it from a running n8n instance: curl -o evaluations/nodes.json http://localhost:5678/types/nodes.json.
Environment Variables:
- N8N_AI_ANTHROPIC_KEY: Required for LLM access.
- LANGSMITH_API_KEY: Required for Langsmith evaluation.
- EVALUATION_CONCURRENCY: Number of parallel test executions (default: 5).
- GENERATE_TEST_CASES: Set to true to generate additional test cases.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:166-194

CLI Evaluation

The CLI is the primary method for local evaluation.

# Run all predefined test cases
pnpm eval

# Run a specific test case by ID
pnpm eval --test-case google-sheets-processing

# Generate additional test cases dynamically
GENERATE_TEST_CASES=true pnpm eval

# Set concurrency for parallel runs
EVALUATION_CONCURRENCY=10 pnpm eval

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:150-159

Langsmith Evaluation

To run evaluations using Langsmith, set the required environment variables and use the eval:langsmith script.

# Set required environment variables
export LANGSMITH_API_KEY=your_api_key
export LANGSMITH_DATASET_NAME=your_dataset_name

# Run evaluation
pnpm eval:langsmith

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:161-164

Resources

AI code review
Bug Bench
AI coding statistics
Coding standards
Agents
Wikis

View on GitHub

Back to directory

Repository

mrge-io/eval-n8n

Overview

Home Getting Started Contributing Guide

System Architecture

Architecture Overview Monorepo Structure Workflow Execution Engine Scaling Architecture Task Runner Architecture

Core Features

Workflows, Nodes, and Connections Credentials Management Triggers Expressions

AI & LangChain Integration

AI Workflow Builder LangChain Nodes AI Workflow Evaluation (Enterprise)

Backend Systems

Configuration Management REST API Authentication & SSO (Enterprise)Permissions & RBAC (Enterprise)Source Control (Enterprise)

Data Management

Database Schema & Migrations Binary Data Management

Frontend

Frontend Architecture Canvas Engine Design System

Deployment & Infrastructure

Deployment with Docker Health Checks

Extensibility & Customization

Creating Custom Nodes

Development & Testing

Testing Strategy

AI Workflow Evaluation (Enterprise)

Relevant source files

The following files were used as context for generating this wiki page:

packages/@n8n/ai-workflow-builder.ee/evaluations/README.md
packages/@n8n/ai-workflow-builder.ee/evaluations/chains/workflow-evaluator.ts
packages/@n8n/ai-workflow-builder.ee/evaluations/types/evaluation.ts
packages/@n8n/ai-workflow-builder.ee/evaluations/cli/runner.ts
packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/evaluator.ts
packages/@n8n/ai-workflow-builder.ee/evaluations/core/test-runner.ts
packages/@n8n/ai-workflow-builder.ee/evaluations/chains/test-case-generator.ts

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:1-5

Architecture Overview

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:7-10

The following diagram illustrates the high-level architecture of the evaluation system.

Generating architecture diagram...

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:7-10, 24-27

Directory Structure

The module's file system is organized to separate concerns, making it extensible and maintainable.

evaluations/
├── cli/                 # CLI evaluation implementation
├── langsmith/          # Langsmith integration
├── core/               # Shared evaluation logic
├── types/              # Type definitions
├── chains/             # LLM evaluation chains
│   ├── evaluators/     # Individual metric evaluators
└── utils/              # Utility functions

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:12-38

Core Components

Workflow Evaluator

This diagram shows the parallel evaluation process orchestrated by the Workflow Evaluator.

Generating architecture diagram...

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/chains/workflow-evaluator.ts:133-152, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:49-56

Individual Evaluators

Each metric category has its own specialized evaluator chain with tailored prompts and scoring logic. These evaluators run in parallel. Examples include:

functionality-evaluator.ts: Focuses on whether the workflow achieves the explicitly requested goals.
connections-evaluator.ts: Analyzes node connections and data flow paths.
data-flow-evaluator.ts: Analyzes data transformations and validation logic.
maintainability-evaluator.ts: Assesses naming, organization, and structural quality.

Test Runner

The core/test-runner.ts is responsible for the execution of a single test case. Its key responsibilities include:

Generating a workflow using the WorkflowBuilderAgent.
Validating the generated workflow structure.
Invoking the evaluateWorkflow orchestrator.
Returning a structured test result, including any errors.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:40-45, packages/@n8n/ai-workflow-builder.ee/evaluations/core/test-runner.ts

Execution Runners

CLI Runner

The cli/runner.ts orchestrates local test execution. It manages test case generation, controls concurrency for parallel test runs, and generates detailed reports in Markdown and JSON formats.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/cli/runner.ts, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:82-86

Langsmith Integration

The Langsmith integration consists of two main parts:

evaluator.ts: A Langsmith-compatible function that adapts Langsmith Run objects into the evaluation system's input format, validates data, and returns structured results.
runner.ts: Orchestrates the evaluation against a Langsmith dataset, handles dataset verification, and reports errors.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/langsmith/evaluator.ts, packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:71-80

Data Structures

The evaluation framework uses Zod schemas to define its core data structures, ensuring type safety and validation.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/types/evaluation.ts

This class diagram shows the main data schemas and their relationships.

Generating architecture diagram...

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/types/evaluation.ts:4-65

Evaluation Metrics

Workflows are evaluated across seven weighted categories. The final score is a weighted average of each category's score.

Category	Weight	Description
Functionality	25%	Assesses if the workflow achieves the intended goal and uses the correct nodes.
Connections	15%	Evaluates if nodes are connected logically and optimally.
Expressions	15%	Checks for correct syntax and valid data path references in n8n expressions.
Node Configuration	15%	Verifies that node parameters are set correctly and required fields are populated.
Efficiency	10%	Scores redundancy, path optimization, and the number of nodes used.
Data Flow	10%	Analyzes the correctness of data transformations and flow between nodes.
Maintainability	5%	Assesses node naming, workflow organization, and modularity.
Structural Similarity	5%	(Optional) Compares the generated workflow structure to a reference workflow.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:96-140, packages/@n8n/ai-workflow-builder.ee/evaluations/chains/workflow-evaluator.ts:13-46.

Violations are categorized by severity:

Critical: Workflow-breaking issues.
Major: Significant problems affecting functionality.
Minor: Non-critical issues or inefficiencies.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:142-146

Test Cases

Test cases provide the prompts and optional reference workflows for the evaluation runs. A set of basic test cases is predefined, and new ones can be generated dynamically.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/chains/test-case-generator.ts:1-12.

Predefined Test Cases

The framework includes a set of standard test cases to cover a range of common scenarios.

ID	Name	Prompt Summary
`invoice-pipeline`	Invoice processing pipeline	Parse invoices, extract data with AI, validate, and store in Airtable.
`ai-news-digest`	Daily AI news digest	Fetch news, summarize with an LLM, and send to Telegram.
`rag-assistant`	RAG knowledge assistant	Build a RAG pipeline for uploaded documents with a chatbot interface.
`email-summary`	Summarize emails with AI	Retrieve emails, analyze with AI for action items, and send a summary to Slack.
`youtube-auto-chapters`	YouTube video chapters	Auto-generate YouTube chapter timestamps from video captions.
`google-sheets-processing`	Process large Google Sheets data	Read a large sheet, enrich each row via an external API, and update the sheet.
`extract-from-file`	Extract data from uploaded files	Accept file uploads, extract text/data, and save in a structured format.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/chains/test-case-generator.ts:89-160.

Running Evaluations

Configuration

nodes.json: A nodes.json file containing all n8n node type definitions is required in the evaluations/ root directory. It can be generated by fetching it from a running n8n instance: curl -o evaluations/nodes.json http://localhost:5678/types/nodes.json.
Environment Variables:
- N8N_AI_ANTHROPIC_KEY: Required for LLM access.
- LANGSMITH_API_KEY: Required for Langsmith evaluation.
- EVALUATION_CONCURRENCY: Number of parallel test executions (default: 5).
- GENERATE_TEST_CASES: Set to true to generate additional test cases.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:166-194

CLI Evaluation

The CLI is the primary method for local evaluation.

# Run all predefined test cases
pnpm eval

# Run a specific test case by ID
pnpm eval --test-case google-sheets-processing

# Generate additional test cases dynamically
GENERATE_TEST_CASES=true pnpm eval

# Set concurrency for parallel runs
EVALUATION_CONCURRENCY=10 pnpm eval

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:150-159

Langsmith Evaluation

To run evaluations using Langsmith, set the required environment variables and use the eval:langsmith script.

# Set required environment variables
export LANGSMITH_API_KEY=your_api_key
export LANGSMITH_DATASET_NAME=your_dataset_name

# Run evaluation
pnpm eval:langsmith

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/README.md:161-164

Resources

AI code review
Bug Bench
AI coding statistics
Coding standards
Agents
Wikis

View on GitHub