AI Workflow Evaluations

Relevant source files

The following files were used as context for generating this wiki page:

• packages/@n8n/ai-workflow-builder.ee/evaluations/tests/output.test.ts
• packages/@n8n/ai-workflow-builder.ee/src/subgraphs/test/integration/techniques.json
• packages/@n8n/ai-workflow-builder.ee/src/code-builder/prompts/index.ts
• packages/@n8n/ai-workflow-builder.ee/src/prompts/agents/discovery.prompt.ts
• packages/cli/src/modules/mcp/tools/workflow-builder/mcp-instructions.ts

AI Workflow Evaluations provide a framework for testing and measuring the accuracy of workflows generated by AI agents. This system compares generated workflow graphs against expected techniques and validates their structural integrity. It captures performance metrics, functionality scores, and detailed feedback to ensure AI-generated automations meet technical standards.

The evaluation process involves two primary agents: the Discovery Agent, which identifies necessary nodes and techniques, and the Code Builder Agent, which generates the executable JavaScript SDK code. Systematic validation follows generation to catch parameter mismatches or logic errors before the workflow is saved.

Sources: packages/@n8n/ai-workflow-builder.ee/src/prompts/agents/discovery.prompt.ts, packages/@n8n/ai-workflow-builder.ee/src/code-builder/prompts/index.ts

Evaluation Architecture and Data Flow

The evaluation harness executes test cases defined as prompts and expected techniques. The process follows a mandatory sequence from requirement analysis to final artifact saving.

Mandatory Evaluation Steps

1. Requirement Analysis: The Discovery Agent extracts services and technologies from the user prompt.
2. Technique Identification: The system maps the prompt to categories such as chatbot, triage, or document_processing.
3. Node Discovery: The agent calls search_nodes to find specific n8n node types and versions.
4. Code Generation: The Code Builder Agent generates complete workflow code in JavaScript format.
5. Validation: The system invokes validate_workflow to check for parameter errors and schema violations.
6. Artifact Persistence: The ArtifactSaver writes the final results, workflow JSON, and feedback to the filesystem.

Sources: packages/@n8n/ai-workflow-builder.ee/src/code-builder/prompts/index.ts:380-450, packages/cli/src/modules/mcp/tools/workflow-builder/mcp-instructions.ts:80-120

Workflow Generation Sequence

This diagram illustrates the flow from a user prompt through agent processing to the final validated workflow artifact.

The sequence shows the synchronous calls between agents and validation tools during a single evaluation run.

Artifact Management and Metrics

The evaluation system generates structured artifacts for every test case. These artifacts allow developers to inspect failures and track agent improvements over time.

Captured Artifacts

Artifact	Format	Description
Prompt	prompt.txt	The original user request.
Workflow	workflow.json	The generated n8n-importable workflow graph.
Feedback	feedback.json	Detailed scores and metrics from evaluators.
Error	error.txt	Detailed stack trace or error message if generation failed.
Summary	summary.json	Aggregated pass rates and duration averages for the entire run.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/tests/output.test.ts:160-250

Evaluation Metrics

The system captures specific data points in feedback.json to measure agent efficiency:

• Subgraph Metrics: Includes nodeCount, discoveryDurationMs, builderDurationMs, and responderDurationMs.
• Evaluator Scores: Captured from different sources like llm-judge (functionality, connections) and programmatic (overall score).
• Evaluator Averages: The average score across all metrics for a specific evaluator.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/tests/output.test.ts:30-100

Technique Classification

Evaluations verify if the AI correctly identifies the intended workflow technique. Test cases in techniques.json provide the ground truth for this classification.

Core Workflow Techniques

• Chatbot: Receiving and replying to chat messages (e.g., Telegram, Slack).
• Scraping and Research: Collecting live information from external APIs or websites.
• Triage: Classifying and routing data for prioritization.
• Content Generation: Creating text, images, or video.
• Document Processing: Extracting content from PDFs, spreadsheets, or images.
• Data Transformation: Cleaning or restructuring JSON data.

Sources: packages/@n8n/ai-workflow-builder.ee/src/subgraphs/test/integration/techniques.json:1-100, packages/@n8n/ai-workflow-builder.ee/src/prompts/agents/discovery.prompt.ts:40-60

Evaluation Result Schema

This diagram defines the relationships between evaluation results, individual examples, and feedback metrics.

The diagram represents the data structure found in the ParsedSummary and ExampleResult interfaces used for artifact saving.

Sources: packages/@n8n/ai-workflow-builder.ee/evaluations/tests/output.test.ts:16-55

Validation and Error Handling

Validation is a critical step in the evaluation cycle. The validate_workflow tool checks the generated JavaScript against the n8n SDK and type definitions.

Validation Scope

• Parameter Names: Ensures parameter names match the exact node type definitions.
• Types: Checks for type mismatches (e.g., comparing a string to a number).
• Discriminators: Verifies that required resource, operation, or mode discriminators are present.
• SDK Syntax: Ensures expr() syntax and node variables are valid.

If validation fails, the system provides a detailed error signal. The ArtifactSaver records these failures in error.txt, which is excluded from the summary averages to maintain score integrity.

Sources: packages/@n8n/cli/src/modules/mcp/tools/workflow-builder/mcp-instructions.ts:100-110, packages/@n8n/ai-workflow-builder.ee/src/code-builder/prompts/index.ts:440-450

Conclusion

AI Workflow Evaluations ensure the reliability of n8n's AI-assisted building experience. By quantifying performance through durational metrics and scoring generated workflows against technical requirements, the system provides a continuous feedback loop for agent development. The generation of structured artifacts facilitates deep inspection of failures, leading to more robust automated workflow generation.