Agent Evaluation Framework

Relevant source files

The following files were used as context for generating this wiki page:

• src/llm/agent-evals/hubspot-agent/index.spec.ts
• src/llm/agent-evals/jira-agent/index.spec.ts
• src/llm/agent-evals/slack-agent/index.spec.ts
• src/llm/agent-evals/common/types.ts
• src/llm/agent-evals/jira-agent/test-data.ts
• src/llm/quix-agent.ts
• src/llm/agent-evals/hubspot-agent/test-data.ts

The Agent Evaluation Framework is a testing system designed to validate the behavior of the QuixAgent. It uses the Jest testing framework to run a suite of predefined scenarios against specific agent integrations like Jira, HubSpot, and Slack. The core purpose is to ensure that the agent, when interacting with a real Large Language Model (LLM), correctly interprets user queries, selects the appropriate tools, and formulates the correct tool calls with the expected arguments.

This framework operates by using a real LLM (e.g., OpenAI) but provides it with mocked tool implementations. This isolates the agent's reasoning and planning capabilities for evaluation. Each test run's detailed results, including the agent's plan, trajectory, and evaluation score, are logged to a test-results.json file for analysis.

Sources: src/llm/agent-evals/jira-agent/index.spec.ts:20-23, 49-52, src/llm/agent-evals/hubspot-agent/index.spec.ts:19-22, 44-47

Architecture and Core Components

The framework is composed of several key components that work together to execute and evaluate test cases for different agent integrations.

This diagram illustrates the flow of a single agent evaluation test, from setup by the Jest runner to the final reporting of results.

Sources: src/llm/agent-evals/jira-agent/index.spec.ts, src/llm/agent-evals/hubspot-agent/index.spec.ts

Key Components

Component	Description	Source Files
QuixAgent	The central agent being tested. Its processWithTools method is the entry point for the evaluation.	src/llm/quix-agent.ts:18
Mocked Tools	For each integration (e.g., Jira, HubSpot), a set of mocked tools is created. These tools return predefined data from the testCase instead of making real API calls.	src/llm/agent-evals/jira-agent/index.spec.ts:70
LLM Provider	An actual LLM instance, typically from OpenAI, used by the agent for reasoning. The API key is sourced from environment variables.	src/llm/agent-evals/jira-agent/index.spec.ts:38
createTrajectoryMatchEvaluator	A utility from the agentevals library that compares the agent's actual tool call trajectory with the expected reference tool calls defined in the test case.	src/llm/agent-evals/jira-agent/index.spec.ts:40-48
testCases	An array of TestCase objects, each defining a specific scenario with user input, chat history, expected tool calls, and mock responses.	src/llm/agent-evals/jira-agent/test-data.ts:4
TestRunDetail	A data structure used to log the complete details of each test run, including the inputs, agent plan, trajectory, and evaluation result.	src/llm/agent-evals/common/types.ts:20

Test Data Structure

The evaluation process is driven by a standardized data structure, TestCase, which defines each test scenario.

Sources: src/llm/agent-evals/common/types.ts:4-18, src/llm/agent-evals/jira-agent/test-data.ts

The TestCase Interface

This interface defines the inputs and expected outputs for a single evaluation.

Field	Type	Description
description	string	A human-readable description of the test scenario.
chat_history	LLMContext[]	An array of previous messages to provide context to the agent.
invocation	object	Contains the user's name (initiator_name) and the message (message) that triggers the agent.
reference_tool_calls	object[]	An array of expected tool calls, including the tool name and arguments. This is the ground truth for evaluation.
expected_response	string	The expected final natural language response from the agent.
tool_mock_response_overrides	object	A map of tool names to their mock return values for this specific test case.

Sources: src/llm/agent-evals/common/types.ts:4-18

Example TestCase

This example from the Jira agent tests demonstrates creating a bug report from a user message.

// From: src/llm/agent-evals/jira-agent/test-data.ts
export const testCases: TestCase<ToolResponseTypeMap>[] = [
  // ...
  {
    description: 'Create a basic JIRA issue (bug) with minimal fields in project FEAT.',
    chat_history: [],
    invocation: {
      initiator_name: 'Eve',
      message: '@Quix Create a bug in the FEAT project about the login button not working'
    },
    reference_tool_calls: [
      {
        name: 'get_jira_issue_types',
        arguments: {
          projectKey: 'FEAT'
        }
      },
      {
        name: 'create_jira_issue',
        arguments: {
          projectKey: 'FEAT',
          summary: 'Login button not working',
          issueTypeId: '10001'
        }
      }
    ],
    expected_response:
      "Okay, I've created a new bug [NEW_ISSUE_KEY] 'Login button not working' in project FEAT. The description is: 'The login button on the main page is unresponsive when clicked.' It's currently in 'To Do' status.",
    tool_mock_response_overrides: {
      create_jira_issue: {
        issueKey: 'FEAT-104',
        // ...
      }
    }
  },
  // ...
];

Sources: src/llm/agent-evals/jira-agent/test-data.ts:100-135

Evaluation Logic and Execution Flow

The core of the evaluation is comparing the agent's actions against the reference_tool_calls. This is handled by the createTrajectoryMatchEvaluator.

Evaluator Configuration

The evaluator is configured to define how strictly it matches the agent's output to the reference.

// From: src/llm/agent-evals/jira-agent/index.spec.ts:40-48
evaluator = createTrajectoryMatchEvaluator({
  trajectoryMatchMode: 'superset',
  toolArgsMatchMode: 'superset',
  toolArgsMatchOverrides: {
    find_jira_ticket: (a, b) => {
      return normalize(a.jql_query as string) === normalize(b.jql_query as string);
    }
  }
});

• trajectoryMatchMode: 'superset': The agent's tool calls must be a superset of the reference calls. This allows the agent to make extra, reasonable calls (e.g., fetching context) without failing the test.
• toolArgsMatchMode: 'superset': The arguments for each tool call must be a superset of the reference arguments.
• toolArgsMatchOverrides: This provides custom logic for comparing arguments for a specific tool. In the Jira agent test, it uses a normalize function to compare JQL queries, ignoring whitespace and quote differences. This makes the test more robust to minor LLM output variations.

Sources: src/llm/agent-evals/jira-agent/index.spec.ts:40-48, src/llm/agent-evals/slack-agent/index.spec.ts:38-47

Test Execution Sequence

The following diagram shows the interactions during a single test case run.

This sequence shows how the test orchestrates the agent's execution, captures its output, and uses the evaluator to verify correctness against the test case's ground truth.

Sources: src/llm/agent-evals/jira-agent/index.spec.ts:79-142, src/llm/quix-agent.ts:22-79

Reporting

After all tests for a suite are complete, the afterAll hook collects all the TestRunDetail objects and writes them to a JSON file (e.g., test-results.json). This file provides a comprehensive log for debugging and analyzing agent performance across all scenarios.

// From: src/llm/agent-evals/hubspot-agent/index.spec.ts:44-48
afterAll(() => {
  const outputPath = path.join(__dirname, 'test-results.json');
  fs.writeFileSync(outputPath, JSON.stringify(allTestRunDetails, null, 2));
  Logger.log(`Test results written to ${outputPath}`);
});

The TestRunDetail object captures everything needed to understand a test run.

Field	Type	Description
description	string	The description from the TestCase.
stepCompleted	string	The final step the agent reached ('tool_selection' or 'agent_execution').
agentPlan	PlanResult['steps']	The step-by-step plan generated by the agent.
agentTrajectory	object[]	The sequence of messages and tool calls during execution.
expectedToolCalls	AIMessage[]	The reference tool calls formatted as AIMessage objects.
evaluationResult	object	The final score and details from the evaluator.

Sources: src/llm/agent-evals/common/types.ts:20-31, src/llm/agent-evals/hubspot-agent/index.spec.ts:44-48

Summary

The Agent Evaluation Framework is a critical tool for ensuring the quality and reliability of the QuixAgent. By testing the agent's reasoning against a variety of integrations and scenarios, it provides confidence that the agent can correctly understand user intent and interact with its available tools. The use of mocked tools isolates the LLM's planning and function-calling abilities, while the flexible TrajectoryMatchEvaluator allows for robust and meaningful validation. The detailed JSON reports serve as an invaluable asset for debugging failures and tracking performance improvements over time.