Evaluation Framework

Relevant source files

The following files were used as context for generating this wiki page:

• evals/README.md
• evals/social-scraping.eval.ts
• evals/spotify-tools.eval.ts
• evals/memory-tools.eval.ts
• evals/first-week-album-sales.eval.ts
• evals/tiktok-analytics-questions.eval.ts
• evals/monthly-listeners-tracking.eval.ts
• package.json

The Evaluation Framework in this project is a Braintrust-based system designed to test and improve the performance of AI agents. It ensures that the AI properly utilizes available tools—such as social scrapers, Spotify data fetchers, and memory storage utilities—rather than defaulting to generic responses or utilizing stale data.

The framework allows developers to define test cases with specific inputs, expected outputs, and metadata. It utilizes scorers like Factuality, AnswerSimilarity, and custom tool-usage scorers to provide quantitative measurements of AI accuracy and reliability.

Sources: evals/README.md:1-3, evals/social-scraping.eval.ts:7-20, evals/spotify-tools.eval.ts:7-14

Core Components and Workflow

The framework operates by running automated scripts that simulate user queries and validate the AI's response against predefined expectations. The process involves calling chat functions, extracting results, and applying scoring metrics.

High-Level Architecture

The following diagram illustrates the flow from an evaluation trigger to the final scored result.

This diagram shows the execution flow of an evaluation test case from initialization to reporting.
Sources: evals/README.md:16-24, evals/social-scraping.eval.ts:117-133

Key Functions and Utilities

Function	Source File	Description
Eval	braintrust	The primary wrapper for defining an evaluation suite.
callChatFunctions	lib/evals	Invokes the AI system with a text input and returns the generated response.
callChatFunctionsWithResult	lib/evals	Similar to callChatFunctions but returns structured tool call data.
extractTextFromResult	lib/evals	Parses the raw AI response to extract the text output for scoring.
createToolsCalledScorer	lib/evals	A custom scorer that rewards the use of required tools and penalizes prohibited ones.

Sources: evals/social-scraping.eval.ts:2-6, evals/first-week-album-sales.eval.ts:3-5, evals/spotify-tools.eval.ts:2-6

Scoring Mechanisms

The framework uses a variety of scorers to evaluate different aspects of AI performance:

• Factuality: Measures the factual accuracy of the AI's response against an expected ground truth. This is primarily used for objective data points like album sales.
• AnswerSimilarity: Evaluates how closely the AI's natural language response matches the expected output format.
• Tool Usage Scoring: Specifically checks if the AI called the correct Model Context Protocol (MCP) tools. For example, in social scraping tests, the AI is expected to use search_web or scrape_instagram_profile instead of contact_team.
• QuestionAnswered: A specialized scorer for analytics-focused queries to ensure the AI provided a concrete answer rather than an explanation of why it couldn't find one.

Sources: evals/first-week-album-sales.eval.ts:4, 75, evals/monthly-listeners-tracking.eval.ts:2, 47, evals/social-scraping.eval.ts:135, evals/tiktok-analytics-questions.eval.ts:3, 52

Evaluation Domains

The framework is segmented into different domains to test specific capabilities:

Social Media and Scraping

Tests if the AI can retrieve fresh data from social platforms. It penalizes the use of contact_team when automated scrapers (e.g., Apify, Instagram/Twitter scrapers) are available.

• Required Tools: search_web, get_apify_scraper, scrape_instagram_profile, search_twitter.
• Metrics: Follower counts, song usage in Reels, and press moments.

Sources: evals/social-scraping.eval.ts:22-35

Industry Data and Tools

Validates the AI's ability to fetch music-specific data such as Spotify analytics or historical album sales.

• Spotify Tools: Focuses on crafting pitches and researching playlist curators using get_spotify_search and get_spotify_artist_albums.
• Album Sales: Uses Factuality to check data on first-week sales for artists like Taylor Swift or Drake.

Sources: evals/spotify-tools.eval.ts:20-26, evals/first-week-album-sales.eval.ts:15-58

Memory and Persistence

Ensures the AI saves information to persistent storage rather than temporary generation tools.

• Required Tools: write_file, create_knowledge_base.
• Penalized Tool: generate_txt_file (when used alone as it lacks persistence).

Sources: evals/memory-tools.eval.ts:7-16

Execution Commands

The evaluation system is integrated into the project's build and test scripts via package.json.

# Run all evaluations in the system
pnpm eval

# Run a specific evaluation file
pnpm eval evals/social-scraping.eval.ts

# Direct execution using Braintrust CLI with required external packages
npx braintrust eval --external-packages playwright playwright-core chromium-bidi @browserbasehq/stagehand

Sources: evals/README.md:16-24, package.json:14

Summary

The Evaluation Framework serves as a critical quality assurance layer for the Recoup AI system. By leveraging Braintrust and custom tool-based scoring, the project maintains a high standard for data accuracy and tool integration, ensuring the AI behaves as a strategic assistant rather than a simple chatbot.