Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• CLAUDE.md
• AGENTS.md
• lib/files/tests/getKnowledgeBaseText.test.ts
• lib/chat/toolChains/tests/toolChains.test.ts
• lib/agents/generalAgent/tests/getGeneralAgent.test.ts
• evals/spotify-tools.eval.ts
• evals/social-scraping.eval.ts
• lib/chats/tests/generateChatTitle.test.ts

The Testing Strategy for the Recoup API is built on a foundation of Test-Driven Development (TDD), emphasizing the creation of tests before implementation to ensure system reliability and adherence to business requirements. The strategy spans multiple layers, including unit testing for business logic, integration testing for handlers, and specialized LLM evaluations for AI agents and tools.

The core objective is to maintain a high-quality codebase where every exported function, API endpoint, and AI toolchain is verified through automated test suites. This ensures that new features do not introduce regressions and that complex logic—such as AI agent routing and social media scraping—functions as expected across various scenarios.
Sources: CLAUDE.md:144-150, AGENTS.md:144-150

Test-Driven Development (TDD) Workflow

The project mandates a strict TDD workflow for all developers and AI coding agents. This process ensures that requirements are clearly defined and verified through code before any feature implementation or bug fix occurs.

The Five-Step TDD Process

1. Write Failing Tests: Create tests in the domain's __tests__ directory describing the expected behavior.
2. Verify Failure: Run the tests to confirm they fail as expected.
3. Implement Minimum Code: Write only the code necessary to pass the tests.
4. Verify Success: Run the tests again to ensure all tests pass.
5. Refactor: Clean up the code while maintaining passing test status.
   Sources: CLAUDE.md:152-162, AGENTS.md:152-162

TDD Execution Flow

The following diagram illustrates the standard cycle for implementing a new feature or fix within the repository.

Testing Infrastructure and Tooling

The project utilizes a modern testing stack centered around Vitest for unit/integration testing and Braintrust for AI evaluations.

Core Testing Commands

Command	Description
pnpm test	Runs the full Vitest test suite.
pnpm test:watch	Runs Vitest in watch mode for active development.
npx braintrust eval <path>	Runs specific LLM evaluation files.
pnpm lint:check	Verifies code style and potential errors.
Sources: CLAUDE.md:31-36, evals/spotify-tools.eval.ts:27

File Organization

Tests are co-located with the source code they verify within __tests__ directories.

• Business Logic: lib/[domain]/__tests__/[filename].test.ts
• API Handlers: lib/[domain]/__tests__/[handler].test.ts
• AI Evaluations: evals/[feature].eval.ts
  Sources: CLAUDE.md:164-171, AGENTS.md:164-171, evals/social-scraping.eval.ts:1

Unit and Integration Testing Patterns

The codebase relies heavily on mocking dependencies to isolate the logic being tested. This is particularly critical for database operations, networking, and AI model calls.

Mocking Strategy

Developers use vi.mock and vi.stubGlobal to replace external services:

• Database: Mocks for @/lib/supabase/* to simulate query returns.
• Networking: Mocking global.fetch for external API calls and getCorsHeaders for API responses.
• AI Models: Mocking generateText to return predictable strings for title generation or routing.
  Sources: lib/files/tests/getKnowledgeBaseText.test.ts:7-17, lib/chats/tests/generateChatTitle.test.ts:7-10, lib/agents/generalAgent/tests/getGeneralAgent.test.ts:13-52

Example Integration Test Structure: API Handler

A standard handler test verifies both successful processing and error handling (e.g., 400 validation errors or 500 server errors).

AI and Agent Evaluations (Evals)

Because the project relies on Large Language Models (LLMs) for complex tasks like artist management and social scraping, traditional unit tests are supplemented with "Evals."

Specialized Scorer Logic

Evals use specific metrics to determine success:

• Tools Called Scorer: Verifies that the AI used the correct set of tools (e.g., get_spotify_search, scrape_instagram_profile) for a given prompt.
• Penalized Tools: Deducts points if the AI uses "lazy" tools (like contact_team) instead of attempting to solve the problem with available scrapers.
  Sources: evals/spotify-tools.eval.ts:29-35, evals/social-scraping.eval.ts:32-41

Evaluation Categories

Category	Required Tools	Objective
Spotify Tools	get_spotify_search, get_spotify_artist_albums	Ensure pitches are data-driven rather than generic.
Social Scraping	scrape_instagram_profile, get_apify_scraper	Force the use of fresh scraped data over stale DB values.
Tool Chains	update_account_info, generate_txt_file	Verify multi-step sequences for artist onboarding.
Sources: evals/spotify-tools.eval.ts:37-45, evals/social-scraping.eval.ts:43-51, lib/chat/toolChains/tests/toolChains.test.ts:54-95

Verification Areas

The Testing Strategy defines specific requirements for different parts of the system:

1. Input Validation Testing

Tests must cover all valid and invalid input combinations for Zod schemas, specifically targeting:

• Missing required fields (should return 400).
• Incorrect UUID formats.
• String length constraints (e.g., chat titles under 20 characters).
  Sources: CLAUDE.md:202-205, lib/chats/tests/generateChatTitle.test.ts:108-115

2. Authentication Context

Tests verify that accountId is correctly derived from either x-api-key or Authorization: Bearer tokens using validateAuthContext(). Manual extraction of accountId from request bodies is strictly tested against to prevent impersonation.
Sources: CLAUDE.md:231-250, AGENTS.md:231-250

3. File Processing

Logic for handling knowledge base files is tested for:

• Supported MIME types (text/plain, text/markdown, application/json, text/csv).
• Excluded types (images, PDFs, audio).
• Handling of failed network fetches.
  Sources: lib/files/tests/getKnowledgeBaseText.test.ts:38-95

4. Git and Deployment Safety

While not code-based, the testing strategy includes manual verification rules:

• Never push to main or test branches.
• Open PRs against the test branch.
• Verify Linting: Run pnpm lint before every commit.
  Sources: CLAUDE.md:8-16, AGENTS.md:8-16

Summary

The Testing Strategy is an exhaustive framework requiring 100% coverage of business logic via Vitest and specialized performance monitoring for AI via Braintrust. By co-locating tests with code and enforcing a "test-first" culture, the project maintains high velocity without compromising the integrity of its music industry AI operations.