Agent A/B Testing
Relevant source files
The following files were used as context for generating this wiki page:
Agent A/B Testing is a core component of the Flow platform's continuous improvement loop. It allows developers and researchers to compare two different agent configurations—typically a baseline "Active" version and a "Candidate" version proposed by the Prompt Rewriter—against a standardized set of test cases known as a Golden Set. This system ensures that updates to agent system prompts, tools, or LLM configurations actually result in measurable performance improvements before they are promoted to production status.
The testing process is automated through background jobs and provides detailed scoring and rationales provided by an LLM judge. By evaluating agents on the same input data, the system determines a "winner" based on average score deltas, enabling data-driven decisions for agent versioning lifecycle management.
Sources: README.md:144-154, README.md:157-170
Continuous Improvement Architecture
The A/B testing system sits at the end of the self-improvement loop. When an agent fails to meet a specific pass rate (typically < 0.7) on a Golden Set, the system triggers a rewrite. The resulting candidate is then pitted against the current baseline.
The A/B test runner executes both genome versions against the same golden items to ensure parity. A candidate only proceeds to the proposal stage if it significantly outperforms the baseline.
Sources: README.md:144-154, README.md:157-160
Data Model and Schemas
The A/B testing infrastructure relies on structured data to track agent versions, test results, and aggregate performance metrics.
Test Result Structures
The frontend and backend communicate using specific data structures to represent the outcome of a comparative test.
| Field | Type | Description |
|---|---|---|
agent_label | "A" | "B" | Identifies which version of the agent produced the result. |
score | number (0.0-1.0) | The quality score assigned by the LLM judge. |
rationale | string | Explanation for the assigned score. |
actual_output | string | The raw response generated by the agent. |
winner | "A" | "B" | The version with the higher average score in the aggregate. |
Sources: apps/web/src/app/(app)/agents/ab-test/[id]/page.tsx:14-35, services/api/flow/interfaces/http/routes/golden_sets.py:276-291
Golden Sets
Golden Sets are curated test cases consisting of an input_text, an expected_output, and specific scoring_criteria. These sets serve as the ground truth for both individual evaluations and A/B comparisons.
Evaluation and Scoring Logic
The evaluation utilizes an LLM judge (typically gpt-4o-mini) to grade agent responses. The judge reviews the agent's output against the expected_output and scoring_criteria defined in the Golden Item.
Scoring Process
- Asynchronous Execution: Tests are triggered asynchronously via API endpoints, often handled by background tasks.
- LLM Judging: For each item, the judge returns a JSON object containing a score (0.0–1.0) and a rationale.
- Aggregation: Scores are averaged across the entire set. A pass rate is calculated based on the percentage of items scoring ≥ 0.7.
Example Scoring Criteria
| Agent Type | Scoring Focus |
|---|---|
| Research Analyst | Citation accuracy, structured findings, and identified limitations. |
| Code Review Agent | Detection of critical vulnerabilities (e.g., SQL injection) and providing correct fixes. |
| Bug Triage | Correct severity classification (P0-P3) and root cause identification. |
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:164-175, services/api/scripts/seed_agents_and_datasets.py:306-315, README.md:175-179
User Interface and Results Visualization
The web interface provides a dedicated dashboard for reviewing A/B test results, located at /agents/ab-test/[id].
Key UI Components
- Scorecards: Visual representation of Agent A vs. Agent B average scores (scaled to 10 for readability).
- Winner Banner: Highlights the winning agent and the performance delta.
- Per-Item Results Table: A side-by-side comparison of how each agent handled specific inputs, including the judge's rationale for different scores.
- Status Badges: Indicates if a test is
running,completed, orfailed.
Lifecycle and Promotion
The final stage of the A/B testing process is the promotion of a candidate genome.
- Genome Snapshotting: Every configuration change or rewrite snapshots a new genome version.
- Status Transitions: Versions move from
CANDIDATEtoACTIVE. The previous version is automatically moved toARCHIVED. - Human-in-the-loop: While the A/B test is automated, the promotion of a
CANDIDATEtypically requires human approval via a generated Proposal.
Sources: README.md:131-142, README.md:152-154, AGENTS.md:27-29
The Agent A/B Testing module provides a robust framework for validating agent improvements. By leveraging LLM-based judging and standardized Golden Sets, it ensures that the "Flow" platform evolves with high technical accuracy and measurable quality gains.