Golden Sets & Evaluations
Relevant source files
The following files were used as context for generating this wiki page:
Golden Sets and Evaluations form the core of the Flow project's continuous improvement loop. This system allows developers and users to benchmark agents against curated test cases, ensuring quality stability and providing data for automated prompt optimization. Golden Sets consist of specific inputs and expected outputs that an LLM judge uses to score agent performance.
Sources: README.md:16-18, README.md:104-106
Architecture and Data Model
The evaluation system is built around three primary entities: Golden Sets, Golden Items, and Golden Results. A Golden Set serves as a container for multiple Golden Items (test cases). When an evaluation is executed, the results for each item are persisted as Golden Results.
The data model supports historical tracking by grouping individual results under an eval_run_id.
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:236-250, apps/web/src/app/(app)/evals/page.tsx:21-25
Key Data Structures
| Entity | Description | Key Fields |
|---|---|---|
| Golden Set | A collection of test cases for a specific domain or agent type. | id, name, description, workspace_id |
| Golden Item | A single test case within a set. | input_text, expected_output, scoring_criteria |
| Golden Result | The outcome of a single item evaluation. | score (0.0-1.0), grading_rationale, actual_output |
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:22-60, apps/web/src/app/(app)/evals/page.tsx:21-25
Evaluation Workflow
Evaluations can be triggered manually via the UI or automatically through a nightly cron job. The process involves sending the Golden Item's input_text to the target agent and then using an LLM judge (typically gpt-4o-mini) to compare the agent's response against the expected_output and scoring_criteria.
The evaluation results are streamed to the frontend using Server-Sent Events (SSE) to provide real-time logging of the evaluation progress.
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:292-310, apps/web/src/app/(app)/evals/page.tsx:149-182, README.md:104-106
Continuous Improvement Loop
Evaluations are a prerequisite for the "Auto-Improve" feature. If an agent's pass rate falls below 0.7 (70%), the system can trigger a Prompt Rewriter that analyzes failures and proposes a new system prompt.
The system tracks metrics such as:
- Pass Rate: Percentage of items with a score $\ge$ 0.7.
- Average Score: The arithmetic mean of all item scores in a run.
- Regression Detection: Comparison between current run results and previous historical data.
Sources: README.md:85-95, apps/web/src/app/(app)/evals/page.tsx:327-347, services/api/flow/interfaces/http/routes/golden_sets.py:255-275
API Endpoints
The system provides a comprehensive REST API for managing Golden Sets and retrieving evaluation history.
| Method | Endpoint | Description |
|---|---|---|
POST | /api/v1/golden-sets | Create a new Golden Set. |
GET | /api/v1/golden-sets | List all Golden Sets in the workspace. |
POST | /api/v1/golden-sets/{set_id}/items | Add a test case to a set. |
POST | /api/v1/golden-sets/{set_id}/evaluate | Trigger an asynchronous evaluation run. |
GET | /api/v1/golden-sets/{set_id}/results | Retrieve detailed results for a specific set. |
GET | /api/v1/golden-sets/{set_id}/history | Get historical performance for regression tracking. |
POST | /api/v1/golden-sets/seed-samples | Seed predefined datasets (Research, Code Review, etc.). |
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:126-311
Sample Datasets
Flow includes several "canonical" datasets used for seeding environments and demonstrating evaluation capabilities. These are defined in the backend and can be deployed via the /seed-samples endpoint.
- Research Analyst: Tests ability to synthesize academic sources and structure reports.
- Code Review Agent: Tests bug detection and security analysis (e.g., SQL injection).
- Lucis Health Protocol: Specialized sets for testing sleep assessment and mental health screening using non-diagnostic language.
- Data Analyst: Tests statistical reasoning and Python code execution quality.
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:22-120, services/api/scripts/seed_agents_and_datasets.py:192-205
The Golden Sets & Evaluations module provides the empirical foundation for agent development in Flow, transforming qualitative LLM outputs into quantitative metrics that drive automated and human-in-the-loop improvements.