Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
- services/api/flow/interfaces/http/routes/golden_sets.py
- services/api/scripts/seed_agents_and_datasets.py
- README.md
- services/api/scripts/seed_skills.py
- apps/web/src/app/(app)/knowledge/page.tsx
- apps/web/vitest.config.ts
- services/api/tests/test_full_improvement_loop.py
- services/api/tests/test_deer_graph_offline.py
The Flow project employs a multi-layered testing strategy that combines traditional unit and integration testing with an advanced "Continuous Improvement Loop" powered by LLM-based evaluation. The strategy is designed to ensure the reliability of both the core infrastructure (FastAPI, Next.js) and the non-deterministic outputs of AI agents.
Key components of the strategy include Golden Set Evaluations, where curated test cases compare agent performance against expected outputs, and an automated feedback mechanism that utilizes user ratings and "agent negatives" to refine retrieval and prompt effectiveness.
Core Testing Frameworks
The project utilizes standardized testing tools across the monorepo to handle both backend and frontend verification.
- Backend (Python): Uses
pytestfor unit and offline tests. Offline tests are specifically designed to run without a database connection. Sources: README.md:214-218 - Frontend (TypeScript): Configured with
vitestfor component and logic testing within the Next.js application. Sources: apps/web/vitest.config.ts - API Testing: Includes full-loop tests that simulate the improvement cycle, verifying how agent genomes evolve based on evaluation results. Sources: services/api/tests/test_full_improvement_loop.py
Golden Set Evaluation System
The primary mechanism for measuring agent quality is the Golden Set system. A "Golden Set" consists of a collection of items, each containing a specific input, an expected output, and scoring criteria.
Evaluation Workflow
The system evaluates an agent's current performance against these benchmarks using an LLM judge (typically gpt-4o-mini).
The LLM judge produces a structured response containing a score (0.0–1.0) and a rationale. Sources: README.md:144-150, services/api/flow/interfaces/http/routes/golden_sets.py:276-291
Key Data Structures
| Field | Description |
|---|---|
input_text | The prompt or data provided to the agent. |
expected_output | The target response or format the agent should achieve. |
scoring_criteria | Technical rubrics for the LLM judge (e.g., "Must cite at least 2 sources"). |
score | A numerical value between 0.0 and 1.0 assigned by the judge. |
grading_rationale | Explanation provided by the judge for the assigned score. |
Sources: services/api/scripts/seed_agents_and_datasets.py:195-208, services/api/flow/interfaces/http/routes/golden_sets.py:325-335
Continuous Improvement Loop
A critical part of the testing strategy is the automated self-improvement cycle, which runs as a nightly cron job (auto_eval_tick).
Prompt Rewriting:* If the pass rate falls below 70%, the PromptRewriter analyzes failure rationales to propose targeted edits. Sources: README.md:129-142
- A/B Testing: The
ABRunnercompares the performance of the existing "Active" agent version against the "Candidate" to ensure new versions provide significant improvements. Sources: README.md:136-140
User Feedback Integration
The testing and improvement strategy extends into production via user interaction.
- Feedback Scores: Users rate answers on a slider (0–100%). Scores below 50% automatically generate "agent negatives." Sources: README.md:195-198
- Retrieval Grader: These negatives are used as few-shot examples in the retrieval pipeline to prevent the agent from repeating low-quality document retrievals. Sources: README.md:198-200, apps/web/src/app/(app)/knowledge/page.tsx:173-180
Integration and Smoke Testing
Infrastructure-level reliability is verified through specific scripts and health checks.
- Seeding Scripts: Scripts like
seed_skills.pyandseed_agents_and_datasets.pyserve as functional tests for the database schema and agent configuration logic. Sources: services/api/scripts/seed_skills.py:17-18, services/api/scripts/seed_agents_and_datasets.py:15-20 - Agentic RAG Verification: When enabled, retrieval logic is tested through a sub-pipeline involving query rewriting and LLM grading. Sources: README.md:231-236
Testing outcomes are monitored through the FlowCallbackHandler, which tracks latency and structured events (llm.start, tool.end, chain.error) to identify performance regressions. Sources: README.md:183-191