Continuous Improvement Loop
Relevant source files
The following files were used as context for generating this wiki page:
The Continuous Improvement Loop in Flow is an automated self-optimization system designed to enhance agent performance through iterative evaluation, failure analysis, and configuration refinement. It leverages "Golden Sets" as benchmarks to measure quality and uses LLM-based reasoning to suggest surgical improvements to agent prompts and skills.
The loop functions as a feedback mechanism where negative performance results—identified during manual runs or scheduled evaluations—trigger a refinement process. This process includes automatic prompt rewriting, A/B testing of candidate versions, and a proposal system for human-in-the-loop approval before promoting improved configurations to active status.
Sources: README.md:92-106, README.md:144-146
Architecture and Data Flow
The improvement loop follows a structured lifecycle from evaluation to promotion. It relies on several key architectural components: the Evaluation Engine, the Prompt/Skill Rewriters, and the Versioning System (Genome).
Optimization Workflow
The following diagram illustrates the automated path from a failed evaluation to a new agent proposal:
A nightly cron job executes this logic to ensure agents evolve alongside changing data or user requirements.
Sources: README.md:95-104
Golden Sets and Evaluation
Golden Sets serve as the ground truth for the improvement loop. These are curated collections of test cases containing specific inputs and their expected outputs.
Evaluation Metrics
Evaluations are conducted using an LLM judge (typically gpt-4o-mini) which assigns a score between 0.0 and 1.0 for each item. A pass rate of 0.7 is the default threshold; performance below this trigger activates the rewriter components.
Sources: README.md:96-112, services/api/flow/interfaces/http/routes/golden_sets.py:236-241
Key Data Structures
| Component | Description |
|---|---|
| Golden Set | A named collection of test items defined by a workspace and description. |
| Golden Item | A specific test case consisting of input_text, expected_output, and scoring_criteria. |
| Golden Result | The outcome of an evaluation, storing score, grading_rationale, and actual_output. |
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:108-115, services/api/flow/interfaces/http/routes/golden_sets.py:196-205
Refinement Mechanisms
When performance falls below thresholds, the system employs specialized rewriters to modify agent logic without losing existing functional strengths.
Skill and Prompt Rewriting
The Skill Rewriter and Prompt Rewriter perform surgical edits to markdown-based instructions. The rewriter analyzes failed items, including the judge's rationale for the failure, to produce an improved body.
_SKILL_REWRITER_SYSTEM = """\
You are an expert AI skill engineer.
Your task: Given the SKILL.md body and failed test cases, improve the body.
Rules:
1. Never touch the YAML frontmatter.
2. Surgical edits.
3. No regressions.
"""
Sources: services/api/flow/application/skill_rewriter.py:40-54
Iterative Versioning (Genome)
Every optimization results in a new "Genome" version. The system maintains a lifecycle of versions to prevent breaking changes:
- CANDIDATE: A newly generated version undergoing testing.
- ACTIVE: The version currently serving production requests.
- ARCHIVED: Previous versions kept for historical reference.
Sources: README.md:79-90
User Feedback Integration
The loop is further augmented by direct user interaction. Users can rate agent responses (0–100%). Scores below 50% automatically generate an "agent negative" record. These negatives are used by the worker grader as few-shot examples in future runs to prevent the recurrence of specific retrieval or synthesis errors.
Sources: README.md:144-148
Summary
The Continuous Improvement Loop ensures that Flow agents are not static entities but evolving systems. by combining automated LLM-based evaluation, surgical prompt engineering, and human-verified A/B testing, the platform maintains a high standard of accuracy and relevance while allowing for rapid, data-driven iterations.