Prompt & Skill Rewriter Engine
Relevant source files
The following files were used as context for generating this wiki page:
The Prompt & Skill Rewriter Engine is the core component of Flow's self-improvement feedback loop. It automates the iterative optimization of agent instructions by analyzing performance failures against "Golden Sets" (curated test cases). When an agent fails to meet quality thresholds, the engine generates targeted edits to either the global system prompt or specific modular skills to address the root causes of those failures.
The engine operates by taking the current instructions, actual outputs from failed tests, and the rationale provided by an LLM judge. It then produces an improved version of the instruction set, a human-readable changelog, and a confidence score. This process enables a continuous improvement cycle where agents can evolve through Agent Genome versioning and A/B testing.
Sources: README.md:123-135, services/api/flow/application/prompt_rewriter.py:1-15
Core Architecture and Data Flow
The engine is divided into two specialized modules: the PromptRewriter for global system instructions and the SkillRewriter for modular SKILL.md files. Both utilize a structured feedback loop driven by evaluation results.
Self-Improvement Workflow
The system follows a specific sequence to promote an improved instruction set:
- Evaluation: Run a Golden Set evaluation; if the pass rate is below a threshold (e.g., 0.7), the loop triggers.
- Analysis: The Rewriter analyzes failed items (input, expected, actual, and judge rationale).
- Proposals: Targeted edits are generated and snapshotted as a
CANDIDATEgenome. - Validation: An A/B test is performed against the baseline.
- Promotion: If the candidate outperforms the baseline, a proposal is created for human approval.
Prompt Rewriter Module
The prompt_rewriter.py module handles the optimization of an agent's main system prompt. It focuses on "surgical edits" rather than wholesale rewrites to maintain the original structure and voice of the agent.
Technical Implementation
The rewriter uses a FailedItem dataclass to encapsulate test failures. The rewrite_prompt function interacts with the OpenAI API (typically gpt-4o-mini) to generate the RewriteResult.
| Component | Description |
|---|---|
FailedItem | Contains input_text, expected_output, actual_output, score, and rationale. |
RewriteResult | Output containing improved_prompt, changelog, failure_analysis, and confidence. |
rewrite_and_snapshot | Orchestrates the full pipeline: rewrite -> snapshot candidate -> restore original. |
Prompt Rewriting Logic
The rewriter follows strict rules during the mutation process:
- Root-cause identification: Analyze why failures happened before proposing fixes.
- No regressions: Ensure changes do not break previously passing items.
- Specific instructions: Add concrete behavioral rules rather than vague guidance.
- Format preservation: Maintain XML, Markdown, or JSON structures present in the original prompt.
Sources: services/api/flow/application/prompt_rewriter.py:27-104
Skill Rewriter Module
The skill_rewriter.py module focuses on modular skills stored in SKILL.md format. It utilizes the same FailedItem data but is designed to handle the unique structure of skills, which includes YAML frontmatter.
SKILL.md Structure Handling
Skills consist of two parts: a YAML frontmatter block and a Markdown body. The SkillRewriter is strictly prohibited from modifying the frontmatter directly, except for the version string.
Versioning and Persistence
When a skill is improved:
- The frontmatter is parsed to extract the current version.
- The version string (e.g.,
1.0) is incremented (e.g.,1.1). - The improved body is prepended with the updated frontmatter.
Sources: services/api/flow/application/skill_rewriter.py:48-93, services/api/flow/application/skill_parser.py:15-46
Integration with Evaluation Systems
The rewriter engine is triggered by the Golden Evaluator. Golden Sets are collections of curated test cases used to measure agent quality.
Evaluation and Triggering
API endpoints facilitate the manual and automatic triggering of evaluations. The evaluate_golden_set function serves as the entry point for determining if an agent requires rewriting.
| Endpoint | Method | Purpose |
|---|---|---|
/api/v1/golden-sets/{set_id}/evaluate | POST | Triggers asynchronous evaluation of an agent against a set. |
/api/v1/golden-sets/{set_id}/results | GET | Retrieves scores and rationales used by the rewriter. |
/api/v1/skills/{skill_id}/improve | POST | Manually triggers the skill improvement loop for a specific skill. |
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:246-270, apps/web/src/app/(app)/agents/[id]/skills/page.tsx:132-160
User Interface and Feedback
The web frontend provides visibility into the rewriter's actions through the Skills management page and the Evaluation dashboard.
- Improvement Proposals: When the rewriter generates a change, users see a notification indicating a proposal has been created, often including a confidence percentage and a snippet of the changelog.
- Diff View: Users can compare the original instruction set with the improved version using the
SkillDiffViewcomponent. - Manual Overrides: While the engine automates improvements, the
SkillEditorallows developers to manually author instructions using a YAML frontmatter and Markdown body template.
Sources: apps/web/src/app/(app)/agents/[id]/skills/page.tsx:142-155, apps/web/src/components/agents/SkillEditor.tsx:11-30
The Prompt & Skill Rewriter Engine transforms Flow from a static agent platform into a self-optimizing system. By grounding prompt mutations in empirical evidence from test failures, it ensures that instruction updates are targeted, verifiable, and continuously driving toward higher performance.