Agent: Test-Driving Skill Changes
Relevant source files
The following files were used as context for generating this wiki page:
The Test-Driving Skill Changes agent is a specialized component of the Infrahub skills development pipeline. It serves as the implementation bridge that takes identified bugs or feature requests through a test-driven development (TDD) cycle to ensure every guidance change is verifiable and resistant to regression.
This agent operates within the "Rule = Test" framework, where no rule change is permitted to ship without corresponding automated evaluation coverage. It focuses on creating deterministic graders and evaluation tasks that validate the output of AI skills against specific, structural requirements.
Sources: AGENTS.md:19-21, AGENTS.md:126-130, dev/guides/adding-a-rule.md:1-5
Core Principles
The agent follows strict architectural and authoring guidelines to maintain the integrity of the skill repository.
Rule = Test Requirement
Every new or modified rule in a skill's rules/ directory must ship with at least one evaluation task and one deterministic grader check. The grader check asserts the rule's outcome by inspecting the structural content of the model's output rather than performing simple substring matches.
Sources: dev/guides/adding-a-rule.md:7-10, AGENTS.md:126-130
The Minimum Change Ladder
Before creating new artifacts, the agent evaluates the "Minimum Change Ladder" to avoid repository bloat. It attempts to resolve gaps at the lowest possible rung before moving to the next.
| Rung | Action |
|---|---|
| 1 | Determine if guidance needs to exist at all based on recurrence of the mistake. |
| 2 | Check if the repository already contains the information in existing rules or references. |
| 3 | Edit an existing rule to cover the concern in a single sentence. |
| 4 | Defer to Infrahub product validators (e.g., infrahubctl schema check). |
| 5 | Add a single line to the skill's SKILL.md workflow. |
| 6 | Create the minimum new artifact (one rule file, one check function, one task). |
Sources: dev/guidelines/minimum-change.md:5-24
Evaluation System Architecture
The evaluation system uses a structured approach to verify skill performance.
The diagram shows the relationship between skill components and the evaluation machinery that verifies them.
Sources: dev/knowledges/architecture.md:12-38, dev/guides/adding-a-rule.md:65-75
Components of the System
| Component | Description | Path |
|---|---|---|
| Eval Configuration | Defines tasks, prompts, and grader paths. | eval.yaml |
| Grader Library | Contains reusable, deterministic check functions. | graders/<skill>/lib.py |
| Task Grader | A script that invokes specific checks for a task. | graders/<skill>/check_<task>.py |
| JSON Projection | Auto-generated task data for external runners. | evaluations/<skill>.json |
Sources: dev/knowledges/architecture.md:73-86, dev/guides/adding-a-rule.md:23-28, dev/guides/adding-a-rule.md:162-166
Implementation Workflow
The agent follows a sequence of steps to test-drive a change.
1. Rule Creation and Registration
The agent writes the rule using the imperative form and provides reasoning (the "why") behind the constraint. Each rule must be linked from the SKILL.md file at the specific workflow step where it is needed to ensure the agent reads it at the correct time.
Sources: dev/guides/adding-a-rule.md:14-21, dev/knowledges/skill-writing-guide.md:38-42
2. Grader Development
The agent develops deterministic Python functions that parse the model's output (e.g., using yaml.safe_load for YAML or ast for Python code). Substring matching is explicitly forbidden because it leads to false positives and negatives.
Sources: dev/guides/adding-a-rule.md:36-40, dev/guides/adding-a-rule.md:55-65
# Example of a deterministic check function
def check_attribute_type(schema: dict, **_) -> tuple[bool, str]:
"""Asserts that 'Text' is used instead of deprecated 'String'."""
for node in schema.get("nodes", []):
for attr in node.get("attributes", []):
if attr.get("kind") == "String":
return False, f"Node {node['name']} uses deprecated 'String'"
return True, "All attributes use correct types"
Sources: dev/guides/adding-a-rule.md:28-35
3. Task Creation
The agent adds a realistic task to eval.yaml. The prompt must exercise the rule without naming it, ensuring the test discriminates between an agent that has read the skill and one that has not.
Sources: dev/guides/adding-a-rule.md:92-105
4. Verification with Fixtures
The agent verifies the grader against four specific types of fixtures to ensure accuracy.
| Fixture Type | Expected Score | Purpose |
|---|---|---|
| Compliant | 1.0 | Verify standard compliance. |
| Compliant Variant | 1.0 | Ensure structural changes (e.g., field order) don't break the check. |
| Violating | < 1.0 | Confirm the check catches obvious errors. |
| Near-Miss | < 1.0 | Verify the check isn't fooled by keywords or comments. |
Sources: dev/guides/adding-a-rule.md:118-125
Tooling and Commands
The agent utilizes the skillgrade utility and several sync scripts to manage the lifecycle of a change.
The sequence illustrates the verification and synchronization process required before a change is ready for a pull request.
Sources: dev/guides/adding-a-rule.md:162-175, AGENTS.md:57-65, AGENTS.md:148-155
Key Commands
- Run specific eval:
skillgrade --eval=<task-name> --trials=1 - Run smoke suite:
skillgrade --smoke - Regenerate JSON:
python scripts/sync-evals.py - Create news fragment:
uv run towncrier create -c "Description" <issue_number>.added.md
Sources: AGENTS.md:57-65, dev/guides/adding-a-skill.md:165-170, AGENTS.md:150-155
Handoff and Integration
Once tests pass and JSON files are synchronized, the agent prepares the final submission. This includes sweeping the repository for old claims that the new rule contradicts and adding a towncrier news fragment to ensure the change is reflected in the CHANGELOG.md upon release.
Sources: dev/guides/adding-a-rule.md:144-149, dev/guidelines/skill-authoring.md:89-94, AGENTS.md:148-155