Agent: Analyzing Skill Bugs
Relevant source files
The following files were used as context for generating this wiki page:
The Analyzing Skill Bugs agent is a specialized contributor skill designed to triage and diagnose defects within the Infrahub-skills repository. It serves as the initial entry point in the automated skill-change pipeline, facilitating the identification of "quiet failures"—guidance gaps where a skill's rules are missing, unclear, or incorrect, leading to inefficient model performance or user friction.
This agent operates within the .agents/skills/ directory, providing instructions for AI coding assistants (such as Claude Code) to analyze repository-specific bugs rather than general application code. It focuses on the relationship between skill prose, deterministic graders, and evaluation tasks to maintain the "Rule = Test" integrity of the project.
Sources: AGENTS.md:18-35, dev/README.md:41-50, dev/knowledges/architecture.md:9-25
The Skill-Change Pipeline
The Analyzing Skill Bugs agent initiates a structured multi-step pipeline for repository maintenance. This pipeline ensures that every bug fix or feature addition follows a rigorous path from diagnosis to pull request.
The diagram shows the sequential flow of contributor skills used to modify the repository.
Sources: dev/README.md:41-50, AGENTS.md:53-61
Triage and Bug Detection
Identifying bugs in a skill requires analyzing session friction rather than system crashes. The agent uses a "Detection Ladder" to establish whether a gap in guidance exists.
The Detection Ladder
- Verifier Verdict: Locate a red-to-green transition where a tool (e.g.,
infrahubctl schema load) initially failed but passed after manual intervention. - Coverage Read: Inspect the
rules/directory for the implicated skill to identify if a specific rule exists or is missing. - Correction Delta: Compare the first authored version of a solution against the final accepted version to identify the specific missing guidance.
- Session-Shape Counters: Count retries, edit churn, and user nudges. These serve as signals to open an investigation but do not constitute proof of a bug on their own.
Sources: skills/infrahub-reporting-skill-gaps/SKILL.md:65-80, dev/guidelines/minimum-change.md:12-25
Bug Classification
The agent distinguishes between three categories of defects:
- Skill Defect: A rule or reference covers the topic but the model still failed (Bug), or no rule exists for a verified requirement (Feature).
- Product Defect: The skill guidance was followed correctly, but the underlying Infrahub platform or SDK misbehaved.
- Documentation Gap: The skill is silent, and an external doc lookup also failed to provide a settled answer.
Sources: skills/infrahub-reporting-skill-gaps/SKILL.md:86-105
Verification and Testing Procedures
Once a bug is analyzed, the agent must verify the defect through the project's testing framework. Verification involves comparing the local working tree against the installed plugin and the evaluation suite.
Verification Tools
| Task | Command | Description |
|---|---|---|
| Freshness Check | uv run invoke freshness | Checks the gap between the installed plugin and the current working tree. |
| Smoke Pass | skillgrade --smoke | Runs the full evaluation suite to detect regressions. |
| Targeted Eval | skillgrade --eval=<task> --trials=1 | Runs a single evaluation task to verify a specific fix. |
Sources: AGENTS.md:154-165, dev/guidelines/skill-authoring.md:105-110
Verification Sequence
The sequence diagram illustrates the steps an agent takes to verify a bug within the local environment.
Sources: dev/guides/adding-a-rule.md:180-210, AGENTS.md:185-195
Rule and Grader Integrity
A bug fix is only complete when it adheres to the "Rule = Test" principle. The Analyzing Skill Bugs agent ensures that changes to the prose are backed by deterministic assertions.
- Parse, Don't Match: Graders must parse the artifact (YAML, Python AST) rather than using substring matching (
"x" in text). Substring matching is prone to false positives from comments or docstrings. - Ranking Evidence: When grading prose reports, the agent looks for specific, ranked evidence markers rather than a list of potential phrasings.
- Fixture Validation: Graders must be tested against four fixtures: Compliant, Compliant Variant, Violating, and Violating Near-miss.
Sources: dev/guides/adding-a-rule.md:85-115, dev/guidelines/graders.md
Version and Metadata Analysis
Bugs are often tied to specific software versions. The agent collects three critical version strings during analysis to ensure maintainers can reproduce the issue:
- Skills Version: Found in the
SKILL.mdfrontmatter of the implicated skill. - Infrahub SDK Version: Obtained via
infrahubctl infoorpip show infrahub-sdk. - Infrahub Server Version: Obtained via
infrahubctl info.
Sources: skills/infrahub-reporting-skill-gaps/SKILL.md:137-155, skills/infrahub-reporting-issues/SKILL.md:155-175
The Analyzing Skill Bugs agent provides the necessary structure to turn developer friction into actionable repository improvements. By following the detection ladder and enforcing the "Rule = Test" lifecycle, it ensures the guidance remains accurate across Infrahub releases.