Skill Change Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Skill Change Pipeline is a structured workflow designed to manage the lifecycle of AI agent skills within the Infrahub ecosystem. It provides a technical framework for contributors to take a bug or a feature idea through a rigorous verification process, culminating in a pull request. The pipeline ensures that every change to a skill is grounded in deterministic testing and adheres to the project's architectural standards.
This system moves changes through four primary stages: analyzing-skill-bugs or grilling-skill-features, test-driving-skill-changes, and implementing-skill-changes. This progression enforces the "Rule = Test" principle, ensuring that no guidance is added to the repository without accompanying automated evaluation coverage.
Sources: dev/README.md:27-31, AGENTS.md:24-27, dev/guidelines/rule-equals-test.md:1-5
Pipeline Architecture and Flow
The pipeline follows a strict sequence to ensure technical accuracy and maintainability. It starts by identifying a gap and moves toward a failing test before any prose is written.
Workflow Progression
The following diagram illustrates the logical flow of a change as it moves through the contributor skills:
This diagram shows the path from initial identification to the final pull request submission through specific contributor skills.
Sources: dev/README.md:27-31, AGENTS.md:24-27
The Rule = Test Mandate
A core requirement of the pipeline is that every new rule must ship with its evaluation coverage in the same change. This prevents "silent rot" where skill prose drifts from intended behavior without failing tests to flag the discrepancy.
Required Components for a Change
Every change to a skill rule must include the following elements:
- Rule File: A new or edited Markdown file under
skills/<skill>/rules/. - Check Function: A Python function in
graders/<skill>/lib.py. - Eval Task: A task definition in the root
eval.yaml. - Task Grader: A script that executes the check against model output.
- Fixtures: At least four fixtures to test the grader's own logic.
- Regenerated JSON: Updated
evaluations/*.jsonfiles.
Sources: AGENTS.md:162-167, dev/guidelines/rule-equals-test.md:1-10
Implementation Stages
1. Analysis and Grilling
The pipeline begins with either analyzing-skill-bugs or grilling-skill-features. In this phase, you determine if the guidance needs to exist by applying the "Minimum Change" ladder. You must verify if the repo already says it, if an existing rule can cover it in one sentence, or if the Infrahub product itself should handle the validation.
Sources: dev/README.md:27-31, dev/guidelines/minimum-change.md:10-25
2. Test-Driven Development (TDD)
Before implementing the rule, you must create a failing test case. This involves defining the eval.yaml task and the grader logic.
- Grader Logic: Graders must parse the model output (e.g., as YAML) rather than using substring matching.
- Verification: You must verify the grader against positive and negative fixtures to ensure it correctly identifies both compliant and non-compliant artifacts.
Sources: dev/guidelines/rule-equals-test.md:15-25, dev/guides/adding-a-rule.md:10-20
3. Implementation and Sweeping
Once a failing test exists, you implement the rule prose. The implementing-skill-changes skill guides the actual writing and ensures that any older, contradictory rules are swept and removed to prevent "information drift."
Sources: dev/README.md:27-31, dev/guidelines/minimum-change.md:38-41
Evaluation and Grading System
The pipeline relies on skillgrade to execute evaluations. Evals are tiered by cost to allow for efficient local testing and thorough CI verification.
| Tier | Execution Context | Description |
|---|---|---|
| Smoke | Local / Release PR | A subset of tasks used for quick verification and as a release gate. |
| Regression | Weekly / Release PR | The full suite of 144+ tasks executed against all 15 skills. |
| Single Eval | Development | skillgrade --eval=<task> --trials=1 used to test specific changes. |
Sources: AGENTS.md:88-92, AGENTS.md:188-192, dev/knowledges/architecture.md:85-95
Sequence of a Rule Test
The sequence for testing a rule within the pipeline is as follows:
This diagram depicts the interaction between the contributor and the automated testing tools during the pipeline execution.
Sources: dev/guides/adding-a-skill.md:165-195, dev/knowledges/architecture.md:85-95
Conclusion
The Skill Change Pipeline enforces a rigorous, test-first approach to skill development. By requiring deterministic graders and synchronized documentation updates, it ensures the Infrahub skills remain technically accurate and free of contradictory guidance. Success in the pipeline is defined by the "Rule = Test" principle, where no rule is considered complete until it is backed by a passing, automated evaluation.