Evaluation System & Runner
Relevant source files
The following files were used as context for generating this wiki page:
The Evaluation System tests that AI skills produce correct and consistent output. It uses skillgrade to execute tasks defined in a central configuration and verifies results using deterministic Python graders. This system ensures that domain-specific rules, such as Infrahub schema naming or relationship identifiers, remain enforced across skill updates and releases.
Sources: dev/knowledges/architecture.md:65-69, dev/guides/running-evals.md:5-9
System Architecture
The evaluation architecture consists of three primary layers: the configuration layer (eval.yaml), the execution engine (skillgrade), and the verification layer (deterministic graders).
The diagram shows the flow from task configuration to the final scoring result, highlighting the separation between the runner and the verification logic.
Components
- Configuration (
eval.yaml): Defines all evaluation tasks, prompts, expected outputs, and grader paths in a single root file. - Execution Engine (
skillgrade): Orchestrates the evaluation by running trials, managing timeouts, and aggregating scores. - Grader Scripts: Deterministic Python files located in
graders/<skill>/that inspect artifacts (usually YAML or Python code) generated by the AI. - Assertion Library (
lib.py): A per-skill registry of check functions that perform structural validation (AST parsing or YAML walking) rather than simple text matching.
Sources: dev/knowledges/architecture.md:70-76, dev/guides/running-evals.md:75-84, dev/guides/adding-a-rule.md:126-135
Task Configuration and Formatting
Every evaluation task lives in the root eval.yaml. A task names the specific skill to load via its instruction block, allowing multiple skills to be tested from the same configuration file.
eval.yaml Structure
The configuration uses a defaults block for global settings and a tasks list for individual test cases.
| Field | Description |
|---|---|
name | Unique identifier for the task. |
trials | Number of times to run the task to ensure consistency. |
instruction | The prompt provided to the AI, including the path to the SKILL.md. |
graders | A list of grader scripts to run, including their run command and weight. |
expected_output | Documentation describing what the correct output should look like. |
assertions | Human-readable descriptions of what the task validates. |
Sources: dev/guides/running-evals.md:63-90, dev/guides/adding-a-skill.md:144-165
Syncing and Projections
The system maintains a per-skill JSON projection of task definitions in the evaluations/ directory. These files are generated by scripts/sync-evals.py and are used by external tools like the /skill-creator runner. You must run this script after every edit to eval.yaml to prevent synchronization drift.
Sources: dev/guides/running-evals.md:17-25, dev/guides/adding-a-rule.md:275-280
Execution and Presets
skillgrade provides three primary presets for local execution, varying by the number of trials and the intended use case.
| Preset Flag | Trials | Primary Use Case |
|---|---|---|
--smoke | 5 | Quick feedback during active development. |
--reliable | 15 | Confirming a fix or improvement. |
--regression | 30 | Pre-release validation or PR gates. |
Sources: dev/guides/running-evals.md:36-46
CI Quality Gates
In Continuous Integration (CI), the system runs each skill with a 0.8 pass rate threshold. If the average score falls below this threshold, the CI quality gate fails.
Sources: dev/guides/running-evals.md:48-54, dev/knowledges/architecture.md:86-90
Deterministic Verification Logic
Graders are designed to be strictly deterministic. They must not use LLM grading, network calls, or rely on timing. Instead, they parse the generated artifacts and validate their structure.
Grader Data Flow
The sequence shows how the grader script mediates between the runner and the internal assertion logic to produce a JSON result.
Implementation Guidelines
- Parsing over Substring Matching: Graders must parse artifacts using
yaml.safe_loadfor YAML orastfor Python code. Text matching (e.g.,"x" in text) is forbidden as it creates false positives from comments or docstrings. - Fixture-Based Verification: New graders are verified against four distinct fixtures to ensure they correctly identify compliance and violations.
| Fixture Type | Expected Outcome |
|---|---|
| Compliant | 1.0 (Direct match with rule example) |
| Compliant Variant | 1.0 (Correct logic with different phrasing/structure) |
| Violating | < 1.0 (Obvious rule breach) |
| Violating Near-Miss | < 1.0 (Uses keywords but breaks logic) |
Sources: dev/guides/adding-a-rule.md:139-158, dev/guides/adding-a-rule.md:214-235, dev/specs/2026-05-18-managing-generators-integration-rules-design.md:104-120
Writing Effective Evaluations
Evaluation prompts should represent realistic user requests rather than abstract commands.
- Avoid Meta-Prompts: Do not tell the AI to "follow the rule" or dictate the exact output schema in the prompt. This verifies the model's training rather than the skill's effectiveness.
- Discrimination Testing: A valid task must fail if the
SKILL.mdis not provided. Developers verify this by temporarily removing theRead the skill at...line from the instruction; a resulting score below 1.0 proves the task discriminates based on the skill content. - Transfer over Retrieval: Tasks should not reuse the worked examples found in
examples.md, as this measures the model's ability to copy rather than its ability to apply patterns to new domains.
Sources: dev/guides/running-evals.md:112-140, dev/knowledges/skill-writing-guide.md:144-155