Writing Python Graders
Relevant source files
The following files were used as context for generating this wiki page:
Python Graders provide automated, deterministic signals to verify that AI skills produce correct and rule-compliant output. They function within the skillgrade evaluation framework to assert specific outcomes for every new rule added to a skill. Graders reside in the graders/ directory, organized by the specific skill they support.
Sources: dev/knowledges/architecture.md:36-41, dev/guides/adding-a-rule.md:12-14
Grader Architecture and Data Flow
The evaluation system follows a three-tier structure to manage and execute tests. The eval.yaml file defines the tasks and prompts. Each task points to a specific Python script that acts as the entry point for grading. This script utilizes a shared library within the skill's grader directory to perform assertions.
The diagram shows the relationship between the evaluation configuration, the execution runner, and the modular grader logic.
Sources: dev/guides/running-evals.md:10-18, dev/guides/adding-a-rule.md:144-162
Component Breakdown
| Component | Path | Purpose |
|---|---|---|
| Task Definition | eval.yaml | Declares the task name, prompt, and the grader command to run. |
| Task Grader | graders/<skill>/check_<task>.py | Orchestrates specific checks for a single evaluation task. |
| Shared Library | graders/<skill>/lib.py | Contains the CHECKS registry and reusable logic for parsing and assertions. |
| Output Artifact | output.yml / output.py | The file generated by the AI model during evaluation that the grader inspects. |
Sources: dev/guides/running-evals.md:57-68, dev/guides/adding-a-rule.md:144-162
Core Principles of Grading
Determinism
Graders must be deterministic. You must never use LLM-based grading, network calls, or logic dependent on timing. Every check must return a hard pass or fail based on the inspection of the parsed artifact.
Sources: dev/guidelines/graders.md:7-9, dev/guides/running-evals.md:104-106
Parsing vs. Substring Matching
Avoid raw text matching (e.g., "x" in text). Substring matching causes false positives by passing answers that mention a trap and false negatives by failing correct answers with different wording. Graders must parse the artifact into a structured format before asserting.
| Artifact Type | Recommended Parsing Method |
|---|---|
| Schema / YAML | yaml.safe_load() then walk the structure. |
| Python Code | ast (Abstract Syntax Tree) module for structural analysis. |
| Shell Commands | shlex.split() for command segment verification. |
| Prose Reports | Rank evidence by locating sections and checking for specific artifacts. |
Sources: dev/guidelines/graders.md:11-19, dev/guides/adding-a-rule.md:100-112
Implementation Patterns
The lib.py Structure
The lib.py file exposes a CHECKS registry that maps assertion names to check functions. Each function accepts the parsed artifact and returns a tuple containing a boolean result and a success/failure message.
def check_naming_convention(schema: dict, **_) -> tuple[bool, str]:
"""Asserts that node names are in PascalCase."""
for node in schema.get("nodes", []):
name = node.get("name", "")
if not name[0].isupper():
return False, f"Node '{name}' is not PascalCase"
return True, "All nodes follow naming conventions"
CHECKS = {
"naming-convention": check_naming_convention,
}
Sources: dev/guides/adding-a-rule.md:82-95, dev/guidelines/graders.md:53-62
AST-Based Python Grading
When grading Python output (e.g., for generators or transforms), use the ast module to inspect logic without executing code. This allows you to verify that specific SDK calls (like client.create) are used correctly or that required loops are present.
Sources: dev/specs/2026-05-18-managing-generators-integration-rules-design.md:139-158, dev/specs/2026-05-19-managing-transforms-integration-rules-design.md:96-115
Verification and Testing
The Four-Fixture Rule
A compliant/violating pair is insufficient for testing a grader. You must verify the grader against four distinct fixtures to ensure accuracy and prevent "laundering" (passing via vocabulary instead of substance).
| Fixture Type | Expected Score | Purpose |
|---|---|---|
| Compliant | 1.0 | Verifies the rule works as documented. |
| Compliant (Variant) | 1.0 | Verifies traversal is robust against refactoring. |
| Violating (Obvious) | < 1.0 | Verifies the grader detects the core violation. |
| Violating (Near-miss) | < 1.0 | Verifies the grader detects violations that use correct keywords/vocabulary. |
Sources: dev/guidelines/graders.md:79-92, dev/guides/adding-a-rule.md:167-183
Integration with eval.yaml
Task grader scripts are invoked by skillgrade as defined in eval.yaml. The script reads the AI-generated artifact (usually output.yml), runs the required checks, and outputs a standardized JSON result.
import json
import sys
from pathlib import Path
from lib import run_checks
CHECKS = ["schema-version", "naming-convention"]
if __name__ == "__main__":
# Task graders print skillgrade JSON to stdout
print(json.dumps(run_checks(CHECKS, Path("output.yml"))))
Sources: dev/guides/adding-a-rule.md:144-162, dev/guides/running-evals.md:92-102
Summary
Python Graders ensure that Infrahub skills remain reliable and effective as the project evolves. By following deterministic parsing patterns and the four-fixture verification rule, authors prevent silent rot and maintain high-quality guidance for AI models. Graders prioritize structural correctness over superficial text matching to provide meaningful feedback to developers and evaluators.