Generators & Artifact Generation
Relevant source files
The following files were used as context for generating this wiki page:
Generators and Artifacts are the primary mechanisms in Infrahub for managing derived infrastructure data. Generators produce new nodes or attributes within the graph based on existing data, while Artifacts typically represent rendered outputs like configuration files or documentation derived from the graph via transformations.
The system uses a precise triggering mechanism to ensure that derived data remains consistent with its source inputs. This mechanism minimizes unnecessary compute by re-running generation only when a relevant data change, Git file change, or dependency update occurs.
Sources: dev/specs/ifc-2738-generator-regen-triggers/spec.md, dev/knowledge/backend/code-generation.md
Regeneration Triggers
Infrahub identifies three categories of changes that trigger the regeneration of a generator or artifact. The system prioritizes correctness, ensuring that over-execution is acceptable but under-execution is not.
Trigger Categories
- Data Change: A modification occurs to a node or relationship that the definition's GraphQL query reads. This is tracked via the
MODIFIED_KINDSpath. - Target Group Membership: A new node joins the target group of a definition. The system runs the generator/artifact for the new member while skipping unchanged members.
- Closure Change: A change occurs in the dependency closure of the definition. This includes the source code (Python/Jinja2), the GraphQL query file, the definition node itself, or user-declared
watchfiles.
Sources: dev/specs/ifc-2738-generator-regen-triggers/spec.md:27-46, dev/specs/ifc-2738-generator-regen-triggers/plan.md:16-20
Regeneration Predicates
The pipeline evaluates regeneration using three specialized predicates. These predicates determine if a specific instance requires a re-run based on the Git diff of the proposed change.
| Predicate | Description |
|---|---|
_query_changed | Checks if the .gql query file associated with the definition was modified. |
_definition_changed | Checks if the definition node itself (e.g., CoreGeneratorDefinition) was changed in the graph. |
_transform_changed | Performs a set intersection between the definition's stored dependency closure and the Git repository diff. |
Sources: dev/specs/infp-409-artifact-regen-triggers/plan.md:14-18, dev/specs/ifc-2738-generator-regen-triggers/plan.md:9-12
Dependency Closure Architecture
A dependency closure represents the complete set of files that influence the output of a generator or transformation. The system builds this closure during Git repository import.
Closure Components
- Source Files: For Python generators, this includes the generator file and its package-directory floor. For Jinja2 artifacts, it includes the template and its includes.
- GraphQL Queries: The
.gqlfile used to fetch data for the generation process. - Manifest: The
.infrahub.ymlfile, ensuring that configuration changes trigger a refresh. - User Declarations: Additional paths specified via the
watch:field in the repository configuration.
Sources: dev/specs/ifc-2738-generator-regen-triggers/spec.md:183-195, dev/specs/ifc-2738-generator-regen-triggers/plan.md:9-12
Data Flow for Triggering
This diagram illustrates how the system evaluates whether to run a generator during a proposed change.
The system uses the RegenerationDefinition protocol to handle both artifacts and generators through a unified logic path.
Sources: dev/specs/ifc-2738-generator-regen-triggers/plan.md:9-12, dev/specs/ifc-2738-generator-regen-triggers/spec.md:173-181
Data Models
Definitions are represented in the graph and the pipeline by specific models that carry the dependency closure attributes.
Core Schema Attributes
Both CoreGeneratorDefinition and CoreTransformation (used by artifacts) store the following attributes to support precise triggering.
| Attribute | Type | Description |
|---|---|---|
dependencies | List (Text) | Serialized list of file paths in the dependency closure. |
dependencies_complete | Boolean | Indicates if the closure is known to be complete. If False, the system falls back to run-on-any-change. |
Sources: dev/specs/ifc-2738-generator-regen-triggers/spec.md:167-172, dev/specs/infp-409-artifact-regen-triggers/plan.md:19-24
Pipeline Models
During a proposed change, the system instantiates pipeline-specific models to evaluate triggers.
# backend/infrahub/generators/models.py
class ProposedChangeGeneratorDefinition(BaseModel):
definition_id: str
definition_name: str
query_id: str
query_name: str
dependencies: list[str] | None = None
dependencies_complete: bool | None = None
source_noun: str = "generator source"
instance_noun: str = "instances"
Sources: backend/infrahub/generators/models.py:7-16, dev/specs/ifc-2738-generator-regen-triggers/plan.md:100-110
User Configuration (watch:)
Users can declare extra dependencies that the automatic closure builder cannot detect, such as sibling packages or external data files. This is configured in the .infrahub.yml file.
SDK Configuration
The watch: field is a strict object that prevents under-execution in complex scenarios. It requires the files: list to be explicitly defined.
generators:
- name: "DeviceGenerator"
class_name: "DeviceGenerator"
file_path: "generators/device.py"
watch:
files:
- "common_helpers/"
- "data/vendor_mappings.json"
Sources: dev/specs/ifc-2738-generator-regen-triggers/spec.md:139-158, dev/specs/ifc-2738-generator-regen-triggers/plan.md:9-12
Summary
Generators and Artifact generation in Infrahub utilize a shared regeneration framework to maintain derived data integrity. By migrating from blunt repository-wide triggers to precise dependency closures, the system reduces pipeline execution time while ensuring that all relevant changes—whether in data, code, or queries—are correctly propagated. The inclusion of the watch: field allows users to extend this precision to complex, cross-package dependencies.
Sources: dev/specs/ifc-2738-generator-regen-triggers/spec.md:16-25, dev/specs/infp-409-artifact-regen-triggers/plan.md:8-13