Transforms and Generators
Relevant source files
The following files were used as context for generating this wiki page:
Transforms and Generators are foundational components of the Infrahub pipeline that allow developers to automate infrastructure data management. These components run as part of Infrahub's internal pipeline to process, generate, and validate infrastructure data as native Python objects.
Generators create or update infrastructure data within the platform, while Transforms convert infrastructure data into different formats, such as device configurations or documentation. Both utilize the Infrahub SDK to interact with the underlying data model through either Jinja2 templates or Python code.
Sources: AGENTS.md:12-14, tasks.py:465-470
Architecture and Execution Flow
Transforms and Generators operate within specific execution contexts that define their security and access boundaries. The system differentiates between logic that runs in the platform core and logic that runs on distributed workers.
The diagram shows how the Infrahub Pipeline routes tasks to different executors based on whether they use Python or Jinja2 templates.
Execution Contexts
Artifact composition and data transformation are governed by an ExecutionContext flag enum. This model ensures that sensitive operations are restricted to appropriate environments.
| Context | Description |
|---|---|
CORE | Most restrictive; used for schema-level validation and computed attributes. |
WORKER | Used by Prefect workers to execute transforms and generators. |
LOCAL | Least restrictive; used for local development and CLI operations. |
Sources: specs/infp-504-artifact-composition/tasks.md:23-28, specs/infp-504-artifact-composition/plan.md:32-35
Generators
Generators automate the creation of infrastructure objects. They are typically implemented in Python and executed during the import or synchronization phases of the Infrahub lifecycle.
Key Logic
Generators process input data—such as files in a repository—and instantiate nodes within the Infrahub graph. The SDK provides mechanisms to track these updates through task logs.
- Registration: Generator definitions are discovered in the repository under specified paths.
- Execution: The generator runs within a specific branch and title context.
- Logging: The system records when generators are processed, such as
create_circuit_endpointsorbackbone_service.
Sources: tasks.py:476-479, tests/fixtures/tasks/mock_query_tasks_04_page3.json:76-118
Transforms
Transforms convert existing infrastructure data into new representations. They support two primary implementation methods: Jinja2 templates and Python scripts.
Jinja2 Transforms
Jinja2 transforms use the Infrahub SDK's templating engine to generate text-based artifacts. The engine supports auto_await for asynchronous operations, allowing templates to fetch external content during rendering.
This sequence illustrates how a Jinja2 transform fetches sub-artifact content through the artifact_content filter during execution on a worker.
Sources: specs/infp-504-artifact-composition/tasks.md:38-40, specs/infp-504-artifact-composition/plan.md:27-30
Advanced Templating Filters
Infrahub extends standard Jinja2 with custom filters to support complex artifact composition.
artifact_content: Inlines rendered content from other artifacts.file_object_content: Inlines content from file objects, rejecting binary data.from_json/from_yaml: Parses strings into structured Python dictionaries for further manipulation within the template.
Sources: specs/infp-504-artifact-composition/tasks.md:65-102, specs/infp-504-artifact-composition/plan.md:14-17
Development Workflow
Developers creating Transforms and Generators must adhere to the SDK's dual async/sync API parity. This ensures that automation scripts work consistently regardless of the execution environment.
- Environment Setup: Install dependencies using
uv sync --all-groups --all-extras. - Implementation: Write Python or Jinja2 logic in the project repository.
- Validation: Run
uv run invoke lintto check for code quality and type safety. - Documentation: If new CLI commands or configurations are added for these components, regenerate documentation with
uv run invoke docs-generate.
Sources: AGENTS.md:18-25, dev/constitution.md:46-52, tasks.py:108-112
Summary
Transforms and Generators represent the active logic layer of the Infrahub platform. By combining Python's flexibility with Jinja2's templating capabilities, they enable automated data generation and artifact production. The use of strict ExecutionContext controls ensures that these automated processes remain secure while providing developers with powerful tools like artifact_content for composite data management.