Data Import and Export
Relevant source files
The following files were used as context for generating this wiki page:
Data Import and Export mechanisms in the Infrahub SDK enable the movement of infrastructure data between the Infrahub server and external file formats. This system abstracts the complexity of GraphQL queries and mutations, allowing developers to treat infrastructure nodes and their relationships as portable objects.
The system supports batch operations for creating or updating nodes from structured files (JSON/YAML) and exporting existing data into line-delimited formats. These capabilities are primarily exposed through the infrahubctl CLI and specialized transfer interfaces.
Sources: AGENTS.md:14-16, dev/specs/001-end-user-cli/spec.md:124-142
Architecture and Components
The data transfer system follows a provider-interface pattern. The transfer module contains interfaces for both importing and exporting, which are then implemented by specific format handlers and orchestrated by the CLI layer.
Core Transfer Interfaces
The SDK defines abstract base classes to ensure consistency across different data formats.
- ExporterInterface: Defines the contract for extracting data from Infrahub. It requires an
exportmethod that takes a target directory, a list of namespaces to include, and the source branch. - ImporterInterface: Defines the contract for loading data into Infrahub. It requires an
import_datamethod that processes a source path and a target branch.
Sources: infrahub_sdk/transfer/exporter/interface.py:7-14, infrahub_sdk/transfer/importer/interface.py:7-14
Data Flow Overview
The following diagram illustrates the high-level flow of data during export and import operations.
The diagram shows how the CLI coordinates between the Infrahub Server and the local file system using specialized transfer logic.
Sources: infrahub_sdk/ctl/exporter.py, infrahub_sdk/ctl/importer.py, infrahub_sdk/transfer/exporter/json.py
Export System
The export system retrieves schema definitions, nodes, and complex relationships from a specific branch and namespace.
Line-Delimited JSON Export
The LineDelimitedJSONExporter is the primary implementation for data extraction. It performs a multi-step process to ensure data integrity:
- Schema Retrieval: Fetches the schema for requested namespaces to map node kinds.
- Relationship Identification: Scans the schema to identify many-to-many relationships (cardinality "many") that require separate tracking.
- Node Retrieval: Uses a batch execution mechanism to fetch all nodes of the identified kinds.
- Serialization: Writes nodes to
nodes.json(one JSON object per line) and many-to-many relationships torelationships.json.
Sources: infrahub_sdk/transfer/exporter/json.py:27-119
Export Components
| Component | Description |
|---|---|
identify_many_to_many_relationships | Analyzes node schemas to find src/dst couples where cardinality is 'many'. |
retrieve_many_to_many_relationships | Executes GraphQL queries to fetch relationship edges using identifiers found in the schema. |
export | Orchestrates the full export process, including directory validation and progress tracking. |
Sources: infrahub_sdk/transfer/exporter/json.py:38-119
Import System
The import system facilitates batch creation and modification of nodes using structured files.
Batch Operations
The infrahubctl create --file and infrahubctl update --file commands leverage the import logic to process multiple objects simultaneously. The system supports both JSON and YAML formats sharing a unified schema.
- File Validation: The importer validates that the input file follows the Infrahub spec object structure (e.g.,
apiVersion: infrahub.app/v1). - Node Instantiation: Parsed data is converted into SDK
InfrahubNodeobjects. - Server Interaction: The objects are sent to the Infrahub server using batch save operations to optimize network overhead.
Sources: dev/specs/001-end-user-cli/spec.md:124-142, infrahub_sdk/ctl/importer.py
Security and Contextual Constraints
The import process is subject to ExecutionContext constraints. Certain operations or filters (like artifact_content retrieval) may be blocked in CORE context but allowed in WORKER context during template rendering associated with imports.
Sources: dev/specs/infp-504-artifact-composition/tasks.md:28-35
Implementation Details
JSON Exporter Logic
The exporter handles pagination and GraphQL execution to prevent memory overflow during large exports.
# Simplified pagination logic for relationship retrieval
# Source: infrahub_sdk/transfer/exporter/json.py:77-87
while has_remaining_items:
offset = (page_number - 1) * page_size
response = await self.client.execute_graphql(
QUERY_RELATIONSHIPS,
variables={
"offset": offset,
"limit": page_size,
"relationship_identifiers": many_relationship_identifiers,
},
branch_name=branch,
tracker=f"query-relationships-page{page_number}",
)
many_relationships.extend(response["Relationship"]["edges"])
Sources: infrahub_sdk/transfer/exporter/json.py:77-87
Relationship Mapping
The system distinguishes between standard relationships and those requiring special handling by checking schema properties:
cardinality == "many"optional is Trueidentifiermust be present.
Sources: infrahub_sdk/transfer/exporter/json.py:44-49
Summary
The Data Import and Export system provides a robust framework for infrastructure portability. By utilizing standardized interfaces and line-delimited formats, it ensures that large-scale infrastructure data can be extracted, versioned in flat files, and re-imported into different Infrahub branches or instances with relationship integrity maintained.