Batch Processing

Relevant source files

The following files were used as context for generating this wiki page:

• infrahub_sdk/batch.py
• tests/unit/sdk/test_batch.py
• infrahub_sdk/client.py
• AGENTS.md
• dev/specs/001-end-user-cli/tasks.md

Batch Processing provides a mechanism to group multiple infrastructure data operations into atomic units. The SDK uses this system to manage large volumes of data efficiently by reducing the number of individual API calls to the Infrahub server. It abstracts complex interactions into native Python objects, providing built-in support for tracking changes and managing execution order.

The system supports both asynchronous and synchronous execution through InfrahubBatch and InfrahubBatchSync classes. This ensures parity across different development environments, allowing users to automate within Infrahub pipelines, integrate with external systems, or build custom applications.
Sources: AGENTS.md:7-15, infrahub_sdk/batch.py:1-20

Architecture and Components

The batch processing architecture centers on the InfrahubBatch class and its synchronous counterpart. These classes act as containers for infrastructure nodes and track their intended states.

Key Classes and Data Structures

Component	Description
InfrahubBatch	The primary async container for tracking and executing multiple node operations.
InfrahubBatchSync	The synchronous version of the batch container for non-async environments.
InfrahubNode	The core data model representing infrastructure objects tracked within a batch.
InfrahubClient	The underlying client used by the batch to communicate with the Infrahub API.

Sources: infrahub_sdk/batch.py:24-40, AGENTS.md:39-45

Class Relationships

The following diagram illustrates the relationship between the batch containers, the client, and the infrastructure nodes.

The batch system maintains a reference to the InfrahubClient to perform network operations and holds a collection of InfrahubNode objects to be processed.
Sources: infrahub_sdk/batch.py:24-60, infrahub_sdk/client.py

Operation Lifecycle

A batch operation follows a specific lifecycle: initialization, node registration, and execution. Users register nodes with the batch, and the batch determines the necessary API calls during execution.

Registration and Execution Flow

1. Initialize: You create a batch instance using the client's create_batch() method.
2. Register: You add nodes to the batch using the add() method.
3. Execute: You call execute() to send the accumulated changes to the server.

The flow ensures that multiple nodes are collected before any network activity occurs, optimizing performance for large datasets.
Sources: infrahub_sdk/batch.py:45-80, tests/unit/sdk/test_batch.py:10-25

CLI Integration

Batch processing is heavily utilized by the infrahubctl CLI for operations involving file inputs. When creating or updating objects from YAML files, the CLI leverages batching to handle multiple objects defined in a single source.

• Create Command: Processes multiple objects from objects.yaml using batch results for a summary.
• Update Command: Applies changes to multiple existing objects via file-based updates.

Sources: dev/specs/001-end-user-cli/tasks.md:105-135

CLI Batch Summary Feature

Feature	Description	File Reference
Batch Summary	Displays a summary of results after processing a file of objects.	infrahub_sdk/ctl/commands/create.py
File Mode	Loads objects via ObjectFile and processes them as a batch.	infrahub_sdk/ctl/commands/create.py

Sources: dev/specs/001-end-user-cli/tasks.md:107-111

Implementation Details

The implementation ensures that InfrahubBatch handles the complexity of API interactions. It tracks which nodes are new, which are modified, and which are marked for deletion.

# From infrahub_sdk/batch.py
class InfrahubBatch:
    def __init__(self, client: InfrahubClient):
        self.client = client
        self.nodes: list[InfrahubNode] = []

    def add(self, node: InfrahubNode) -> None:
        """Adds a node to the batch for tracking."""
        self.nodes.append(node)

    async def execute(self) -> None:
        """Executes all tracked operations against the API."""
        # Logic to send nodes to server

Sources: infrahub_sdk/batch.py:24-55

Batch processing provides a high-level abstraction for managing infrastructure data at scale. By grouping operations into a single execution context, it reduces API overhead and ensures consistent data application across the Infrahub environment.