State Management
Relevant source files
The following files were used as context for generating this wiki page:
State management is a core component of Pulumi, responsible for storing and managing the state of infrastructure resources. The state acts as a source of truth, mapping the resources defined in a Pulumi program to the real-world resources provisioned in a cloud provider. This system is designed around a Backend interface, which abstracts the physical storage of state, allowing for different implementations such as the Pulumi Cloud service or self-managed backends like local files or cloud object storage.
The primary data structure for state is the "checkpoint" or "snapshot," which captures the complete state of a stack at a point in time. This includes all resources, their properties, dependencies, and any pending operations. The engine uses this information to compute the minimal set of changes required to align the actual infrastructure with the desired state defined in the code.
Core Interfaces
The state management system is built upon two fundamental interfaces: Backend and Stack.
Backend Interface
The Backend interface defines the contract for storing and retrieving stack information. It abstracts away the details of where and how stack state is persisted.
Sources: pkg/backend/backend.go:42-169
The diagram below illustrates the relationship between the Backend and Stack interfaces.
This class diagram shows the Backend interface, which manages multiple Stack instances, and the key methods each interface exposes.
Key methods of the Backend interface include:
| Method | Description |
|---|---|
GetStack | Retrieves a Stack object for a given stack reference. |
CreateStack | Creates a new stack. |
RemoveStack | Removes a stack. |
ListStacks | Returns a list of available stacks for a user. |
GetHistory | Retrieves the update history for a stack. |
Preview, Update, Refresh, Destroy | Executes the corresponding lifecycle operations on a stack. |
ImportDeployment | Imports a raw deployment (snapshot) into the backend. |
Sources: pkg/backend/backend.go:42-169
Stack Interface
The Stack interface represents a single, isolated instance of a Pulumi program. It provides access to the stack's configuration and its state snapshot.
| Method | Description |
|---|---|
Ref | Returns the unique reference for the stack. |
Config | Returns the stack's configuration map. |
Snapshot | Returns the latest deployment snapshot for the stack. |
Backend | Returns the backend this stack belongs to. |
Sources: pkg/backend/backend.go:172-185
Backend Implementations
Pulumi provides multiple backend implementations to suit different use cases.
HTTP State Backend
The httpstate.Backend is the default implementation that interacts with the Pulumi Cloud service via an HTTP API. It handles authentication, API requests, and state persistence on the service.
Sources: pkg/backend/httpstate/backend.go
The following sequence diagram shows the flow for creating a stack using the httpstate backend.
This diagram illustrates how a CreateStack call on the httpstate.Backend translates to a POST request to the Pulumi Cloud API.
Sources: pkg/backend/httpstate/backend.go:503-548
DIY Backend
The diy.Backend (Do-It-Yourself) provides a way to self-manage state. It stores state as JSON files, typically in a local directory (~/.pulumi/stacks) or in a cloud object storage bucket (like S3, GCS, or Azure Blob Storage) via a URL scheme (e.g., s3://...).
This backend uses a locking mechanism to prevent concurrent modifications to the same stack's state.
Sources: pkg/backend/diy/backend.go
The flowchart below shows how the DIY backend retrieves a stack's state.
This diagram outlines the process diy.Backend follows to safely read a stack's checkpoint file from storage.
Sources: pkg/backend/diy/backend.go:261-295
State Snapshot (Checkpoint)
The core data structure for persisting state is the Checkpoint, which contains a Snapshot. A snapshot represents the complete state of a stack's resources at a specific point in time.
Sources: pkg/resource/stack/checkpoint.go:34-42
Snapshot Structure
The deploy.Snapshot struct contains all the necessary information to reconstruct the state of the infrastructure.
| Field | Type | Description |
|---|---|---|
Magic | string | A magic string to identify the checkpoint file format. |
Version | int | The version of the checkpoint format. |
Time | time.Time | The timestamp of when the snapshot was taken. |
Manifest | Manifest | Contains information about the plugins used for the deployment. |
SecretsManager | secrets.Manager | The secrets manager used to encrypt/decrypt secret values. |
Resources | []*resource.State | A list of all resources in the stack. |
PendingOperations | []resource.Operation | A list of operations that were pending but did not complete. |
Sources: pkg/resource/stack/checkpoint.go:121-137
Resource State
Each resource within the Resources list is represented by a resource.State struct.
This ER diagram shows the relationship between a Snapshot and its resource.State objects, including dependencies and parent-child relationships.
Key fields of resource.State:
- URN (Uniform Resource Name): A unique, logical identifier for the resource within Pulumi.
- Type: The resource's type token (e.g.,
aws:s3/bucket:Bucket). - ID: The physical ID of the resource in the cloud provider.
- Inputs: The input properties used to create the resource.
- Outputs: The output properties returned by the provider after creation.
- Parent: The URN of the parent resource, if any.
- Dependencies: A list of URNs of resources this resource depends on.
- Provider: A reference to the provider instance used to manage this resource.
Sources: pkg/resource/stack/checkpoint.go:133, pkg/resource/stack/checkpoint.go:167-190
State Integrity and Repair
Pulumi includes mechanisms to ensure the integrity of the state file and to repair it if it becomes invalid. An invalid state can occur if, for example, a resource is listed before its dependencies.
Verification
The Snapshot.VerifyIntegrity() method checks for common consistency issues, such as ensuring that all dependencies of a resource appear before the resource itself in the resource list.
Sources: pkg/resource/stack/checkpoint.go:278-306
Repair
The pulumi state repair command provides an automated way to fix invalid state files. It operates by loading the snapshot with integrity checks disabled, attempting to fix it, and then writing it back if the repair is successful.
Sources: pkg/cmd/pulumi/state/state_repair.go
The repair process involves two main steps:
- Topological Sort (
Toposort): Reorders the resources in the snapshot to ensure they respect their dependency graph. - Pruning (
Prune): Removes any dangling references, such as dependencies on resources that no longer exist in the state.
Sources: pkg/cmd/pulumi/state/state_repair.go:132-139, pkg/resource/stack/checkpoint.go:308-360
The following flowchart illustrates the logic of the state repair command.
This flowchart shows the decision-making process within the pulumi state repair command, from loading the invalid state to confirming and writing the repaired version.
Sources: pkg/cmd/pulumi/state/state_repair.go:94-177
Conclusion
Pulumi's state management system is a robust and flexible architecture centered around the Backend interface and the Snapshot data structure. This design allows for different storage backends, from the managed Pulumi Cloud service to self-hosted solutions, while maintaining a consistent model for tracking infrastructure state. The system includes built-in verification and repair mechanisms to ensure state integrity, providing reliability for infrastructure deployments.