Data Management

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• internal/cloud/backend.go
• internal/plans/planproto/planfile.proto
• internal/rpcapi/terraform1/dependencies/dependencies.pb.go
• internal/builtin/providers/terraform/resource_data_test.go
• internal/cloud/e2e/migrate_state_remote_backend_to_tfc_test.go
• internal/cloud/testdata/plan-json-full/plan-redacted.json

Terraform's core function is to manage infrastructure as code, which involves several key data management concepts. This includes describing infrastructure in configuration files, generating execution plans to preview changes, and storing the state of managed infrastructure. Data is handled through structured formats like execution plans and state files, and managed via backends that determine where this data is stored and how operations are executed.

A central piece of data management is the Execution Plan, which shows what Terraform will do when an apply command is run. Another is the Resource Graph, which maps all resources and their dependencies, allowing for parallel and efficient infrastructure provisioning. The system supports various backends for state storage, with the cloud backend offering integration with HCP Terraform and Terraform Enterprise for advanced data management capabilities, including remote operations and state migration.

Sources: README.md:10-25, internal/cloud/backend.go

Core Data Structures

Terraform relies on well-defined data structures for its operations. These are primarily defined using Protocol Buffers, ensuring a consistent and versioned format for plans, state, and provider schemas.

Terraform Plan File

A Terraform plan is a structured file that contains the proposed changes to the infrastructure. The root message for this structure is Plan, defined in the protobuf schema.

The diagram below illustrates the high-level structure of a Plan message.

Key components of a plan include:

• version: A version number for the plan format itself.
• resource_changes: A list of proposed changes to resource instances.
• resource_drift: A list of changes detected between the prior state and the real-world infrastructure.
• output_changes: Proposed changes to root module outputs.
• variables: The input variable values used to generate the plan.
• backend: A description of the backend configuration at the time of the plan.

A plan can also be represented in JSON format, which includes provider schemas, planned values, and resource changes.

Sources: internal/plans/planproto/planfile.proto:8-100, internal/cloud/testdata/plan-json-full/plan-redacted.json

Provider Schema

Provider schemas define the structure of resources, data sources, and the provider's own configuration. This includes attributes, their types, and whether they are optional, required, or computed. The schema is defined using a ProviderSchema message.

This diagram shows the hierarchical structure of a provider schema.

• ProviderSchema: The top-level schema for a provider.
• Schema: Describes the schema for a single object, like a resource or data source.
• Schema_Block: Defines a configuration block, containing attributes and nested blocks.
• Schema_Attribute: Defines a single argument or attribute within a block.

This structure is used internally to validate configurations and understand the data types for each resource attribute.

Sources: internal/rpcapi/terraform1/dependencies/dependencies.pb.go:1075-1110, internal/rpcapi/terraform1/dependencies/dependencies.pb.go:1210-1437

Cloud Backend Data Management

The cloud backend provides integration with HCP Terraform and Terraform Enterprise, offering robust data management features like remote state storage, remote operations, and workspace management.

Sources: internal/cloud/backend.go:42-50

Configuration

The cloud backend is configured within the terraform block of a configuration file. The configuration determines how Terraform CLI connects to and interacts with the remote environment.

The following diagram shows the flow for resolving the final cloud configuration from HCL files and environment variables.

The key configuration options are summarized below:

Attribute	Type	Environment Variable	Description
hostname	string	TF_CLOUD_HOSTNAME	The hostname of the Terraform Enterprise or HCP Terraform instance. Defaults to app.terraform.io.
organization	string	TF_CLOUD_ORGANIZATION	The name of the organization containing the workspaces. This is required.
token	string	N/A	The authentication token. It's recommended to use terraform login instead of setting this directly.
workspaces	block	TF_WORKSPACE	A block to configure workspace mapping using name, tags, or project.

Sources: internal/cloud/backend.go:153-181, internal/cloud/backend.go:561-662

Workspace Mapping

The cloud backend maps local CLI workspaces to remote workspaces in HCP Terraform using different strategies. This is configured inside the workspaces block.

Mapping Strategies

1. Name Strategy: A single remote workspace is specified by name. This conflicts with the tags attribute and the TF_WORKSPACE environment variable.
2. Tags Strategy: Workspaces are selected based on a set of tags. This can be a set of strings (key-only tags) or a map of strings (key-value tags).
3. None Strategy: If neither name nor tags is provided, the TF_WORKSPACE environment variable is used to determine the workspace name.

The following diagram illustrates the logic for determining the workspace mapping strategy.

This logic is implemented in the WorkspaceMapping.Strategy() method. The backend uses this strategy to list, select, or create remote workspaces. For example, when using the tags strategy, the Workspaces() method lists all remote workspaces matching the specified tags.

Sources: internal/cloud/backend.go:1210-1237, internal/cloud/backend.go:819-883

State Migration

Terraform supports migrating state from one backend to another, such as from a local or remote backend to the cloud backend. This process is typically initiated during terraform init when a backend change is detected.

The sequence diagram below shows a simplified flow of migrating state from a remote backend to the cloud backend.

This process is demonstrated in end-to-end tests, which simulate user input and verify that the state is correctly copied to the new HCP Terraform workspace.

Sources: internal/cloud/e2e/migrate_state_remote_backend_to_tfc_test.go:23-65

terraform_data Resource

The built-in terraform provider includes a terraform_data resource. This resource is a data store that can be used to hold arbitrary values and trigger updates based on changes to those values.

Its primary attributes are:

• input: An arbitrary value or structure provided by the user.
• output: A computed attribute that always mirrors the value of input.
• triggers_replace: A value that, when changed, forces the resource to be replaced. This is useful for triggering updates on other resources that depend on terraform_data.
• id: A unique ID for the resource instance, computed on creation.

The behavior of this resource is defined by its plan and apply logic. During a plan, if the input changes, the output is marked as unknown. If triggers_replace changes, the entire resource is planned for replacement. During apply, the output is set to the value of input, and a new id is generated if the resource is being created.

The following test case illustrates how an update to triggers_replace forces a replacement of the resource.

// Source: internal/builtin/providers/terraform/resource_data_test.go:149-169
"update-trigger": {
    prior: cty.ObjectVal(map[string]cty.Value{
        "input":            cty.StringVal("input"),
        "output":           cty.StringVal("input"),
        "triggers_replace": cty.NullVal(cty.DynamicPseudoType),
        "id":               cty.StringVal("not-quite-unique"),
    }),
    proposed: cty.ObjectVal(map[string]cty.Value{
        "input":            cty.StringVal("input"),
        "output":           cty.StringVal("input"),
        "triggers_replace": cty.StringVal("new-value"),
        "id":               cty.StringVal("not-quite-unique"),
    }),
    planned: cty.ObjectVal(map[string]cty.Value{
        "input":            cty.StringVal("input"),
        "output":           cty.UnknownVal(cty.String),
        "triggers_replace": cty.StringVal("new-value"),
        "id":               cty.UnknownVal(cty.String).RefineNotNull(),
    }),
},

This resource is particularly useful for managing data dependencies and forcing actions in a declarative way within a Terraform configuration.

Sources: internal/builtin/providers/terraform/resource_data_test.go

Conclusion

Terraform's data management is a comprehensive system built on structured, versioned data formats for plans and state. The cloud backend extends these capabilities by integrating with HCP Terraform, providing centralized state management, remote operations, and flexible workspace mapping. Specialized resources like terraform_data offer fine-grained control over data flow and dependencies within a configuration. Together, these features enable safe, efficient, and version-controlled infrastructure management.

Sources: README.md:10-25, internal/cloud/backend.go