Troubleshooting
Relevant source files
The following files were used as context for generating this wiki page:
Troubleshooting within the Terraform AWS Provider encompasses a range of activities, from handling transient API errors and eventual consistency issues to identifying and categorizing software bugs. The provider employs internal mechanisms for resilience, such as retry logic, and relies on a structured system for issue management and comprehensive testing to ensure stability.
This document outlines the key components and workflows related to troubleshooting. It covers the programmatic error handling and retry systems built into the provider, the process for triaging and labeling issues within the repository, and the testing patterns used to verify resource lifecycle management and prevent regressions.
Error Handling
The provider includes a dedicated package for handling common error patterns, especially those returned by the AWS SDK. These helpers allow resource logic to distinguish between different types of failures, such as a resource not being found versus a timeout or other API error.
Sources: internal/tfresource/errors.go, internal/service/codecommit/approval_rule_template_test.go:203-222
Key Error Functions
The tfresource package provides functions to check for specific error conditions. This is critical in Destroy functions, where a NotFound error indicates a successful deletion.
| Function | Description |
|---|---|
NotFound | Checks if an error, or its underlying cause, indicates that a resource was not found. |
TimedOut | Checks if an error, or its underlying cause, is a timeout error, typically from a Retry operation. |
Sources: internal/tfresource/errors.go:11-26
The following diagram illustrates the logic used in CheckDestroy test functions to verify resource deletion.
This diagram shows the process of verifying a resource has been destroyed by checking for a 'Resource Not Found' error from the API.
Sources: internal/service/codecommit/approval_rule_template_test.go:203-222, internal/service/connect/contact_flow_module_test.go:203-225, internal/service/schemas/schema_test.go:220-241
Retry and Stabilization Logic
Interacting with cloud APIs often requires handling transient errors and waiting for resources to reach a desired state. The provider uses a generic retry mechanism to manage these situations.
Sources: internal/tfresource/retry.go
The Retry Function
The core of this system is the Retry function, which accepts a Retryable function and a timeout duration. It repeatedly executes the Retryable function until it returns a non-error result or the timeout is exceeded. The Retryable function can return a special RetryError to control the retry loop.
The RetryError struct contains the original error and allows the operation to signal whether the error is considered terminal (should stop retries) or transient (should continue retrying).
The following diagram shows the sequence of a retryable operation.
This sequence illustrates how the Retry function handles success, retryable errors, and terminal errors during an operation.
Sources: internal/tfresource/retry.go:10-48
Issue Triage and Management
Troubleshooting extends to managing and categorizing issues reported by the community. The project uses a set of predefined GitHub labels to classify the nature of issues and pull requests, which helps maintainers prioritize and address problems efficiently.
Sources: infrastructure/repository/labels-workflow.tf
Common Troubleshooting Labels
The following labels are directly related to identifying and resolving problems within the provider:
| Label | Description |
|---|---|
bug | Addresses a defect in current functionality. |
crash | Results from or addresses a Terraform crash or kernel panic. |
eventual-consistency | Pertains to eventual consistency issues. |
regression | Pertains to a degraded workflow resulting from an upstream patch or internal enhancement. |
needs-triage | Waiting for first response or review from a maintainer. |
stale | Old or inactive issues managed by automation, if no further action taken these will get closed. |
timeouts | Pertains to timeout increases. |
waiting-response | Maintainers are waiting on response from community or contributor. |
technical-debt | Addresses areas of the codebase that need refactoring or redesign. |
Sources: infrastructure/repository/labels-workflow.tf:1-190
Acceptance Testing for Resilience
Acceptance tests are a primary tool for troubleshooting and preventing issues. They validate resource behavior and catch regressions. Two key patterns are used to ensure resources are robust against common failure modes.
Resource Disappearance Tests
These tests verify that the provider can recover correctly if a resource managed by Terraform is deleted outside of Terraform (e.g., via the AWS Console). The test creates a resource, programmatically deletes it, and then runs terraform apply again, expecting Terraform to detect the change and plan to recreate the resource.
The flow of a disappearance test is as follows:
This diagram shows the workflow for testing how the provider handles out-of-band resource deletion.
Sources: internal/service/codecommit/approval_rule_template_test.go:56-74, internal/service/connect/contact_flow_module_test.go:138-156, internal/service/schemas/schema_test.go:94-112
Resource Destruction Tests
Every resource includes a CheckDestroy function in its acceptance tests. This function runs after terraform destroy and confirms that the resource has been successfully removed from the AWS account. It does this by calling the AWS API to get the resource and asserting that it receives a "not found" error.
This test pattern is crucial for preventing "dangling" resources that are removed from Terraform state but not from the cloud provider.
Sources: internal/service/codecommit/approval_rule_template_test.go:203-222, internal/service/connect/contact_flow_module_test.go:203-225, internal/service/schemas/schema_test.go:220-241