Error Handling and Retries
Relevant source files
The following files were used as context for generating this wiki page:The error handling and retry mechanism within the Terraform AWS Provider is a critical system for managing the asynchronous and eventually consistent nature of AWS APIs. It provides a resilient framework for handling transient errors, polling for resource states, and ensuring that Terraform operations complete successfully despite temporary issues. This system combines a generic, reusable retry and waiter framework with service-specific error identification and handling logic.
The core of this system is built upon the helper/resource package from the Terraform Plugin SDK, particularly its Retry and StateChangeConf functionalities. The provider extends and standardizes the usage of these tools to create robust resource lifecycle management (Create, Read, Update, Delete) functions. This ensures a consistent approach to dealing with common challenges like IAM role propagation delays, API throttling, and waiting for resources to reach a target state (e.g., active, available).
Core Retry and Waiter Framework
The central components for managing retries and waiting for specific states are located in the internal/tfresource package. These provide abstractions over the Terraform Plugin SDK's utilities.
StateChangeConf
The StateChangeConf struct is the primary configuration for any operation that needs to wait for a specific state. It defines the parameters for the polling loop, including target states, pending states, timeouts, and the function to execute for refreshing the state. The provider re-exports this struct from the SDK for internal use.
Sources: internal/tfresource/retry.go:19-21
The diagram below shows the key components of the StateChangeConf struct.
Sources: internal/tfresource/retry.go:19-21
| Field | Description |
|---|---|
Refresh | A function that returns the current state of the resource. |
Pending | A list of states that are considered "in-progress" or "transient". The waiter will continue polling if the resource is in one of these states. |
Target | A list of states that are considered "successful". The waiter will stop and return successfully if the resource reaches one of these states. |
Timeout | The total duration to wait before timing out. |
Delay | The time to wait between Refresh calls. |
Retry | A function to check if a given error is retryable. |
Sources: internal/tfresource/retry.go:19-21
Waiter Functions
The internal/tfresource/waiter.go file provides convenient wrappers for creating and using StateChangeConf.
StateChangeConf: A re-export of the SDK'sresource.StateChangeConf.WaitForState: A re-export of the SDK'sresource.StateChangeConf.WaitForState. This function initiates the polling loop based on the provided configuration.StateRefreshFunc: A type alias for theRefreshfunction signature, promoting code clarity.
Sources: internal/tfresource/waiter.go:13-21
Retry Logic
The Retry function provides a generic mechanism to retry an operation until it succeeds or a timeout is reached. It is often used for handling API calls that might fail due to eventual consistency issues, such as an IAM role not being fully propagated.
The following flowchart illustrates the execution flow of the Retry function.
This diagram shows how the Retry function repeatedly executes a given function, handling retryable errors by waiting and trying again, until it either succeeds, times out, or encounters a non-retryable error.
Sources: internal/tfresource/retry.go:13-15
Custom Error Types and Helpers
The provider defines several utility functions and types to standardize error identification.
General Error Helpers
The internal/tfresource/errors.go file contains helpers to identify common error scenarios.
| Function | Description |
|---|---|
IsNotFound(err error) | Checks if an error is a "not found" error. This is crucial for Read functions to detect when a resource has been deleted outside of Terraform. |
NotFound(err error) | A predicate function that returns true if the error is a NotFoundError. |
TimedOut(err error) | Checks if an error is a timeout error from a StateChangeConf waiter. |
Sources: internal/tfresource/errors.go:12-36
Service-Specific Error Helpers
Many services define their own error-checking helpers to wrap the AWS SDK's error types. This creates a consistent, readable pattern for checking specific API error codes. For example, the EC2 service has a helper to check for AWS error codes.
// internal/service/ec2/errors.go
// isAWSErr is a helper function that returns true if the error is an awserr.Error
// and its Code matches the provided code.
func isAWSErr(err error, code string) bool {
return tfawserr.ErrCodeEquals(err, code)
}
Sources: internal/service/ec2/errors.go:10-13
This pattern is used throughout the provider to handle service-specific transient errors, such as InvalidParameterValueException in Lambda.
Practical Application: Lambda Function
The aws_lambda_function resource provides excellent examples of the error handling and retry framework in action. Creating and updating Lambda functions can involve significant delays due to IAM role propagation and VPC ENI attachment.
Handling IAM Propagation Delay
When a Lambda function is created, the associated IAM role may not be immediately available for Lambda to assume. This results in an InvalidParameterValueException. The resourceFunctionCreate function handles this by wrapping the CreateFunction API call in a resource.Retry block.
// internal/service/lambda/function.go:315-334
err := resource.Retry(lambdaFunctionCreateTimeout, func() *resource.RetryError {
_, err := conn.CreateFunction(params)
if tfawserr.ErrMessageContains(err, lambda.ErrCodeInvalidParameterValueException, "The role defined for the function cannot be assumed by Lambda") {
log.Printf("[DEBUG] Received %s, retrying CreateFunction", err)
return resource.RetryableError(err)
}
// ... other retryable errors
if err != nil {
return resource.NonRetryableError(err)
}
return nil
})
Sources: internal/service/lambda/function.go:315-334
Waiting for Function State
After the initial API call, a Lambda function goes through several states (Pending, Active, Failed). The provider must wait for the function to become Active before proceeding. This is accomplished using a StateChangeConf and its WaitForState method.
The following sequence diagram shows the interaction between the provider, the SDK, and the AWS API when waiting for a Lambda function to become active.
The waitForFunctionCreation and waitForFunctionUpdate functions implement this logic by setting up a StateChangeConf to poll the function's state until it reaches Active or Successful, respectively.
Sources: internal/service/lambda/function.go:1096-1123
Error Handling in Acceptance Tests
The provider's acceptance tests also have a standardized way of handling and expecting errors. The internal/acctest/error.go file defines helpers for this purpose.
ErrorCheck(t *testing.T, serviceID string): This is the primary function used inresource.TestCaseto check for errors. It wrapsErrorCheckSkipMessagesContaining.ErrorCheckSkipMessagesContaining(t *testing.T, msgs ...string): This function returns aresource.ErrorCheckFuncthat will fail the test if an error occurs, unless the error message contains one of the specified substrings. This is useful for ignoring known, non-fatal API errors that can occur during testing.
An example from the Lambda acceptance tests shows how it's used to ignore a specific InvalidParameterValueException:
// internal/service/lambda/function_test.go
func init() {
acctest.RegisterServiceErrorCheckFunc(lambda.EndpointsID, testAccErrorCheckSkip)
}
func testAccErrorCheckSkip(t *testing.T) resource.ErrorCheckFunc {
return acctest.ErrorCheckSkipMessagesContaining(t,
"InvalidParameterValueException: Unsupported source arn",
"InvalidParameterValueException: CompatibleArchitectures are not",
)
}
Sources: internal/service/lambda/function_test.go:34-43, internal/acctest/error.go:11-28
Summary
The error handling and retry system is a cornerstone of the AWS provider's stability. By combining the generic Retry and StateChangeConf patterns from the Terraform SDK with custom error types and service-specific helpers, the provider can gracefully handle the complexities of interacting with a large-scale, distributed system like AWS. This ensures that Terraform operations are resilient to transient failures and can reliably manage resources through their entire lifecycle.