Code Generation

Relevant source files

The following files were used as context for generating this wiki page:

• internal/generate/listpages/main.go
• internal/generate/tags/README.md
• internal/service/amplify/generate.go
• internal/service/ec2/generate/createtags/main.go
• internal/service/waf/generate.go
• internal/generate/tags/README_listtags.md
• internal/service/codecommit/approval_rule_template_test.go

The terraform-provider-aws repository utilizes a code generation framework to automate the creation of boilerplate code for various AWS services. This system ensures consistency in handling common patterns like API pagination and resource tagging. The generation process is triggered by the standard Go toolchain command go generate ./..., which is often wrapped in a make gen target.

The core of this system consists of several generator programs located in the internal/generate directory. These programs are invoked via //go:generate directives placed in generate.go files within specific service packages (e.g., internal/service/amplify/generate.go). These directives pass flags to the generator programs to customize the output for the specific needs of each AWS service.

Generator Invocation

Code generation is initiated from generate.go files located within each service's package directory. These files contain one or more //go:generate comments that specify which generator to run and what flags to use.

Sources: internal/service/amplify/generate.go, internal/service/waf/generate.go

For example, the amplify service uses generators for both pagination (listpages) and tagging (tags):

//go:generate go run ../../generate/listpages/main.go -ListOps=ListApps
//go:generate go run ../../generate/tags/main.go -ListTags -ServiceTagsMap -UpdateTags
// ONLY generate directives and package declaration! Do not add anything else to this file.

package amplify

Sources: internal/service/amplify/generate.go:1-5

Similarly, the waf service uses the same generators but with different flags tailored to its API:

//go:generate go run ../../generate/listpages/main.go -ListOps=ListByteMatchSets,ListGeoMatchSets... -Paginator=NextMarker
//go:generate go run ../../generate/tags/main.go -ListTags -ListTagsInIDElem=ResourceARN ... -UpdateTags
// ONLY generate directives and package declaration! Do not add anything else to this file.

package waf

Sources: internal/service/waf/generate.go:1-5

The following diagram illustrates the invocation flow from the make command to the final generated Go file.

Sources: internal/generate/tags/README.md, internal/generate/listpages/main.go

Pagination Helper Generation (listpages)

The listpages generator is responsible for creating helper functions that simplify handling of paginated list operations from the AWS Go SDK. It generates ...Pages and ...PagesWithContext functions that loop through API call results until all pages have been retrieved.

Sources: internal/generate/listpages/main.go

Execution Flow

The generator performs the following steps:

1. Parses command-line flags (-ListOps, -Paginator, -Export).
2. Determines the AWS service name from the current working directory.
3. Loads and parses the source code of the corresponding AWS Go SDK package.
4. For each function name provided in -ListOps, it finds the function signature in the parsed SDK code.
5. Uses a Go template to generate the ...Pages and ...PagesWithContext wrapper functions.
6. Formats the generated Go code and writes it to list_pages_gen.go.

Sources: internal/generate/listpages/main.go:50-93

The following diagram visualizes the internal process of the listpages generator.

Sources: internal/generate/listpages/main.go:73-93, 185-201

Configuration Flags

The behavior of the listpages generator is controlled by the following command-line flags.

Flag	Description	Default	Example
ListOps	A comma-separated list of AWS SDK list operation function names to generate wrappers for.	""	-ListOps=ListApps,ListJobs
Paginator	The name of the pagination token field in the output struct of the list operation.	NextToken	-Paginator=NextMarker
Export	If set, the generated functions will be exported (start with an uppercase letter).	false	-Export

Sources: internal/generate/listpages/main.go:21-25

Generated Code

The generator uses a template to produce a consistent function structure that handles the pagination loop.

// From functionTemplate constant
func {{ .Name }}Pages(conn {{ .RecvType }}, input {{ .ParamType }}, fn func({{ .ResultType }}, bool) bool) error {
	// ...
}

func {{ .Name }}PagesWithContext(ctx context.Context, conn {{ .RecvType }}, input {{ .ParamType }}, fn func({{ .ResultType }}, bool) bool) error {
	for {
		output, err := conn.{{ .AWSName }}WithContext(ctx, input)
		if err != nil {
			return err
		}

		lastPage := aws.StringValue(output.{{ .Paginator }}) == ""
		if !fn(output, lastPage) || lastPage {
			break
		}

		input.{{ .Paginator }} = output.{{ .Paginator }}
	}
	return nil
}

Sources: internal/generate/listpages/main.go:250-273

Tag Management Generation

A significant part of the code generation system is dedicated to handling AWS resource tags. A primary generator at internal/generate/tags/main.go and several specialized sub-generators (e.g., createtags) create consistent functions for tagging operations like ListTags, TagResource, and UntagResource.

Sources: internal/generate/tags/README.md

tags Generator

This is the main generator for tag-related functions. It is highly configurable via flags to accommodate the wide variety of tagging implementations across different AWS services.

Configuration Flags

The table below lists some of the key flags for the main tags generator.

Flag	Description	Example Use
GetTag	Generate a GetTag function.	-GetTag
ListTags	Generate a ListTags function.	-ListTags
UpdateTags	Generate an UpdateTags function.	-UpdateTags
ServiceTagsMap	Generate service tags from a map.	-ServiceTagsMap
ServiceTagsSlice	Generate service tags from a slice of structs.	-ServiceTagsSlice
ListTagsOp	Specifies the name of the list tags API operation.	-ListTagsOp=ListTags
ListTagsOutTagsElem	The field name for tags in the list tags output struct.	-ListTagsOutTagsElem=TagList
TagOp	The name of the tag resource API operation.	-TagOp=AddTags
UntagOp	The name of the untag resource API operation.	-UntagOp=DeleteTags
TagTypeKeyElem	The key field name within the service's tag struct.	-TagTypeKeyElem=TagKey
TagTypeValElem	The value field name within the service's tag struct.	-TagTypeValElem=TagValue

Sources: internal/generate/tags/README.md

createtags Generator

Located at internal/service/ec2/generate/createtags/main.go, this is a specialized generator for creating tags for a new resource, specifically tailored for EC2 but designed to be reusable.

Sources: internal/service/ec2/generate/createtags/main.go

Configuration Flags

Flag	Description	Default
RetryCreateOnNotFound	Retry the create operation if the resource is not found.	true
TagOp	The name of the tag creation function.	CreateTags
TagOpBatchSize	The batch size for the tag operation.	""
TagInIDElem	The input field for the resource identifier(s).	Resources
TagInIDNeedSlice	Whether the identifier input field requires a slice.	true
TagInTagsElem	The input field for the tags themselves.	Tags

Sources: internal/service/ec2/generate/createtags/main.go:19-27

Generated Code (CreateTags)

This generator produces a CreateTags function based on its template. The function handles batching (if configured) and can retry on NotFound errors to handle eventual consistency issues.

// Simplified from templateBody
func CreateTags(conn *ec2.EC2, identifier string, tagsMap interface{}) error {
	tags := tftags.New(tagsMap)

	input := &ec2.CreateTagsInput{
		Resources: aws.StringSlice([]string{identifier}),
		Tags:      Tags(tags.IgnoreAWS()),
	}

	_, err := tfresource.RetryWhenNotFound(EventualConsistencyTimeout, func() (interface{}, error) {
		// ... call conn.CreateTags(input)
	})

	if err != nil {
		return fmt.Errorf("error tagging resource (%s): %w", identifier, err)
	}

	return nil
}

Sources: internal/service/ec2/generate/createtags/main.go:114-159