Orchestrating Kubernetes with EKS

Relevant source files

The following files were used as context for generating this wiki page:

• packages/@aws-cdk/aws-eks-legacy/README.md
• packages/@aws-cdk/aws-eks/package.json
• packages/@aws-cdk/lambda-layer-kubectl/package.json
• README.md
• design/construct-tree.md

The AWS Cloud Development Kit (CDK) provides the @aws-cdk/aws-eks construct library to define and manage Amazon Elastic Kubernetes Service (EKS) clusters programmatically. This library abstracts the complexity of setting up an EKS cluster, allowing developers to define the control plane, worker node capacity, and Kubernetes resource manifests directly within their CDK application using high-level, object-oriented APIs.

This approach integrates Kubernetes workload definitions with AWS infrastructure provisioning. It streamlines the entire lifecycle, from cluster creation and IAM integration to deploying applications via raw manifests or Helm charts. The library uses a combination of standard AWS CloudFormation resources and custom resources to provide programmatic control over the cluster, including managing the aws-auth ConfigMap and executing kubectl commands during deployment.

Sources: packages/@aws-cdk/aws-eks-legacy/README.md, README.md

Core Component: The EKS Cluster

The central component of the library is the eks.Cluster construct. Instantiating this construct provisions a complete EKS cluster, including a dedicated VPC by default.

A key architectural decision when creating a cluster is the kubectlEnabled property.

• kubectlEnabled: true (Default): The cluster is provisioned using a CloudFormation custom resource. This approach grants the CDK programmatic access to the cluster via kubectl, enabling features like applying Kubernetes manifests (cluster.addResource) and managing IAM-to-RBAC mappings (cluster.awsAuth) directly from the CDK app. The IAM entity that creates the stack does not automatically get system:masters permissions.
• kubectlEnabled: false: The cluster is provisioned using the standard AWS::EKS::Cluster CloudFormation resource. This disables programmatic kubectl access from the CDK. The IAM entity that creates the stack is automatically granted system:masters permissions. However, features like cluster.addResource, cluster.addChart, and cluster.awsAuth will not be available.

Sources: packages/@aws-cdk/aws-eks-legacy/README.md

The following diagram illustrates the high-level provisioning flow when creating an eks.Cluster.

This diagram shows how the kubectlEnabled flag determines the provisioning mechanism for the EKS cluster.
Sources: packages/@aws-cdk/aws-eks-legacy/README.md

Managing Cluster Capacity

The eks.Cluster construct provides multiple ways to configure and add worker node capacity.

Sources: packages/@aws-cdk/aws-eks-legacy/README.md

Default and Custom Capacity

By default, a cluster is created with two m5.large instances. This can be customized or disabled.

Method	Description	Example
defaultCapacity	Specifies the number of instances for the default capacity group during cluster initialization. Set to 0 to disable.	new eks.Cluster(this, 'cluster', { defaultCapacity: 10 });
defaultCapacityInstance	Sets the EC2 instance type for the default capacity.	defaultCapacityInstance: new ec2.InstanceType('m2.xlarge')
addCapacity()	Adds a new Auto Scaling Group with a specified instance type and capacity to the cluster.	cluster.addCapacity('custom-nodes', { instanceType: new ec2.InstanceType('t2.medium') });
addAutoScalingGroup()	Adds a pre-configured EC2 Auto Scaling Group to the cluster.	cluster.addAutoScalingGroup(myAsg, { bootstrapOptions: { ... } });

Sources: packages/@aws-cdk/aws-eks-legacy/README.md

Spot Capacity

You can provision worker nodes using EC2 Spot Instances by specifying the spotPrice property when adding capacity. When using spot instances, the nodes are automatically labeled with lifecycle=Ec2Spot and tainted with PreferNoSchedule. The Spot Termination Handler is also installed as a DaemonSet to gracefully handle spot instance terminations.

Sources: packages/@aws-cdk/aws-eks-legacy/README.md

Node Bootstrapping

When adding capacity, the nodes must be bootstrapped to join the EKS cluster. The bootstrapOptions property allows customization of the /etc/eks/bootstrap.sh script. This can be used to pass extra arguments to the kubelet, such as custom node labels or taints.

cluster.addCapacity('spot', {
  instanceType: new ec2.InstanceType('t3.large'),
  bootstrapOptions: {
    kubeletExtraArgs: '--node-labels foo=bar,goo=far',
    awsApiRetryAttempts: 5
  }
});

Sources: packages/@aws-cdk/aws-eks-legacy/README.md

Deploying Kubernetes Resources

With kubectlEnabled: true, the CDK can deploy and manage Kubernetes resources, including raw manifests and Helm charts, as part of the cdk deploy process.

Kubernetes Manifests

The cluster.addResource() method (or the KubernetesResource construct) applies one or more Kubernetes resource manifests to the cluster. These manifests are defined as JavaScript objects. Since these are implemented as CloudFormation resources, deleting the resource from the CDK app will cause kubectl delete to be executed on the next deployment.

cluster.addResource('mypod', {
  apiVersion: 'v1',
  kind: 'Pod',
  metadata: { name: 'mypod' },
  spec: {
    containers: [
      {
        name: 'hello',
        image: 'paulbouwer/hello-kubernetes:1.5',
        ports: [ { containerPort: 8080 } ]
      }
    ]
  }
});

Sources: packages/@aws-cdk/aws-eks-legacy/README.md

Helm Charts

The cluster.addChart() method (or the HelmChart construct) deploys a Helm chart from a specified repository. The CDK manages the lifecycle of the Helm release using helm upgrade --install for creations/updates and helm uninstall for deletions. The underlying mechanism uses a Lambda function that includes helm and kubectl binaries, provided by the @aws-cdk/lambda-layer-kubectl package.

Sources: packages/@aws-cdk/aws-eks-legacy/README.md, packages/@aws-cdk/lambda-layer-kubectl/package.json

The deployment flow for a Helm chart is shown below.

This sequence diagram shows how a cdk deploy command triggers a CloudFormation Custom Resource, which in turn invokes a Lambda function to execute Helm commands against the EKS cluster.
Sources: packages/@aws-cdk/aws-eks-legacy/README.md, packages/@aws-cdk/aws-eks/package.json

IAM and Access Control

The library provides robust mechanisms for mapping AWS IAM principals (users and roles) to Kubernetes RBAC groups.

Masters Role

To grant manual kubectl access to the cluster, you can define a mastersRole. This IAM role will be mapped to the system:masters Kubernetes group. When deployed, the CDK provides an aws eks update-kubeconfig command as a CloudFormation output, which configures kubectl to use this role.

// Define an IAM role that can be assumed by account users
const clusterAdmin = new iam.Role(this, 'AdminRole', {
  assumedBy: new iam.AccountRootPrincipal()
});

// Define the cluster and map the role to the "masters" RBAC group
new eks.Cluster(this, 'Cluster', {
  mastersRole: clusterAdmin
});

Sources: packages/@aws-cdk/aws-eks-legacy/README.md

AWS Auth ConfigMap Management

The cluster.awsAuth property provides an API to manage the aws-auth ConfigMap within Kubernetes. This is the standard mechanism for EKS to authorize IAM principals. This feature is only available if kubectlEnabled is true.

The following diagram illustrates the relationship between IAM, the aws-auth ConfigMap, and Kubernetes RBAC.

This diagram shows how the addMastersRole method in the CDK app translates an IAM Role into an entry in the aws-auth ConfigMap, granting it system:masters privileges in the cluster.
Sources: packages/@aws-cdk/aws-eks-legacy/README.md

Convenience methods like addUserMapping and addMastersRole simplify this process. Additionally, when capacity is added via addCapacity or addAutoScalingGroup, the instance role of the nodes is automatically mapped to allow them to join the cluster.

Sources: packages/@aws-cdk/aws-eks-legacy/README.md

Conclusion

The @aws-cdk/aws-eks construct library provides a comprehensive, infrastructure-as-code solution for managing Amazon EKS. By representing both the AWS infrastructure (like VPCs and IAM roles) and Kubernetes resources (like deployments and services) within a single programmatic framework, it unifies the development and operational experience. It abstracts away significant boilerplate and complexity, enabling developers to build, deploy, and manage containerized applications on Kubernetes with the full power and expressiveness of modern programming languages.

Sources: packages/@aws-cdk/aws-eks-legacy/README.md, README.md