AWS Feed
Automate Amazon EKS upgrades with infrastructure as code
In this post, we explain how to use managed node groups to upgrade Amazon Elastic Kubernetes Service (Amazon EKS) cluster nodes in parallel from 1.19 to 1.20. Users can use the AWS Service Catalog to support an automated workflow with granular controls. This capability provides the option to upgrade the control plane and nodes. A list of Amazon EKS features, based on Kubernetes version, can be found in the open source publicly available change log document.
The solution
Multiple components, configurations, and applications run in Kubernetes clusters. Operations and developers want to be able to use new features that help optimize the environment. Upgrading a Kubernetes cluster can be a manual and time-consuming effort if developers must perform common validation and procedural steps.
You can upgrade clusters through the AWS Management Console or through the command line interface with eksctl or kubectl tools. Upon Amazon EKS cluster enablement, there is no auditing in AWS CloudTrail. Both options are manual and are not logged for tracking and auditing.
The solution described in this article automates the Amazon EKS upgrade process using managed node groups, a launch template, infrastructure as code, and container images with Python code. The three container images issue eksctl and kubectl commands along with status checks. The cluster upgrade is encapsulated into simple parameters. In the background, the commands and configuration are captured in logs.
The AWS Lambda container image types are:
- Control plane upgrade: Upgrades the Amazon EKS control plane to the specified version.
- DaemonSet upgrade image: Upgrades the aws-node, coredns, and kube-proxy DaemonSets to match the control plane version.
- Node group upgrade image: Upgrades the node group launch template version to match the Amazon EKS cluster version.
The control plane workflow is shown in the preceding image. Shown in the black-numbered circles in the image:
- An AWS Lambda container image is created using AWS CloudFormation that sends a control plane upgrade request and stores update ID.
- The AWS Lambda container request is received and starts a control plane upgrade.
- The AWS Lambda container takes the update ID and checks the status of the control plane upgrade progress.
- If the control plane upgrade is successful, an AWS CloudFormation template deploys a managed node group with Kubernetes version 1.20. If the Amazon EKS cluster is unhealthy, the control upgrade will fail.
The data plane workflow shown in the yellow-numbered circles:
- The control plane upgrade was successful and an AWS CloudFormation template deployed a managed node group with Kubernetes version 1.20.
- The AWS Lambda container request is received and starts a data plane upgrade on the different types of DaemonSets one at a time.
- If all upgrades on the data plane are successful, an AWS CloudFormation template deploys a managed node group with Kubernetes version 1.20. If the Amazon EKS cluster is in an unhealthy status, the control upgrade fails.
- The AWS Lambda container request is received and starts a data plane upgrade on the managed node groups.
Prerequisites
To follow the steps to provision the pipeline deployment, you must have the following:
- An AWS account with local credentials properly configured (typically under
~/.aws/credentials
). - The latest version of the AWS CLI. For more information, refer to the documentation for installing, updating, and uninstalling the AWS CLI.
- An eksctl client to interact with the cluster’s API.
- A Docker client to build container images.
- A git client to clone the source code provided.
Note: View the code within the GitHub repository. AWS best practices recommend reducing the AWS Identity and Access Management (IAM) policy to meet your company’s requirements. These permissions are for demonstration only and are not production ready.
Walkthrough
- Clone the source code repository found in the following location:
- Create an Amazon Elastic Kubernetes cluster. The following command creates a new Amazon EKS cluster, but you can modify the steps for your use case if you have an existing cluster:
The output should look like the following:
- Authenticate to the Amazon Elastic Container Registry (Amazon ECR) repository where the Amazon EKS cluster is running:
The output should look like the following:
- Build and push container images to the Amazon ECR repository with a bash script:
Note: The ecr_push_image.sh script can generate an error due to Docker-related limits.
Deploying the CloudFormation templates
To deploy your templates, complete the following steps.
- Deploy the IAM role configuration template:
The output should look like the following:
- Set local environment variables IAM identity mapping:
- Check that the local environment variables were properly exported:
The output should look like the following:
- Update IAM identity mapping to associate the AWS Lambda execution role to the Amazon EKS cluster RBAC:
The output should look like the following:
- Open the
parameters/controlplane-cluster.json
file and provide the EksClusterName, EksUpdateClusterVersion, and IAMStackName. - Deploy the configuration template to start the cluster control plane upgrade. This step can take up to 60 minutes to complete:
- Check the present version of the cluster:
The output should look like the following:
- Open the
parameters/dataplane-daemonset.json
file and provide the EksClusterName and IAMStackName. - Deploy the configuration template to start the cluster data plane DaemonSet upgrade:
The output should look like the following:
Check Amazon CloudWatch Logs for the following output for each DaemonSet type:
- Deploy the configuration template to start the cluster data plane node groups upgrade:
The output should look like the following:
Clean up
- Delete the node group stack:
- Delete the DaemonSet stack:
- Delete the cluster stack:
- Delete the IAM stack:
- Delete container images and repository:
Conclusion
We have explained how to to upgrade a managed Kubernetes cluster using Amazon EKS in a repeatable pattern with configurations files, templates, and code. The activities during the upgrade process are logged in Amazon CloudWatch Logs. That information can be used for monitoring, alerting, and audits. The time that you would spend deploying these changes can be spent on other priorities, instead.