TLS-enabled Kubernetes clusters with ACM Private CA and Amazon EKS

AWS Feed
TLS-enabled Kubernetes clusters with ACM Private CA and Amazon EKS

In this blog post, we show you how to set up end-to-end encryption on Amazon Elastic Kubernetes Service (Amazon EKS) with AWS Certificate Manager Private Certificate Authority. For this example of end-to-end encryption, traffic originates from your client and terminates at an Ingress controller server running inside a sample app. By following the instructions in this post, you can set up an NGINX ingress controller on Amazon EKS. As part of the example, we show you how to configure an AWS Network Load Balancer (NLB) with HTTPS using certificates issued via ACM Private CA in front of the ingress controller.

AWS Private CA supports an open source plugin for cert-manager that offers a more secure certificate authority solution for Kubernetes containers. cert-manager is a widely-adopted solution for TLS certificate management in Kubernetes. Customers who use cert-manager for application certificate lifecycle management can now use this solution to improve security over the default cert-manager CA, which stores keys in plaintext in server memory. Customers with regulatory requirements for controlling access to and auditing their CA operations can use this solution to improve auditability and support compliance.

Solution components

• Kubernetes is an open-source system for automating the deployment, scaling, and management of containerized applications.
• Amazon EKS is a managed service that you can use to run Kubernetes on Amazon Web Services (AWS) without needing to install, operate, and maintain your own Kubernetes control plane or nodes.
• cert-manager is an add on to Kubernetes to provide TLS certificate management. cert-manager requests certificates, distributes them to Kubernetes containers, and automates certificate renewal. cert-manager ensures certificates are valid and up-to-date, and attempts to renew certificates at an appropriate time before expiry.
• ACM Private CA enables the creation of private CA hierarchies, including root and subordinate CAs, without the investment and maintenance costs of operating an on-premises CA. With ACM Private CA, you can issue certificates for authenticating internal users, computers, applications, services, servers, and other devices, and for signing computer code. The private keys for private CAs are stored in AWS managed hardware security modules (HSMs), which are FIPS 140-2 certified, providing a better security profile compared to the default CAs in Kubernetes. Private certificates help identify and secure communication between connected resources on private networks such as servers, mobile and IoT devices, and applications.
• AWS Private CA Issuer plugin. Kubernetes containers and applications use digital certificates to provide secure authentication and encryption over TLS. With this plugin, cert-manager requests TLS certificates from Private CA. The integration supports certificate automation for TLS in a range of configurations, including at the ingress, on the pod, and mutual TLS between pods. You can use the AWS Private CA Issuer plugin with Amazon Elastic Kubernetes Service, self managed Kubernetes on AWS, and Kubernetes on-premises.
• The AWS Load Balancer controller manages AWS Elastic Load Balancers for a Kubernetes cluster. The controller provisions the following resources.
  • An AWS Application Load Balancer (ALB) when you create a Kubernetes Ingress.
  • An AWS Network Load Balancer (NLB) when you create a Kubernetes Service of type LoadBalancer.

Different points for terminating TLS in Kubernetes

How and where you terminate your TLS connection depends on your use case, security policies, and need to comply with regulatory requirements. This section talks about four different use cases that are regularly used for terminating TLS. The use cases are illustrated in Figure 1 and described in the text that follows.

Figure 1: Terminating TLS at different points

1. At the load balancer: The most common use case for terminating TLS at the load balancer level is to use publicly trusted certificates. This use case is simple to deploy and the certificate is bound to the load balancer itself. For example, you can use ACM to issue a public certificate and bind it with AWS NLB. You can learn more from How do I terminate HTTPS traffic on Amazon EKS workloads with ACM?
2. At the ingress: If there is no strict requirement for end-to-end encryption, you can offload this processing to the ingress controller or the NLB. This helps you to optimize the performance of your workloads and make them easier to configure and manage. We examine this use case in this blog post.
3. On the pod: In Kubernetes, a pod is the smallest deployable unit of computing and it encapsulates one or more applications. End-to-end encryption of the traffic from the client all the way to a Kubernetes pod provides a secure communication model where the TLS is terminated at the pod inside the Kubernetes cluster. This could be useful for meeting certain security requirements. You can learn more from the blog post Setting up end-to-end TLS encryption on Amazon EKS with the new AWS Load Balancer Controller.
4. Mutual TLS between pods: This use case focuses on encryption in transit for data flowing inside Kubernetes cluster. For more details on how this can be achieved with Cert-manager using an Istio service mesh, please see the Securing Istio workloads with mTLS using cert-manager blog post. You can use the AWS Private CA Issuer plugin in conjunction with cert-manager to use ACM Private CA to issue certificates for securing communication between the pods.

In this blog post, we use a scenario where there is a requirement to terminate TLS at the ingress controller level, demonstrating the second example above.

Figure 2 provides an overall picture of the solution described in this blog post. The components and steps illustrated in Figure 2 are described fully in the sections that follow.

Figure 2: Overall solution diagram

Prerequisites

Before you start, you need the following:

• An AWS account
• The IAM security principal that you’re using must have permissions to work with Amazon EKS IAM roles and service-linked roles, AWS CloudFormation, Amazon Virtual Private Cloud (Amazon VPC), and related resources. For more information, see Actions, resources, and condition keys for Amazon Elastic Container Service for Kubernetes and Using service-linked roles in the IAM User Guide. In addition, this IAM security principal should have an AWSCertificateManagerPrivateCAFullAccess IAM managed policy attached to it.
• The AWS Command Line Interface (AWS CLI), with the kubectl and eksctl tools installed and configured. Follow the instructions in Getting started with Amazon EKS – eksctl in the Amazon EKS User Guide.
• Helm CLI.

Verify that you have the latest versions of these tools installed before you begin.

Provision an Amazon EKS cluster

If you already have a running Amazon EKS cluster, you can skip this step and move on to install NGINX Ingress.

You can use the AWS Management Console or AWS CLI, but this example uses eksctl to provision the cluster. eksctl is a tool that makes it easier to deploy and manage an Amazon EKS cluster.

This example uses the US-EAST-2 Region and the T2 node type. Select the node type and Region that are appropriate for your environment. Cluster provisioning takes approximately 15 minutes.

To provision an Amazon EKS cluster

1. Run the following eksctl command to create an Amazon EKS cluster in the us-east-2 Region with Kubernetes version 1.19 and two nodes. You can change the Region to the one that best fits your use case.

   eksctl create cluster 
   --name acm-pca-lab 
   --version 1.19 
   --nodegroup-name acm-pca-nlb-lab-workers 
   --node-type t2.medium 
   --nodes 2 
   --region us-east-2
   

2. Once your cluster has been created, verify that your cluster is running correctly by running the following command:

   $ kubectl get pods --all-namespaces
   NAMESPACE NAME READY STATUS RESTARTS AGE
   kube-system aws-node-t94rp 1/1 Running 0 3m4s
   kube-system aws-node-w7dm6 1/1 Running 0 3m19s
   kube-system coredns-56b458df85-6tgjl 1/1 Running 0 10m
   kube-system coredns-56b458df85-8gp94 1/1 Running 0 10m
   kube-system kube-proxy-2pjx7 1/1 Running 0 3m19s
   kube-system kube-proxy-hz8wq 1/1 Running 0 3m4s 

You should see output similar to the above, with all pods in a running state.

Install NGINX Ingress

NGINX Ingress is built around the Kubernetes Ingress resource, using a ConfigMap to store the NGINX configuration.

To install NGINX Ingress

1. Use the following command to install NGINX Ingress:

   kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.46.0/deploy/static/provider/aws/deploy.yaml
   

2. Run the following command to determine the address that AWS has assigned to your NLB:

   $ kubectl get service -n ingress-nginx
   NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
   ingress-nginx-controller LoadBalancer 10.100.214.10 a3ebe22e7ca0522d1123456fbc92605c-8ac7f1d49be2fc42.elb.us-east-2.amazonaws.com 80:32598/TCP,443:30624/TCP 14s
   ingress-nginx-controller-admission ClusterIP 10.100.118.1 <none> 443/TCP 14s 

3. It can take up to 5 minutes for the load balancer to be ready. Once the external IP is created, run the following command to verify that traffic is being correctly routed to ingress-nginx:

   curl http://a3ebe22e7ca0522d1123456fbc92605c-8ac7f1d49be2fc42.elb.us-east-2.amazonaws.com
   <html>
   <head><title>404 Not Found</title></head>
   <body>
   <center><h1>404 Not Found</h1></center>
   <hr><center>nginx</center>
   </body>
   </html>
   

Note: Even though, it’s returning an HTTP 404 error code, in this case curl is still reaching the ingress controller and getting the expected HTTP response back.

Configure your DNS records

Once your load balancer is provisioned, the next step is to point the application’s DNS record to the URL of the NLB.

You can use your DNS provider’s console, for example Route53, and set a CNAME record pointing to your NLB. See CNAME record type for more details on how to setup a CNAME record using Route53.

This scenario uses the sample domain rsa-2048.example.com.

rsa-2048.example.com CNAME a3ebe22e7ca0522d1123456fbc92605c-8ac7f1d49be2fc42.elb.us-east-2.amazonaws.com

As you go through the scenario, replace rsa-2048.example.com with your registered domain.

Install cert-manager

cert-manager is a Kubernetes add-on that you can use to automate the management and issuance of TLS certificates from various issuing sources. It runs within your Kubernetes cluster and will ensure that certificates are valid and attempt to renew certificates at an appropriate time before they expire.

You can use the regular installation on Kubernetes guide to install cert-manager on Amazon EKS.

After you’ve deployed cert-manager, you can verify the installation by following these instructions. If all the above steps have completed without error, you’re good to go!

Note: If you’re planning to use Amazon EKS with Kubernetes pods running on AWS Fargate, please follow the cert-manager Fargate instructions to make sure cert-manager installation works as expected. AWS Fargate is a technology that provides on-demand, right-sized compute capacity for containers.

Install aws-privateca-issuer

The AWS PrivateCA Issuer plugin acts as an addon (see external cert configuration) to cert-manager that signs certificate requests using ACM Private CA.

To install aws-privateca-issuer

1. For installation, use the following helm commands:

   kubectl create namespace aws-pca-issuer helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
   helm repo update
   helm install awspca/aws-pca-issuer --generate-name --namespace aws-pca-issuer
   

2. Verify that the AWS Private CA Issuer is configured correctly by running the following command and ensure that it is in READY state with status as Running:

   $ kubectl get pods --namespace aws-pca-issuer
   NAME READY STATUS RESTARTS AGE
   aws-pca-issuer-1622570742-56474c464b-j6k8s 1/1 Running 0 21s
   

3. You can check the chart configuration in the default values file.

Create an ACM Private CA

In this scenario, you create a private certificate authority in ACM Private CA with RSA 2048 selected as the key algorithm. You can create a CA using the AWS console, AWS CLI, or AWS CloudFormation.

To create an ACM Private CA

Download the CA certificate using the following command. Replace the <CA_ARN> and <Region> values with the values from the CA you created earlier and save it to a file named cacert.pem:

aws acm-pca get-certificate-authority-certificate --certificate-authority-arn <CA_ARN> -- region <region> --output text > cacert.pem

Once your private CA is active, you can proceed to the next step. You private CA will look similar to the CA in Figure 3.

Figure 3: Sample ACM Private CA

Set EKS node permission for ACM Private CA

In order to issue a certificate from ACM Private CA, add the IAM policy from the prerequisites to your EKS NodeInstanceRole. Replace the <CA_ARN> value with the value from the CA you created earlier:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "awspcaissuerpolicy", "Effect": "Allow", "Action": [ "acm-pca:GetCertificate", "acm-pca:DescribeCertificateAuthority", "acm-pca:IssueCertificate" ], "Resource": "<CA_ARN>" } ]
}

Create an Issuer in Amazon EKS

Now that the ACM Private CA is active, you can begin requesting private certificates which can be used by Kubernetes applications. Use aws-privateca-issuer plugin to create the ClusterIssuer, which will be used with the ACM PCA to issue certificates.

Issuers (and ClusterIssuers) represent a certificate authority from which signed x509 certificates can be obtained, such as ACM Private CA. You need at least one Issuer or ClusterIssuer before you can start requesting certificates in your cluster. There are two custom resources that can be used to create an Issuer inside Kubernetes using the aws-privateca-issuer add-on:

• AWSPCAIssuer is a regular namespaced issuer that can be used as a reference in your Certificate custom resources.
• AWSPCAClusterIssuer is specified in exactly the same way, but it doesn’t belong to a single namespace and can be referenced by certificate resources from multiple different namespaces.

To create an Issuer in Amazon EKS

1. For this scenario, you create an AWSPCAClusterIssuer. Start by creating a file named cluster-issuer.yaml and save the following text in it, replacing <CA_ARN> and <Region> information with your own.

   apiVersion: awspca.cert-manager.io/v1beta1
   kind: AWSPCAClusterIssuer
   metadata: name: demo-test-root-ca
   spec: arn: <CA_ARN> region: <Region>
   

2. Deploy the AWSPCAClusterIssuer:

   kubectl apply -f cluster-issuer.yaml
   

3. Verify the installation and make sure that the following command returns a Kubernetes service of kind AWSPCAClusterIssuer:

   $ kubectl get AWSPCAClusterIssuer
   NAME AGE
   demo-test-root-ca 51s
   

Request the certificate

Now, you can begin requesting certificates which can be used by Kubernetes applications from the provisioned issuer. For more details on how to specify and request Certificate resources, please check Certificate Resources guide.

To request the certificate

1. As a first step, create a new namespace that contains your application and secret:

   $ kubectl create namespace acm-pca-lab-demo
   namespace/acm-pca-lab-demo created
   

2. Next, create a basic X509 private certificate for your domain.
   Create a file named rsa-2048.yaml and save the following text in it. Replace rsa-2048.example.com with your domain.

kind: Certificate
apiVersion: cert-manager.io/v1
metadata: name: rsa-cert-2048 namespace: acm-pca-lab-demo
spec: commonName: www.rsa-2048.example.com dnsNames: - www.rsa-2048.example.com - rsa-2048.example.com duration: 2160h0m0s issuerRef: group: awspca.cert-manager.io kind: AWSPCAClusterIssuer name: demo-test-root-ca renewBefore: 360h0m0s secretName: rsa-example-cert-2048 usages: - server auth - client auth privateKey: algorithm: "RSA" size: 2048

• For a certificate with a key algorithm of RSA 2048, create the resource:

  kubectl apply -f rsa-2048.yaml -n acm-pca-lab-demo
  

• Verify that the certificate is issued and in READY state by running the following command:

  $ kubectl get certificate -n acm-pca-lab-demo
  NAME READY SECRET AGE
  rsa-cert-2048 True rsa-example-cert-2048 12s
  

• Run the command kubectl describe certificate -n acm-pca—lab-demo to check the progress of your certificate.
• Once the certificate status shows as issued, you can use the following command to check the issued certificate details:

  kubectl get secret rsa-example-cert-2048 -n acm-pca-lab-demo -o 'go-template={{index .data "tls.crt"}}' | base64 --decode | openssl x509 -noout -text
  

Deploy a demo application

For the purpose of this scenario, you can create a new service—a simple “hello world” website that uses echoheaders that respond with the HTTP request headers along with some cluster details.

To deploy a demo application

1. Create a new file named hello-world.yaml with below content:

   apiVersion: v1
   kind: Service
   metadata: name: hello-world namespace: acm-pca-lab-demo
   spec: type: ClusterIP ports: - port: 80 targetPort: 8080 selector: app: hello-world
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata: name: hello-world namespace: acm-pca-lab-demo
   spec: replicas: 3 selector: matchLabels: app: hello-world template: metadata: labels: app: hello-world spec: containers: - name: echoheaders image: k8s.gcr.io/echoserver:1.10 args: - "-text=Hello World" imagePullPolicy: IfNotPresent resources: requests: cpu: 100m memory: 100Mi ports: - containerPort: 8080
   

2. Create the service using the following command:

   $ kubectl apply -f hello-world.yaml
   

Expose and secure your application

Now that you’ve issued a certificate, you can expose your application using a Kubernetes Ingress resource.

To expose and secure your application

1. Create a new file called example-ingress.yaml and add the following content:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata: name: acm-pca-demo-ingress namespace: acm-pca-lab-demo annotations: kubernetes.io/ingress.class: "nginx"
   spec: tls: - hosts: - www.rsa-2048.example.com secretName: rsa-example-cert-2048 rules: - host: www.rsa-2048.example.com http: paths: - path: / pathType: Exact backend: service: name: hello-world port: number: 80
   

2. Create a new Ingress resource by running the following command:

   kubectl apply -f example-ingress.yaml 

Access your application using TLS

After completing the previous step, you can to access this service from any computer connected to the internet.

To access your application using TLS

1. Log in to a terminal window on a machine that has access to the internet, and run the following:

   $ curl https://rsa-2048.example.com --cacert cacert.pem 

2. You should see an output similar to the following:

   Hostname: hello-world-d8fbd49c6-9bczb Pod Information: -no pod information available- Server values: server_version=nginx: 1.13.3 - lua: 10008 Request Information: client_address=192.162.32.64 method=GET real path=/ query= request_version=1.1 request_scheme=http request_uri=http://rsa-2048.example.com:8080/ Request Headers: accept=*/* host=rsa-2048.example.com user-agent=curl/7.62.0 x-forwarded-for=52.94.2.2 x-forwarded-host=rsa-2048.example.com x-forwarded-port=443 x-forwarded-proto=https x-real-ip=52.94.2.2 x-request-id=371b6fc15a45d189430281693cccb76e x-scheme=https Request Body: -no body in request-…
   

   This response is returned from the service running behind the Kubernetes Ingress controller and demonstrates that a successful TLS handshake happened at port 443 with https protocol.

3. You can use the following command to verify that the certificate issued earlier is being used for the SSL handshake:

   echo | openssl s_client -showcerts -servername www.rsa-2048.example.com -connect www.rsa-2048.example.com:443 2>/dev/null | openssl x509 -inform pem -noout -text
   

Cleanup

To avoid incurring future charges on your AWS account, perform the following steps to remove the scenario.

Delete the ACM Private CA

You can delete the ACM Private CA by following the instructions in Deleting your private CA.

As an alternative, you can use the following commands to delete the ACM Private CA, replacing the <CA_ARN> and <Region> with your own:

1. Disable the CA.

   aws acm-pca update-certificate-authority 
   --certificate-authority-arn <CA_ARN>
   --region <Region>
   --status DISABLED
   

2. Call the Delete Certificate Authority API

   aws acm-pca delete-certificate-authority 
   --certificate-authority-arn <CA_ARN>
   --region <Region>
   --permanent-deletion-time-in-days 7
   

Continue the cleanup

Once the ACM Private CA has been deleted, continue the cleanup by running the following commands.

1. Delete the services:

   kubectl delete -f hello-world.yaml
   

2. Delete the Ingress controller:

   kubectl delete -f example-ingress.yaml
   

3. Delete the IAM NodeInstanceRole, replace role name with your EKS Node instance role created for the demo:

   aws iam delete-role --role-name eksctl-acm-pca-lab-nodegroup-acm-pca-nlb-lab-workers-NodeInstanceProfile-XXXXXXX
   

4. Delete the Amazon EKS cluster using ekctl command:

   eksctl delete cluster acm-pca-lab --region us-east-2
   

You can also clean up from your Cloudformation console by deleting the stacks named eksctl-acm-pca-lab-nodegroup-acm-pca-nlb-lab-workers and eksctl-acm-pca-lab-cluster.

Conclusion

In this blog post, we showed you how to set up a Kubernetes Ingress controller with a service running in Amazon EKS cluster using AWS Load Balancer Controller with Network Load Balancer and set up HTTPS using private certificates issued by ACM Private CA. If you have questions or want to contribute, join the aws-privateca-issuer add-on project on GitHub.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Certificate Manager forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.