Continuous delivery with server-side Swift on Amazon Linux 2

Amazon Web Services Feed
Continuous delivery with server-side Swift on Amazon Linux 2

In January, I published an article describing how to use AWS tools to build, test, and release server-side Swift code on two platforms: Amazon Elastic Container Service (Amazon ECS) and Elastic Compute Cloud (Amazon EC2) running Ubuntu Linux. Recently Swift.org has released official support for the Amazon Linux 2 operating system. This article is a follow-up to my previous article and will explain how to build a continuous delivery process that targets Amazon Linux 2. This article is intended to be self-contained, and contains code and content updates from the previous article for the new operating system and Swift version.

Swift is a general-purpose programming language released by Apple in 2014. It aimed to provide many of the features of Objective-C, such as type safety, late binding, and dynamic dispatch, in a package that would improve developer productivity and code safety. Although Swift has been a popular replacement for Objective-C for iOS development, it is gaining interest for server-side development as well. The combination of type safety and developer productivity makes Swift an interesting alternative to other popular server-side programming languages, and especially interesting for development teams that are already using Swift for iOS development. Currently teams that are trying to give engineers ownership of product features by promoting full-stack development require those engineers to context switch between Swift for iOS development and a language like Golang or Ruby for server API development. Using a single language for both front end and back can lower that context switching cost and improve overall developer productivity.

A fast and flexible continuous integration and delivery pipeline is critical to ensure that developers get fast feedback on potential issues in their code, and to keep delivery lead times as quick as possible. A good CI/CD pipeline is a great complement to the productivity and safety that Swift provides; the programming language ensures those benefits at the code level, and the pipeline enforces quality and speed during the integration and deployment process.

This post explains how to use Amazon Web Services (AWS) to create a continuous delivery pipeline for server-side Swift code. We will be creating a pipeline using managed AWS services, such as AWS CodePipeline, AWS CodeBuild, and AWS CodeDeploy. This pipeline will compile, build, test, and deploy a simple Swift web service onto EC2 virtual machines running Amazon Linux 2.

Overview of the pipeline architecture

The pipeline is designed to test, build, and deploy our Swift web service. Before starting, let’s step back and think about what properties we want this pipeline to have.

• Fast: The pipeline should pick up changes without developer input, and execute the build, test, and deployment process as quickly as possible.
• Flexible: Technical and business needs change. Our architecture should make creating additional pipeline steps or modify existing ones simple, quick, and without disruption.
• Elastic: We want to make sure that we’re achieving our goal of a fast pipeline without paying for idle capacity. We want power when we need it, but don’t want to pay for that power on nights and weekends.

We’ve taken an opinionated view of how to build this pipeline with AWS services to achieve these goals, using CodePipeline, CodeBuild, CodeDeploy, and other AWS services; however, this is not all-or-nothing. CodePipeline supports a variety of services to build, test, and deploy code, and CodePipeline itself can be swapped with other services, such as Jenkins, Concourse CI, and others. If you’re curious about the Jenkins integrations with CodePipeline, see the Create a Four-Stage Pipeline tutorial.

We encourage you to take this setup as a foundation for your own pipeline, to give you ideas of how components can fit together. Don’t hesitate to modify it and make it your own.

Let’s learn about each of the components in our setup.

CodePipeline: CI/CD pipeline

First let’s examine the pipeline itself. AWS CodePipeline is a fully managed continuous delivery service that will let us tie together the source, test, build, and deployment steps of our pipeline, while integrating with other AWS services and respecting least-privileges access.

CodePipeline organizes actions that should be performed within stages. Our pipeline will have four stages:

1. Source
2. Test
3. Build
4. Deploy

The Source stage will respond to changes in the associated Swift Git repository. The Test stage will run unit tests on the code. The Build stage will run two actions in parallel: one to build a Swift binary and one to build a Docker container. Finally, the Deploy stage will deploy the Swift binary to EC2.

CodeCommit: Source code

The first stage of our pipeline is the Source stage, which is responsible for managing the connection with the source code repository and pulling the latest code revision when there is an update. In our pipeline, we have used CodeCommit, an AWS-managed Git source code repository. CodePipeline supports other source code repositories, such as GitHub, GitHub Enterprise, or even an S3 bucket.

S3: Artifact repository

Our pipeline will need to create and use artifacts in the pipeline, such as the source code from the Git repository, the Swift binaries compiled in the CodePipeline build stage, and so on. With the exception of the Docker image, which will be stored in an ECR container registry, CodePipeline will use a special artifact bucket to store these artifacts.

ECR: Container repositories

Amazon Elastic Container Registry (Amazon ECR) is a fully managed Docker registry that we are using to hold the custom CodeBuild image used to compile the Swift binaries.

Later we’ll examine how our pipeline uses AWS CodeBuild to create binaries for our Swift application, but for now the important thing to know is that CodeBuild allows us to run a set of build commands on top of a base environment, which is a Docker image pre-installed with the required build setup. In our case, we created a custom Docker image with the Swift toolchain and libraries so that CodeBuild can compile the Swift binaries. ECR will act as a place to store this customized base image.

EC2: Virtual machine hosting

In this version of the pipeline we will be hosting the application code directly on EC2, using the Amazon Linux 2 operating system. The EC2 instances are running in an autoscaling group, which ensures that we provision or remove EC2 instances based on incoming demand, and removes any EC2 instance marked as unhealthy. We are running these instances attached to an Application Load Balancer (ALB) to distribute traffic to all healthy EC2 instances.

CodeBuild: Test and build

We will be using CodeBuild to test and build our Swift code. CodeBuild provides prepackaged build environments in the form of optimized Docker images that have been set up to provide the toolchain you need to build and test code in a variety of programming languages. In our case, we also will take advantage of the ability to create custom build environments in CodeBuild to test and compile our Swift application.

We also provide a buildspec file to tell CodeBuild which steps to perform for the build process. The buildspec is a yaml file that describes the commands needed to run during each phase of the build process (e.g., an install phase to install any prerequisite tools or libraries, the build phase, a post-build phase to package artifacts), and a section describing which artifacts of the build process should be stored for use by subsequent steps. The buildspec file can either be specified as part of the CodeBuild project definition, or can be included as part of the project source code. In our case we will take the latter approach, with the buildspec.*.yml files in the sample application.

CodeBuild will be used in two different ways in our pipeline:

1. A build environment that compiles the Swift code and pushes the resulting binaries to the S3 artifact bucket. The buildspec file used is buildspec.ec2.yml in the project source directory.
2. A test environment that runs the Swift application’s unit tests. The buildspec file used is buildspec.test.yml in the project source directory.

CodeDeploy: Deploy to live systems

Our pipeline needs to deploy the Swift binaries to our Amazon Linux 2 EC2 instances. Deploying on EC2 requires rules and guidance: for example we need to provide instructions on how to take a new binary from our S3 artifact repository and either deploy or roll back on an EC2 instance.

CodeDeploy helps us out here. With CodeDeploy, we can provide instructions on what steps should be taken for each stage in a deployment on an EC2 instance, specifically:

• BeforeInstall
• AfterInstall
• ApplicationStart
• ApplicationStop
• ValidateService
• BeforeBlockTraffic
• AfterBlockTraffic
• BeforeAllowTraffic
• AfterAllowTraffic

For details on how each stage is used, see the AppSpec ‘hooks’ Section for an EC2/On-Premises Deployment in the AWS CodeDeploy documentation.

Although we don’t need to provide scripts for each deployment stage, we have the ability to do so via an AppSpec file provided with the deployment. In this example, we take advantage of the ApplicationStop and ApplicationStart to stop and start a systemd service, and BeforeInstall and AfterInstall to ensure that we have the right directory structure and permissions.

CodeDeploy also allows us to specify a variety of deployment configurations, such as deploying in place to one instance at a time, half of the instances in place at a time, blue/green style deployments, or your own custom configurations. CodeDeploy will monitor the deployments, provide notifications for success or failure, and roll back in case of a failed deployment.

CloudFormation: Update and maintain the pipeline

As we’ll learn in the following section, we are using CloudFormation to set up and configure all the AWS resources in the pipeline, the EC2 components, and all the networking and security components. CloudFormation allows us to automate the setup and configuration in a repeatable way, and to track infrastructure updates as code changes.

Getting started

Let’s get started building the pipeline. The AWS resources described in the previous section and the required configuration will be provisioned through a CloudFormation script.

Setting up an AWS account, CloudFormation, and Docker

The first step is to decide on the AWS account where the resources will be provisioned. If you already have an account provisioned that you want to use, then you can skip this step. Otherwise go to https://portal.aws.amazon.com/billing/signup and create a new account.

Note that the CloudFormation script will provision resources that will result in a monthly bill. Be sure to destroy the CloudFormation stack to remove those resources once you are done using them. See the Cleanup section at the end of this article for instructions on how to tear down the resources.

To run the CloudFormation creation scripts from the CLI, you’ll need a user with CLI access. From the AWS console, navigate to the IAM service, then click on Add User. Pick a user name and make sure that Programmatic access is selected. Next, Add existing policies directly and select Administrator access. Make sure you copy or download the Access Key ID and Secret Access Key.

If you don’t have the AWS CLI installed already, let’s do that now. The download link and instructions are available at AWS Command Line Interface. Make sure you’ve configured the CLI with the Access Key ID and Secret Access Key you wrote down in the previous step.

Create a new S3 bucket that will store your packaged CloudFormation templates. If you have a preexisting account with a bucket that you want to use for this purpose, you can use it. The user running the scripts should have read/write access to this bucket, and it should be in the same account as where you want the resources to be provisioned.

We will be creating and uploading a custom Docker image in a later step, containing the Swift toolchain for CodeBuild to use. Make sure you have Docker installed and configured on your workstation.

To push code from your workstation to Git, we will be using the Git CLI. Make sure you have a working installation on your workstation. If you don’t have Git installed already, it is available for multiple operating systems from Git downloads.

Download and configure the CloudFormation setup package

The CloudFormation template, helper scripts, sample application, and the CodeBuild Swift build image are bundled together in a file located on GitHub. (You can also browse the GitHub page and clone it from here: https://github.com/aws-samples/aws-pipeline-server-side-swift-blog/tree/al2.) Download that file and extract the contents. Update package.json so that both lines with CF_BUCKET= is followed by the name of the CloudFormation S3 bucket you configured in the previous step.

In create-stack.json you can optionally update the name of the CloudFormation stack that will be create; it defaults to swift-build. You can also change the tags that will be applied to all created resources.

Deploying the CloudFormation template

We can now deploy the CloudFormation template. There’s a helper script located in ./scripts/create-stack.sh that will package up and run the CloudFormation script. You can run that yourself:

CF_BUCKET=<your CloudFormation S3 bucket name here> ./scripts/create-stack.sh

If you happen to have npm installed, there’s also a handy shortcut: npm run-script create will run the same commands for you.

We now need to wait for the script to finish, which will take approximately 15-20 minutes. You can view the status either by looking at the AWS console in the CloudFormation section or by running aws cloudformation wait stack-create-complete.

Note: The CloudFormation script creates an ECR repository named codebuild/swift. If you get CloudFormation errors while trying to create those repositories, check that you don’t have existing repositories with that name.

Creating the CodeBuild custom build runtime

In this step we’ll create the custom CodeBuild runtime image that will be used when CodeBuild compiles the Swift binary.

First, let’s create the Docker image. Navigate to ECR within the AWS console, click on codebuild/swift and follow the instructions in View push commands from within the codebuild-image folder. You’ll be asked to log in to ECR, then build the image, then tag it so that it’s associated with the ECR repository, and finally push the newly created image to AWS. The steps will look something like this (the repository name will be different based on your region and account):

cd codebuild-image
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 123456789123.dkr.ecr.us-east-1.amazonaws.com)
docker build -t codebuild/swift .
docker tag codebuild/swift:latest 123456789123.dkr.ecr.us-west-2.amazonaws.com/codebuild/swift:latest
docker push 123456789123.dkr.ecr.us-west-2.amazonaws.com/codebuild/swift:latest

Pushing the Swift application code

Now that the pipeline is in place with the correct CodeBuild custom runtime, everything is ready and we can push the application code.

We have a sample Swift web application in the app folder that we’ll be using to test the pipeline. From the project root navigate to the ./app directory and initialize a Git repository: cd app && git init.

Prior to using the Git CLI to push the web application code to CodeCommit, you may have to configure AWS and Git to enable access. There are two methods described in Setting Up for AWS CodeCommit in the AWS documentation, which involves either HTTP access or SSH access. Either access method will work fine; although I used SSH access in the following examples, the steps once setup is complete will be the same in either case.

Navigate to the CodeCommit service in the AWS console, then locate the repository created by the CloudFormation script (it should start with the stack name that you used, e.g. swift-build-...). Click on that repository and follow the instructions to initialize CodeCommit with the contents of the app git repository.

git remote add origin ssh://git-codecommit.us-west-2.amazonaws.com/v1/repos/swift-build-Pipeline-1234567890ABC-Repo
git add .
git commit -m "Initial commit"
git push --set-upstream origin master

Confirming the pipeline execution

CodePipeline should now pick up on the change in the CodeCommit repository and start the pipeline. Let’s confirm that.

Navigate to CodePipeline. Click on the pipeline whose name starts with the CloudFormation stack name. A few seconds after the git push it should pick up the change in the CodeCommit repository and launch. After 10-15 minutes it should complete all four stages: Source, Test, Build, and Release.

Once the pipeline has completed successfully, let’s make sure that the application was correctly deployed to the EC2 servers. Navigate to the EC2 AWS console, then click on the sidebar to find the Load Balancer section. First click on the load balancer whose name starts with swift-EC2. Copy the DNS name and paste it into a browser window. Confirm that it responds with “It works!”

Optionally, you can also verify the EC2 service is working from the command line:

curl $(aws cloudformation describe-stacks --stack-name swift-build --query "Stacks[0].Outputs[?OutputKey=='Ec2LbUrl'].OutputValue" --output text)

Pushing a code change

As a final check, let’s make a quick change to the application source code and make sure that it updates the services correctly.

Update the swift-codebuild-app repository. Edit Sources/App/routes.swift and modify return "It works!" to read return "It works! With an update!" or whichever updated message you prefer. Commit those changes and push them to the CodeCommit repository:

git add .
git commit -m "Changed application message"
git push

Now wait for the pipeline to update and confirm that the two load balancers update with the correct message. Use the DNS address as we did in the previous step to check that the update was deployed to the EC2 servers.

Cleanup

Once you are done trying out the pipeline and are no longer using it, be sure to destroy the CloudFormation stack to save on the monthly cost of unused resources.

Before destroying the CloudFormation stack, we need to make sure that the S3 bucket and the ECR repositories are empty. You can do that manually via the AWS management console, or via the command line:

aws s3 rm s3://$(aws cloudformation describe-stacks --stack-name swift-build --query "Stacks[0].Outputs[?OutputKey=='S3ArtifactBucket'].OutputValue" --output text)/ --recursive
IMAGES_TO_DELETE=$(aws ecr list-images --repository-name codebuild/swift --query 'imageIds[*]' --output json)
aws ecr batch-delete-image --repository-name codebuild/swift --image-ids "$IMAGES_TO_DELETE"

Once the repositories and the bucket are empty, destroy the CloudFormation stack from the AWS management console or from the command line: aws cloudformation delete-stack --stack-name swift-build

Troubleshooting tips

Hopefully you were able to deploy the pipeline successfully. If you did run into trouble, here are a couple of potential issues that you may have faced and tips on how to fix them.

Name collisions

The CloudFormation script uses dynamic names for most AWS resources, but there are a couple of examples where the pipeline makes an assumption that no other resources are using a given name. Specifically, the CloudFormation script creates a repository name called codebuild/swift for the Docker image used to provide the Swift toolchain. If you get an error from CloudFormation saying that it can’t create the ECR repository of that name, check that you don’t already have a name collision.

Further optimizations and next steps

There are a few ways that we can make our pipeline even better.

A more sophisticated pipeline will probably need a staging environment. We could easily add to our pipeline so that it deploys first to an environment with limited access, so that we can perform either a manual review or a set of automated tests before deploying to the production site. That pipeline might look something like this:

Another improvement would be to generate more useful reports. Currently, the second stage of the pipeline runs unit tests, but the only way for developers to get feedback on their code is by viewing the CodeBuild logged output. Fortunately, CodeBuild supports the ability to gather information from test reports and associates that report with the CodeBuild execution. See Working with Test Reporting in CodeBuild for more information.

Finally, a good pipeline will check a variety of quality metrics, not just the functionality as tested by unit tests and integration tests. CodePipeline allows you to integrate performance, security, load testing, and so on into your pipeline. One approach is to do something similar to what we did in this article and leverage CodeBuild to host your test scripts, but CodePipeline also allows you to integrate with tools such as BlazeMeter, Ghost Inspector, Micro Focus StormRunner, Runscope API, or your existing Jenkins jobs. Once the pipeline is this mature, manual approvals can either be eliminated or used only for highly sensitive modules. A full pipeline with these additional checks may look something like this:

Best of luck developing with server-side Swift! I hope this post helps you get off the ground in your effort to build, test, and deploy your Swift code on AWS, and I encourage you to experiment with the provided CloudFormation code.

Feature image via Pixabay.