Introducing AWS SAM Pipelines: Automatically generate deployment pipelines for serverless applications

AWS Feed
Introducing AWS SAM Pipelines: Automatically generate deployment pipelines for serverless applications

Today, AWS announces the public preview of AWS SAM Pipelines, a new capability of AWS Serverless Application Model (AWS SAM) CLI. AWS SAM Pipelines makes it easier to create secure continuous integration and deployment (CI/CD) pipelines for your organizations preferred continuous integration and continuous deployment (CI/CD) system.

This blog post shows how to use AWS SAM Pipelines to create a CI/CD deployment pipeline configuration file that integrates with GitLab CI/CD.

AWS SAM Pipelines

A deployment pipeline is an automated sequence of steps that are performed to release a new version of an application. They are defined by a pipeline template file. AWS SAM Pipelines provides templates for popular CI/CD systems such as AWS CodePipeline, Jenkins, GitHub Actions, and GitLab CI/CD. Pipeline templates include AWS deployment best practices to help with multi-account and multi-Region deployments. AWS environments such as dev and production typically exist in different AWS accounts. This allows development teams to configure safe deployment pipelines, without making unintended changes to infrastructure. You can also supply your own custom pipeline templates to help to standardize pipelines across development teams.

AWS SAM Pipelines is composed of two commands:

1. sam pipeline bootstrap, a configuration command that creates the AWS resources required to create a pipeline.
2. sam pipeline init, an initialization command that creates a pipeline file for your preferred CI/CD system. For example, a Jenkinsfile for Jenkins or a .gitlab-ci.yml file for GitLab CI/CD.

Having two separate commands allows you to manage the credentials for operators and developer separately. Operators can use sam pipeline bootstrap to provision AWS pipeline resources. This can reduce the risk of production errors and operational costs. Developers can then focus on building without having to set up the pipeline infrastructure by running the sam pipeline init command.

You can also combine these two commands by running sam pipeline init –bootstrap. This takes you through the entire guided bootstrap and initialization process.

Getting started

The following steps show how to use AWS SAM Pipelines to create a deployment pipeline for GitLab CI/CD. GitLab is an AWS Partner Network (APN) member to build, review, and deploy code. AWS SAM Pipelines creates two deployment pipelines, one for a feature branch, and one for a main branch. Each pipeline runs as a separate environment in separate AWS accounts. Each time you make a commit to the repository’s feature branch, the pipeline builds, tests, and deploys a serverless application in the development account. For each commit to the Main branch, the pipeline builds, tests, and deploys to a production account.

Prerequisites

• An AWS account with permissions to create the necessary resources.
• Install AWS Command Line Interface (CLI) and AWS SAM CLI.
• A verified GitLab account: This post assumes you have the required permissions to configure GitLab projects, create pipelines, and configure GitLab variables.
• Create a new GitLab project and clone it to your local environment

Create a serverless application

Use the AWS SAM CLI to create a new serverless application from a Quick Start Template.

Run the following AWS SAM CLI command in the root directory of the repository and follow the prompts. For this example, can select any of the application templates:

sam init

Creating pipeline deployment resources

The sam pipeline bootstrap command creates the AWS resources and permissions required to deploy application artifacts from your code repository into your AWS environments.

For this reason, AWS SAM Pipelines creates IAM users and roles to allow you to deploy applications across multiple accounts. AWS SAM Pipelines creates these deployment resources following the principal of least privilege:

Run the following command in a terminal window:

sam pipeline init --bootstrap

This guides you through a series of questions to help create a .gitlab-ci.yml file. The --bootstrap option enables you to set up AWS pipeline stage resources before the template file is initialized:

1. Enter 1, to choose AWS Quick Start Pipeline Templates
2. Enter 2 to choose to create a GitLab CI/CD template file, which includes a two stage pipeline.
3. Next AWS SAM reports “No bootstrapped resources were detected.” and asks if you want to set up a new CI/CD stage. Enter Y to set up a new stage:
   

Set up the dev stage by answering the following questions:

1. Enter “dev” for the Stage name.
   
2. AWS SAM CLI detects your AWS CLI credentials file. It uses a named profile to create the required resources for this stage. If you have a development profile, select that, otherwise select the default profile.
3. Enter a Region for the stage name (for example, “eu-west-2”).
4. Keep the pipeline IAM user ARN and pipeline and CloudFormation execution role ARNs blank to generate these resources automatically.
5. An Amazon S3 bucket is required to store application build artifacts during the deployment process. Keep this option blank for AWS SAM Pipelines to generate a new S3 bucket.If your serverless application uses AWS Lambda functions packaged as a container image, you must create or specify an Amazon ECR Image repository. The bootstrap command configures the required permissions to access this ECR repository.
6. Enter N to specify you are not using Lambda functions packages as container images.
7. Press “Enter” to confirm the resources to be created.
   

AWS SAM Pipelines creates a PipelineUser with an associated ACCESS_KEY_ID and SECRET_ACCESS_KEY which GitLab uses to deploy artifacts to your AWS accounts. An S3 bucket is created along with two roles PipelineExecutionRole and CloudFormationExecutionRole.

Make a note of these values. You use these in the following steps to configure the production deployment environment and CI/CD provider.

Creating the production stage

The AWS SAM Pipeline command automatically detects that a second stage is required to complete the GitLab template, and prompts you to go through the set-up process for this:

1. Enter “Y” to continue to build the next pipeline stage resources.
   
2. When prompted for Stage Name, enter “prod”.
3. When asked to Select a credential source, choose a suitable named profile from your AWS config file. The following example shows that a named profile called “prod” is selected.
4. Enter a Region to deploy the resources to. The example uses the eu-west-1 Region.
5. Press enter to use the same Pipeline IAM user ARN created in the previous step.
6. When prompted for the pipeline execution role ARN and the CloudFormation execution role ARN, leave blank to allow the bootstrap process to create them.
7. Provide the same answers as in the previous steps 5-7.

The AWS resources and permissions are now created to run the deployment pipeline for a dev and prod stage. The definition of these two stages is saved in .aws-sam/pipeline/pipelineconfig.toml.

AWS SAM Pipelines now automatically continues the walkthrough to create a GitLab deployment pipeline file.

Creating a deployment pipeline file

The following questions help create a .gitlab-ci.yml file. GitLab uses this file to run the CI/CD pipeline to build and deploy the application. When prompted, enter the name for both the dev and Prod stages. Use the following example to help answer the questions:

Deployment pipeline file

A .gitlab-ci.yml pipeline file is generated. The file contains a number of environment variables, which reference the details from AWS SAM pipeline bootstrap command. This includes using the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY securely stored in the GitLab CI/CD repository.

The pipeline file contains a build and deploy stage for a branch that follows the naming pattern ‘feature-*’. The build process assumes the TESTING_PIPELINE_EXECUTION_ROLE in the testing account to deploy the application. sam build uses the AWS SAM template file previously created. It builds the application artifacts using the default AWS SAM build images. You can further customize the sam build –use-container command if necessary.

By default the Docker image used to create the build artifact is pulled from Amazon ECR Public. The default Node.js 14 image in this example is based on the language specified during sam init. To pull a different container image, use the --build-image option as specified in the documentation.

sam deploy deploys the application to a new stack in the dev stage using the TESTING_CLOUDFORMATION_EXECUTION_ROLE.The following code shows how this configured in the .gitlab-ci.yml file.

build-and-deploy-feature:
stage: build
only:
- /^feature-.*$/
script:
- . assume-role.sh ${TESTING_PIPELINE_EXECUTION_ROLE} feature-deployment
- sam build --template ${SAM_TEMPLATE} --use-container
- sam deploy --stack-name features-${CI_COMMIT_REF_NAME}-cfn-stack
--capabilities CAPABILITY_IAM
--region ${TESTING_REGION}
--s3-bucket ${TESTING_ARTIFACTS_BUCKET}
--no-fail-on-empty-changeset
--role-arn ${TESTING_CLOUDFORMATION_EXECUTION_ROLE}

The file also contains separate build and deployments stages for the main branch. sam package prepares the application artifacts. The build process then assumes the role in the production stage and prepares the application artifacts for production. You can customize the file to include testing phases, and manual approval steps, if necessary.

Configure GitLab CI/CD credentials

GitLab CI/CD uses the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY to authenticate to your AWS account. The values are associated with a new user generated in the previous sam pipeline init --bootstrap step. Save these values securely in GitLab’s CI/CD variables section:

1. Navigate to Settings > CI/CD > Variables and choose expand.
2. Choose Add variable, and enter in the key name and value for the AWS_SECRET_ACCESS_KEY noted down in the previous steps:
   
3. Repeat this process for the AWS_ACCESS_KEY_ID:
   

Creating a feature branch

Create a new branch in your GitLab CI/CD project named feature-1:

1. In the GitLab CI/CD menu, choose Branches from the Repository section. Choose New branch.
2. For Branch name, enter branch “feature-1” and in the Create from field choose main.
3. Choose Create branch.
   

Configure the new feature-1 branch to be protected so it can access the protected GitLab CI/D variables.

1. From the GitLab CI/CD main menu, choose to Settings then choose Repository.
2. Choose Expand in the Protected Branches section
3. Select the feature-1 branch in the Branch field and set Allowed to merge and Allowed to push to Maintainers.
4. Choose Protect
   
   

Trigger a deployment pipeline run

1. Add the AWS SAM application files to the repository and push the branch changes to GitLab CI/CD:

git checkout -b feature-1
git add .
git commit -am “added sam application”
git push --set-upstream origin feature-1 

This triggers a new pipeline run that deploys the application to the dev environment. The following screenshot shows GitLab’s CI/CD page.

AWS CloudFormation shows that a new stack has been created in the dev stage account. It is named after the feature branch:

To deploy the feature to production, make a pull request to merge the feature-1 branch into the main branch:

1. In GitLab CI/CD, navigate to Merge requests and choose New merge request.
2. On the following screen, choose feature-1 as the Source branch and main as the Target branch.
3. Choose Compare branches and continue, and then choose Create merge request.
4. Choose Merge

This merges the feature-1 branch to the main branch, triggering the pipeline to run the production build, testing, and deployment steps:

Conclusion

AWS SAM Pipelines is a new feature of the AWS SAM CLI that helps organizations quickly create pipeline files for their preferred CI/CD system. AWS provides a default set of pipeline templates that follow best practices for popular CI/CD systems such as AWS CodePipeline, Jenkins, GitHub Actions, and GitLab CI/CD. Organizations can also supply their custom pipeline templates via Git repositories to standardize custom pipelines across hundreds of application development teams. This post shows how to use AWS SAM Pipelines to create a CI/CD deployment pipeline for GitLab.

Watch guided video tutorials to learn how to create deployment pipelines for GitHub Actions, GitLab CI/CD, and Jenkins.

For more learning resources, visit https://serverlessland.com/explore/sam-pipelines.