Share reusable infrastructure as code by using AWS CloudFormation modules and StackSets

AWS Feed
Share reusable infrastructure as code by using AWS CloudFormation modules and StackSets

It is common for customers to have multiple teams creating infrastructure as code (IaC) templates (for example, by using AWS CloudFormation). Because there is duplication of the common resources used in these templates, you might understandably feel like you’re reinventing the wheel. By sharing these common definitions as CloudFormation modules, you can provide access to your entire organization created in AWS Organizations using CloudFormation StackSets. This facilitates the reuse of common building blocks and reduces duplication and effort.

This post builds on information in the Using AWS CloudFormation Modules to Improve Enterprise Security post, where we discuss the advantages and best practices of CloudFormation Modules. I’ll show how you can share your modules across the organization and then consume them in your CloudFormation templates.

AWS CloudFormation allows you to model a collection of related AWS and third-party resources, provision them quickly and consistently, and manage them throughout their lifecycles, by treating infrastructure as code. A CloudFormation template describes your desired resources and their dependencies so you can launch and configure them together as a stack.

CloudFormation modules provide a mechanism to reuse predefined blocks of resources in CloudFormation templates. Using CloudFormation StackSets, you can deploy modules across multiple accounts, organizational units (OUs), and Regions to save development time by facilitating code reuse.

Solution overview

In the walkthrough, I’ll show you how to publish CloudFormation modules to a central repository and then share them with other accounts in your organization.

Figure 1 shows an overview of the solution:

Figure 1: Solution overview

Here are the steps in the workflow:

1. Create a CloudFormation module and publish it to an S3 bucket.
2. Create a CloudFormation template that references this module.
3. Create a CloudFormation stack set from the template to be deployed across multiple AWS accounts and Regions.
4. Consume the module from any CloudFormation template created in the deployed Regions and accounts.

Prerequisites

Before you begin, complete the following tasks:

• Enable all features in AWS Organizations and enable trusted access. Most organizations will be created with all-features enabled by default, so confirm this is the case for your organization and if not enable all features.
• Register a delegated administrator to any accounts that will publish modules.
• Install the CloudFormation Command Line Interface (CLI) to create and register modules.

CloudFormation modules are registered as private extensions in CloudFormation. When you use private extensions in your CloudFormation stacks, you will incur charges in your account that are in addition to charges for the created resources. This is because private resource types implement custom logic that runs during resource create, read, update, list, and delete operations. For more information, see AWS CloudFormation pricing.

Walkthrough

I use the Amazon S3 and AWS CloudFormation consoles to deploy this solution, but it’s available as a CloudFormation template on GitHub.

Here are the high-level steps:

1. Create a CloudFormation module.
2. Create an S3 bucket that where the module will be stored.
3. Grant the organization access to the S3 bucket.
4. Create the CloudFormation template for the stack set.
5. Deploy the stack set to the OUs and Regions.
6. Create stacks using the new module in any account and Region where the solution is deployed.

Create a CloudFormation module

Create a directory and then cd into it. Use the CloudFormation CLI to initialize a project.

When asked if you want to develop a new resource or module, enter m for module. Enter a name for your module (for example, MyOrganization::MyNamespace::MyApp::MODULE).

Figure 2: cfn init

This will create a sample JSON file in the fragments directory. Delete this file and create a YAML file with a descriptive name (for example, myapp-template.yaml).

This file will contain the CloudFormation resources to be created every time the module is deployed. This example creates an AWS Lambda-backed custom resource in CloudFormation that performs the initial seeding of a lookup table in Amazon DynamoDB. Custom resources can be complex and verbose to create, so it makes sense to reuse these components wherever possible.

AWSTemplateFormatVersion: "2010-09-09" Description: DynamoDB Data Seeder Parameters: DynamoDBTable: Description: Target DynamoDB Table Type: String Resources: DataSeederLambdaFunction: Type: AWS::Lambda::Function Properties: Code: ZipFile: | var AWS = require('aws-sdk'); var dynamodb = new AWS.DynamoDB(); exports.handler = (event, context, callback) => { console.log("REQUEST RECEIVED:n" + JSON.stringify(event)); if (event.RequestType == "Delete") { sendResponse(event, context, "SUCCESS"); return; } var responseStatus = "FAILED"; var responseData = {}; var tableName = process.env.TABLE_NAME; var params = { RequestItems: { [tableName]: [ { PutRequest: { Item: { "IndustryId": { S: "a8adda08-9746-4f35-aad6-2a68ed51ac77" }, "IndustrySegment": { S: "Agriculture" }, "IndustrySubType": { S: "Farming" }, } } }, { PutRequest: { Item: { "IndustryId": { S: "a446065a-a56f-4d4a-956b-242c82470b66" }, "IndustrySegment": { S: "Agriculture" }, "IndustrySubType": { S: "Forestry" }, } } }, { PutRequest: { Item: { "IndustryId": { S: "71759529-4471-427d-88ce-0f333460b832" }, "IndustrySegment": { S: "Financial Services" }, "IndustrySubType": { S: "Banking" }, } } }, { PutRequest: { Item: { "IndustryId": { S: "c154fb63-8f56-4ded-9e1f-88225401d320" }, "IndustrySegment": { S: "Financial Services" }, "IndustrySubType": { S: "Insurance" }, } } } ] } }; dynamodb.batchWriteItem(params, function(err, data) { if (err) { responseData = {Error: "Loading data failed."}; console.log(responseData.Error + ":n", err); } else { responseStatus = "SUCCESS"; } sendResponse(event, context, responseStatus, responseData); }); } function sendResponse(event, context, responseStatus, responseData) { var responseBody = JSON.stringify({ Status: responseStatus, Reason: "See the details in CloudWatch Log Stream: " + context.logStreamName, PhysicalResourceId: context.logStreamName, StackId: event.StackId, RequestId: event.RequestId, LogicalResourceId: event.LogicalResourceId, Data: responseData }); console.log("RESPONSE BODY:n", responseBody); var https = require("https"); var url = require("url"); var parsedUrl = url.parse(event.ResponseURL); var options = { hostname: parsedUrl.hostname, port: 443, path: parsedUrl.path, method: "PUT", headers: { "content-type": "", "content-length": responseBody.length } }; console.log("SENDING RESPONSE...n"); var request = https.request(options, function(response) { console.log("STATUS: " + response.statusCode); console.log("HEADERS: " + JSON.stringify(response.headers)); context.done(); }); request.on("error", function(error) { console.log("sendResponse Error:" + error); context.done(); }); request.write(responseBody); request.end(); } Handler: index.handler Runtime: nodejs12.x Timeout: 30 Role: !GetAtt "LambdaExecutionRole.Arn" Environment: Variables: TABLE_NAME: !Ref DynamoDBTable LambdaExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - sts:AssumeRole Path: / Policies: - PolicyName: root PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: arn:aws:logs:*:*:* - Effect: Allow Action: - dynamodb:BatchWriteItem Resource: !Join - "" - - "arn:aws:dynamodb:" - !Ref "AWS::Region" - ":" - !Ref "AWS::AccountId" - ":table/" - !Ref "DynamoDBTable" DataLoaderCustomResource: Type: Custom::DataLoader Properties: ServiceToken: !Join - "" - - "arn:aws:lambda:" - !Ref "AWS::Region" - ":" - !Ref "AWS::AccountId" - ":function:" - !Ref "DataSeederLambdaFunction"

Next, create a bucket where the CloudFormation module will be stored.

Open the Amazon S3 console and choose Create bucket. Under General configuration, enter a name for the bucket. Leave the other settings at their defaults and choose Create bucket.

Figure 3: Create bucket

Use the CloudFormation CLI to create the module deployment package. This will create the required schema file and zip the directory so that it can be submitted to the CloudFormation registry. Run the following command from the root of the module directory you created earlier:

cfn submit --dry-run

On the Upload page, choose Add files and then navigate to the ZIP file.

Figure 4: Upload to S3

Grant your organization access to the S3 bucket

With the module now deployed to the bucket, grant the organization access to it. Edit the S3 bucket’s policy to allow requests from the organization.

In the S3 Console, locate the bucket created in the previous step. Choose the bucket, and then choose the Permissions tab. In Bucket Policy, add a statement to this policy to allow access from the organization. This allows any accounts in the organization to access this artifact when deploying this CloudFormation module. Replace the placeholder with your organization ID:

{ "Version":"2012-10-17", "Statement":[ { "Sid": "AllowGetObject", "Effect": "Allow", "Principal": { "AWS": "*" }, "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::mymodulebucket-abc123/*", "arn:aws:s3:::mymodulebucket-abc123" ], "Condition": { "StringEquals": { "aws:PrincipalOrgID": "o-abc123ab" } } } ]
}

Create the CloudFormation template for the stack set

Now share the module with the organization using a stack set. The accounts in the organization can then consume and provision the module as part of their CloudFormation templates.

The template for the stack set is the same as any other CloudFormation template. It contains ModuleVersion, which is the resource type used to create CloudFormation modules through a CloudFormation template. (Parameterizing this template allows reuse for other modules.)

AWSTemplateFormatVersion: 2010-09-09 Parameters: ModuleBucketName: Type: String Description: S3 Bucket Name for CloudFormation Module Deployment Packages ModuleName: Type: String Description: Name of the CloudFormation Module to be created Default: "MyOrganization::MyNamespace::MyApp::MODULE" ModuleKey: Type: String Description: Object Key for CloudFormation Module Package in S3 Resources: AppModuleVersion: Type: "AWS::CloudFormation::ModuleVersion" Properties: ModuleName: !Ref ModuleName ModulePackage: !Join ["/", ["s3:/", !Ref ModuleBucketName, !Ref ModuleKey]]

Deploy the stack set to the OUs and AWS Regions

Now deploy the stack set to the target OUs and AWS Regions.

In the left pane of the AWS CloudFormation console, choose StackSets. Choose Service-managed and then choose Create StackSet. In Choose a template, choose Upload a template file and then choose the template you created in the previous step.

Figure 5: Choose a template

In Specify StackSet details, enter a name and parameters required by the CloudFormation template:

• The name of your S3 bucket.
• The name of the ZIP file you uploaded to the S3 bucket.
• The name of the module type you created earlier.

Figure 6: Specify StackSet details

On the Configure StackSet Options page, choose service-managed permissions to allow CloudFormation to create the required resources automatically.

On Add stacks to StackSet, choose whether to deploy the module to all accounts in an organization or to specific OUs. For Automatic deployment, choose Enabled. In Specify regions, choose the AWS Regions in which you want the module to be available. Under Deployment Options select whether to deploy the stack set serially or in parallel.

Figure 7: Add stacks to StackSet

Verify all details are correct on the Review page and choose Submit.

It will take a few minutes to deploy to the additional accounts and Regions. If you get errors, make sure you have enabled all features of AWS Organizations and enabled trusted access. Make sure that the account you are using is a delegated administrator for CloudFormation StackSets in the organization. See the walkthrough prerequisites.

Create stacks using the new module in any deployed account and Region

The module is now available for consumption in any of the AWS accounts and Regions deployed in the previous step.

You can now reference this module in a new CloudFormation template deployed in a different AWS account and Region. By doing so, you will save yourself the time and effort of rewriting the definition of the module.

AWSTemplateFormatVersion: 2010-09-09 Description: DynamoDB Data Seeder Parameters: AppName: Description: App Name for Resource Tagging Type: String Default: WidgetApp Resources: IndustryDataAppTable: Type: AWS::DynamoDB::Table Properties: AttributeDefinitions: - AttributeName: "IndustrySegment" AttributeType: "S" - AttributeName: "IndustrySubType" AttributeType: "S" KeySchema: - AttributeName: "IndustrySegment" KeyType: "HASH" - AttributeName: "IndustrySubType" KeyType: "RANGE" ProvisionedThroughput: ReadCapacityUnits: 5 WriteCapacityUnits: 5 TableName: !Join ["", [!Ref AppName, IndustryDataAppTable]] Tags: - Key: AppName Value: !Ref AppName DataSeeder: Type: MyOrganization::MyNamespace::DataSeeder::MODULE Properties: DynamoDBTable: !Ref IndustryDataAppTable

When the stack deployment is complete, you can see the default data in DynamoDB. This data was seeded into the table by the custom resource defined in the module.

Figure 8: Events tab

Choose the DynamoDB table and then choose the Items tab. Ensure that the items are displayed in the table.

Figure 9: WidgetAppIndustryDataAppTable

Conclusion

Code reuse is an important part of programming. Code reuse leads to shorter development times and ultimately, to a more secure and better-tested end application. This is no different for IaC templates like CloudFormation.

In this post, I’ve shown how you can publish modules to a central repository and then propagate those modules to other business units, facilitating the reuse of common template building blocks or approved sets of resources.

As a next step, implement a CI/CD pipeline that validates the module, uploads it to the central repository in S3, and then deploys it using StackSets to the member accounts in the organization. All a module author would need to do is to check the module into a source control repository, such as AWS CodeCommit.

If your organization is using AWS Control Tower, an alternative approach is to use the Customizations for Control Tower solution to orchestrate the stack set deployment.

About the author