Jul 30 2020
0

Amazon Web Services Feed
AWS CloudFormation support for AWS Service Catalog products

This blog post was updated on 7/21/2020 to reflect recent changes to how AWS Service Catalog obtains outputs from provisioned products. For more information see Provisioned product outputs are now available in AWS Service Catalog.

You can use AWS Service Catalog to create preconfigured products that your developers can launch. In a large organization, it’s typical for a cross-functional team like a Cloud Center of Excellence (CCOE) to maintain the catalog for the organization. An AWS Service Catalog product can contain one or more AWS resources. Many customers use AWS Service Catalog to restrict access to resources, such as AWS APIs, using a launch constraint. Launch constraints allow an AWS Service Catalog end user to launch an AWS Service Catalog product without requiring elevated permissions to AWS resources.

An AWS Service Catalog product also allows a CCOE to enforce configuration standards within a customer’s products, while granting development teams flexibility to customize AWS resources using parameters. Despite the flexibility of parameters, it is difficult for a CCOE to determine every development team’s architecture patterns and requirements. Accommodating the broad requirements of development teams leads to a large portfolio of AWS Service Catalog products that need to be maintained.

Recently, we announced AWS CloudFormation support for AWS Service Catalog products. In this blog post, I’ll walk you through how to leverage this new feature to provide development teams the freedom to create complex architectures. The building blocks for development teams will be AWS Service Catalog products configured as simple hardened components. Using simple components will help the CCOE reduce their workload while implementing preventative controls that manage an organization’s risk.

 

Why wouldn’t I simply use native AWS CloudFormation or native AWS Service Catalog to provision resources?


For many customers, the CCOE is responsible for maintaining the AWS environment. This includes establishing security, governance, and operating controls that allow the business to leverage AWS at scale while managing risk. Typically, the CCOE owns the whitelisting process that approves the use of AWS services. The output of this process is a directive control—namely, the configuration standards and guidelines that development teams must follow when they use the AWS service. Establishing these controls takes time because the CCOE must evaluate each AWS service.

Standards and guidelines are a great start, but ultimately the business needs to ensure that directive controls are followed by creating preventative, detective, and reactive controls. The most important of these controls are preventative controls. Why allow bad things to happen if we can prevent them? The easiest preventative control is simply not giving a user permission to use an AWS service. Ideally, your preventative controls are more nuanced than this because these controls can prevent the business benefits of an AWS service from being realized.

Next, you need to decide where to place the preventative control. Mature AWS customers leverage continuous integration/continuous deployment (CI/CD) pipelines and AWS CloudFormation to deploy into AWS. Like the applications they support, CI/CD pipelines come in all shapes and sizes. In the short term, fitting preventative controls into each development team’s CI/CD pipeline is time consuming and potentially disruptive. The diagram below shows where you would place preventative controls in a simplistic CI / CD pipeline.

CI/CD

 

A simple component configured with preventative controls in Service Catalog streamlines developer adoption of these hardened AWS service configurations, while providing the flexibility for developers to design their own architectures. The diagram below shows the shift of preventative controls from the existing CI / CD pipeline into a Service Catalog product.

CI/CD

 

To further highlight the advantages of using CloudFormation support for AWS Service Catalog products, I’ll review common methods for deploying AWS resources with this new feature:

  1. Native AWS CloudFormation
  2. Native AWS Service Catalog products
  3. AWS CloudFormation support for AWS Service Catalog products

Using AWS CloudFormation to provision AWS resources

The following diagram illustrates web and application tiers deployed using native AWS CloudFormation. The AWS CloudFormation template describes the AWS resources in the colored box above it.

 

Pros and cons of this method:

  • Pros
    • The CCOE does not need to write infrastructure as code for development teams.
    • Developers write infrastructure as code for their applications; this gives them the flexibility to create architectures that meet their needs. This model is easier to scale than having a central team write all the infrastructure as code.
  • Cons
    • Developers require permissions to directly access APIs for resources they are creating.
      • It is difficult to implement preventative controls for resource configuration.
      • Enterprises may view this as too much risk, especially when they are early in their cloud adoption journey.
    • Developers need to understand the configuration options for each AWS resource.
    • The CCOE needs to create detective and reactive controls to ensure that AWS services are used in a manner consistent with the company’s standards.

Using AWS Service Catalog to provision AWS resources

The following diagram illustrates web and application tiers deployed using a single AWS Service Catalog product.

 

Pros and cons of this method:

  • Pros
    • The CCOE is able to restrict access to AWS APIs, while still providing developers with the ability to provision AWS resources.
    • It’s easy to ensure that AWS resources adhere to security / governance standards.
    • Developers can launch AWS Service Catalog products from the console without writing any code.
    • Developers do not need to understand the configuration options for each AWS resource.
  • Cons
    • There’s less flexibility for developers. Developers can’t use AWS CloudFormation to describe their infrastructure.
    • The CCOE needs to create a product per architecture.
      • This may not meet developer requirements.
      • The complexity and size of the AWS Service Catalog product is increased.
      • Workload for the CCOE is increased.

Using AWS CloudFormation support to provision AWS Service Catalog products

This diagram illustrates combining the previous two provisioning methods to deploy the web and application tiers using AWS CloudFormation support for AWS Service Catalog products.

Pros and cons of this method:

  • Pros
    • Restricts access to AWS APIs, while still providing developers with ability to provision AWS resources.
    • Easy to ensure that AWS resources adhere to security / governance standards.The CCOE is responsible for creating simple components instead of complete architectures.
    • Developers write their own infrastructure as code for their applications. Using parameters, they can customize the simple components to meet their needs.
  • Cons
    • The CCOE needs to provide development teams with AWS Service Catalog resources IDs.
    • Developers need to understand the configuration options for each AWS resource.

How to create simple component products in AWS Service Catalog

Before a development team can launch an AWS Service Catalog product, the CCOE will need to create the AWS Service Catalog portfolio and products. Here is an outline of the steps I used to set up my Service Catalog environment and share it with an end user:

  1. Create a portfolio for your product.
  2. Create the Application Load Balancer (ALB) product.
  3. Create the Auto Scaling group product.
  4. Add both products to your “Dev” portfolio.
  5. Create launch constraints for the Application Load Balancer product.
  6. Create launch constraints for the Auto Scaling group product.
  7. Grant the appropriate user, group, or role permissions to the portfolio.

In this section, I’ll provide the commands and the input parameters for each command. I’m using the cli-input-json parameter to improve the readability of the parameters for each command. If you did not know the format for the parameter files, you could use the generate-cli-skeleton parameter with each command to obtain it (example: aws servicecatalog create-portfolio –generate-cli-skeleton).

Step 1: Create a portfolio for your product

Now let’s create a portfolio called Development Whitelisted Services. Make sure to capture the portfolio ID that is returned. I’ve highlighted this in the output section that follows.

Command:
aws servicecatalog create-portfolio --cli-input-json file://portfolio_params.json
File: Portfolio_params.json

{ "DisplayName": "Development Whitelisted Services", "Description": "Development Whitelisted Services", "ProviderName": "Internal", "IdempotencyToken": "devport_v1"
}
Output:

{ "PortfolioDetail": { "DisplayName": "Development Whitelisted Servicest", "Description": "Development Whitelisted Servicest", "ProviderName": "Internal", "CreatedTime": 1525833224.251, "Id": "port-e3btb2ahk7mhm", "ARN": "arn:aws:catalog:us-east-1:123456780912:portfolio/port-e3btb2ahk7mhm" }
}

 

Step 2: Create the Application Load-Balancer product

When you create the product you will be required to specify the Amazon S3 location of the AWS CloudFormation template that describes the AWS resources the product will create when launched. Make sure to save the ProductId and the ProvisioningArtifactDetail ID from this step. These values are required inputs for the AWS CloudFormation template that launches these products. I’ve highlighted these values in the output section that follows.

Command:
aws servicecatalog create-product --cli-input-json file://ALB_product_params.json
File: ALB_product_params.json
{ "Name": "Application-Load-Balancer", "Owner": "COE", "Description": "Hardened Application Load Balancer ", "SupportDescription": "Contact COE for assistance", "SupportEmail": "coe@mycompany.com", "SupportUrl": "https://support.mysite.com", "ProductType": "CLOUD_FORMATION_TEMPLATE", "ProvisioningArtifactParameters": { "Name": "v1", "Description": "Application Load Balancer Initial Version", "Info": { "LoadTemplateFromURL": "https://s3.amazonaws.com/REPLACE_WITH_YOUR_BUCKET/sc_alb_product_creation.yml" }, "Type": "CLOUD_FORMATION_TEMPLATE" }, "IdempotencyToken": "v1" }

 

Output:
{ "ProductViewDetail": { "ProductViewSummary": { "SupportDescription": "Contact COE for assistance", "Name": "Application-Load-Balancer", "HasDefaultPath": false, "ShortDescription": "Hardened Application Load Balancer ", "SupportUrl": "https://support.mysite.com", "Owner": "COE", "SupportEmail": "coe@mycompany.com", "Type": "CLOUD_FORMATION_TEMPLATE", "Id": "prodview-kl2oub4hevt5m", "ProductId": "prod-6w72oh5zxhhq" }, "Status": "CREATED", "ProductARN": "arn:aws:catalog:ap-southeast-1:123456780912:product/prod-6w72oh5zxhhq", "CreatedTime": 1525141671.0 }, "ProvisioningArtifactDetail": { "CreatedTime": 1525141671.0, "Description": "Application Load Balancer Initial Version", "Type": "CLOUD_FORMATION_TEMPLATE", "Id": "pa-24rn64zlqrlim", "Name": "v1" }
}

 

Step 3: Create the Auto Scaling group product

When you create the product, you will be required to specify the Amazon S3 location of the AWS CloudFormation template that describes the AWS resources the product will create when launched. Make sure to save the ProductId and the ProvisioningArtifactDetail ID from this step. These values will be required inputs for the AWS CloudFormation template that creates launches these products. I’ve highlighted these values below in the output.

Command:
aws servicecatalog create-product --cli-input-json file://ASG_product_params.json
File: ASG_product_params.json
{ "Name": "Auto-Scaling-Group", "Owner": "COE", "Description": "Hardened Auto Scaling Group", "SupportDescription": "Contact COE for assistance", "SupportEmail": "coe@mycompany.com", "SupportUrl": "https://support.mysite.com", "ProductType": "CLOUD_FORMATION_TEMPLATE", "ProvisioningArtifactParameters": { "Name": "v1", "Description": "Auto-Scaling-Group Initial Version", "Info": { "LoadTemplateFromURL": "https://s3.amazonaws.com/REPLACE_WITH_YOUR_BUCKET1/sc_asg_product_creation.yml" }, "Type": "CLOUD_FORMATION_TEMPLATE" }, "IdempotencyToken": "asgv1"
}
Output:
{ "ProductViewDetail": { "ProductViewSummary": { "SupportDescription": "Contact COE for assistance", "Name": " Auto-Scaling-Group", "HasDefaultPath": false, "ShortDescription": "Hardened Auto Scaling Group", "SupportUrl": "https://support.mysite.com", "Owner": "COE", "SupportEmail": "coe@mycompany.com", "Type": "CLOUD_FORMATION_TEMPLATE", "Id": "prodview-jvevyj6gxhqqy", "ProductId": "prod-z4cfjwincopw" }, "Status": "CREATED", "ProductARN": "arn:aws:catalog:ap-southeast-1:123456780912:product/prod-z4cfjwincopw", "CreatedTime": 1525141671.0 }, "ProvisioningArtifactDetail": { "CreatedTime": 1525141671.0, "Description": “Auto-Scaling-Group Initial Version", "Type": "CLOUD_FORMATION_TEMPLATE", "Id": "pa-t7dgaiuzm2dgs", "Name": "v1" }
}

 

Step 4: Add both products to my “Dev” portfolio

I need to associate my products with my portfolio. Access to products is granted at the portfolio level. I’m using the portfolio ID from step 1 and the product IDs from steps 2 and 3.

Commands:

aws servicecatalog associate-product-with-portfolio --cli-input-json file://associate_asg_params.json aws servicecatalog associate-product-with-portfolio --cli-input-json file://associate_alb_params.json

 

File: associate_alb_params.json
{ "ProductId": "prod-6w72oh4zxhhhq", "PortfolioId": "port-e3btb2ahk7mhm"
}

 

File: associate_asg_params.json
{ "ProductId": " prod-z4cfjwincopw ", "PortfolioId": "port-e3btb2ahk7mhm"
}

 

Output:

No text is returned if the command is successful.

 

Step 5: Create launch constraints for the Application-Load-Balancer product

The launch constraint associates an AWS Identity and Access Management (IAM) role that contains the permissions necessary to launch the product. By leveraging this feature, developers will not require permissions to AWS APIs. I’m assuming the role has been previously created.

Command:

aws servicecatalog create-constraint --cli-input-json file://create-launch-constraint_alb_params.json

 

File: create-launch-constraint_alb_params.json
{ "PortfolioId": "port-e3btb2ahk7mhm", "ProductId": "prod-6w72oh4zxhhhq", "Parameters":"{"RoleArn":"arn:aws:iam::123456780912:role/sc_alb_creation_role"}", "Type": "LAUNCH", "Description": "Grant SC permission to launch the ALB"
}

 

Output:
{ "Status": "CREATING", "ConstraintParameters": "{"RoleArn":"arn:aws:iam:: 123456780912:role/sc_alb_creation_role"}", "ConstraintDetail": { "Owner": "123456780912", "ConstraintId": "cons-63xhtbppz63au", "Type": "LAUNCH", "Description": "Grant SC permission to launch the ALB" }
}

 

Step 6: Create launch constraints for the Auto Scaling group product

The launch constraint associates an IAM role that contains permissions necessary to launch the product. I’m assuming the role has been previously created

Command:

aws servicecatalog create-constraint --cli-input-json file://create-launch-constraint_asg_params.json

 

File: create-launch-constraint_asg_params.json
{ "PortfolioId": "port-e3btb2ahk7mhm", "ProductId": " prod-z4cfjwincopw", "Parameters":"{"RoleArn":"arn:aws:iam::123456780912:role/sc_asg_creation_role"}", "Type": "LAUNCH", "Description": "Grant SC permission to launch the ALB"
}

 

Output:
{ "Status": "CREATING", "ConstraintParameters": "{"RoleArn":"arn:aws:iam:: 123456780912:role/sc_asg_creation_role"}", "ConstraintDetail": { "Owner": "123456780912", "ConstraintId": "cons-iafbtiuo32isc", "Type": "LAUNCH", "Description": "Grant SC permission to launch the ASG" }
}

 

Step 7: Grant the appropriate user, group, or role permissions to the portfolio

Before users can launch the products I need to grant them permissions.

Command:

aws servicecatalog associate-principal-with-portfolio --cli-input-json file://associate_principal_params.json

 

File: associate_principal_params.json
{ "PortfolioId": "port-e3btb2ahk7mhm", "PrincipalARN": "arn:aws:iam::123456780912:role/Developers", "PrincipalType": "IAM"
}

 

Output:

No text is returned if the command is successful.

 

How to obtain the product ID or provisioning artifact ID from the CLI

If you didn’t capture the product ID or provisioning artifact ID when you created your product, you can use the following commands to find them. Notice for each command I am using the –query option to reduce the size of the response. The first command helps me find the product ID.

Command

aws servicecatalog search-products-as-admin --query 'ProductViewDetails[*][ProductViewSummary][?starts_with(Name,`Application-Load-Balancer`) == `true`].{Name:Name,Id:ProductId}'

 

Output
[ [ { "Name": "Application-Load-Balancer", "Id": "prod-6w72oh4zxhhhq" } ], []
]

 

This second command helps me find the provisioning artifact ID. I’ll need to use the product ID I obtained earlier (prod-6w72oh4zxhhhq) as an input for this command. The describe-product-as-admin command will return multiple artifacts, once for each product version. The query parameter can help reduce the output.

Command

aws servicecatalog describe-product-as-admin --id prod-6w72oh4zxhhhq --query 'ProvisioningArtifactSummaries[?Name==`v12`].{Name:Name,Id:Id}'

 

Output
[ { "Name": "v12", "Id": "pa-2mb2bgvx6a6ds" }
]

 

The CCOE will need to provide the development teams the product ID, provisioning artifact ID, and the list of parameters for each product. Keep in mind that the provisioning artifact ID is going to change with each update of the product. These values will be unique to each AWS account/Region the development teams uses. You will want to implement a scalable process that avoids development teams having to ask the CCOE for these values. Many customers use an AWS Lambda-backed custom resource to discover resource values based on parameters passed into the custom resource. Here is an example of an AWS Lambda function that can be invoked as a custom resource.  It returns the product Id and provision artifact ID back to AWS CloudFormation.  To keep my example simple, I’m going to pass these values in using AWS CloudFormation parameters.

 

Using the AWS Service Catalog simple component products to create the infrastructure

I’m going to use a single AWS CloudFormation template to launch my AWS Service Catalog products. I’m not explicitly including tags in this template. I would expect they would be passed during the create-stack API call or configured as part of the AWS Service Catalog product. As mentioned earlier, product ID and product artifact IDs are required inputs. I’m passing these in as parameters. In a real-world scenario there may be dependencies between the components we would want to consider.

In this example I’m going to reference the target group Amazon Resource Name (ARN) created in the ALB template when I create my Auto Scaling group.

Snippet from the ALB product’s CloudFormation output section: In this snippet I am outputting the ALB Target Group ARN.   I will pass this value to the Auto Scaling product.

Outputs: ALBTarget: Description: HelloWorld ELB Value: !Ref ALBTarget

 

Snippet from resource section of the CloudFormation template that deploys the solution: I can use the !GetAtt intrinsic function to obtain the ALB Target Group ARN created by AWS Service Catalog product and pass it into the Auto Scaling group parameters.

Resources:
ApacheASG:
Type: "AWS::ServiceCatalog::CloudFormationProvisionedProduct"
Properties:
ProvisioningParameters:
- Key: "ALBTargetGroup"
Value: !GetAtt ApacheALB.Outputs.ALBTarget

The following is the complete AWS CloudFormation template I am using to create my infrastructure from the AWS Service Catalog simple component products.

###################################################################################
# This template creates the web and application tiers for a three-tier web
# application using AWS Service Catalog products
################################################################################### AWSTemplateFormatVersion: 2010-09-09
Description: > This template creates the web and application tiers for a three-tier web application using AWS Service Catalog products # This section allows the development team to pass values into the template to customize the AWS Service Catalog Products
Parameters: ServerType: Description: App tier to deploy Apache/Tomcat Type: String Default: "apache" SNSTopicARN: Description: Notification topic ARN for monitoring and alerting Type: String Default: "arn:aws:sns:us-west-2:151490602336:NotifyMe" AppName: Description: Application Name Type: String Default: "HelloWorld" InfraTier: Description: Account group Type: String Default: "web" EnvNum: Description: Environment number Type: String Default: "1" AppInstanceType: Description: EC2 instance type Type: String AllowedPattern: "[a-z][0-9][.][-a-zA-Z0-9]+" Default: "t2.micro" EC2SecurityGroup: Description: Comma delimited list of security group IDs to assign to EC2 instances Type: String ConstraintDescription: Security groups must already exist Default: "sg-2960494d" OwnerEmail: Description: Email of user launching stack Type: String Default: "coe@mydomain.com" CertificateArn: Description: Certificate to apply to ELB if HTTPS Type: String Default: "arn:aws:acm:us-west-2:151490602336:certificate/4fdda395-7856-433c-83da-0943c8e0399a" ALBProductId: Description: Product Id for the ALB Product Type: String Default: "prod-tcrhldubey2a2" ALBArtifactId: Description: Artifact Id (version) for the ALB product Type: String Default: "pa-mcgm4s5vwrftw" ASGArtifactId: Description: Product Id for the ASG Product Type: String Default: "pa-pb2g62ofllmtk" ASGProductId: Description: Artifact Id (version) for the ASG product Type: String Default: "prod-sexyzwlytdk72" Resources: ApacheALB: Type: "AWS::ServiceCatalog::CloudFormationProvisionedProduct" Properties: # from aws servicecatalog search-products-as-admin ProductId: !Ref ALBProductId # from aws servicecatalog describe-product-as-admin --id ProvisioningArtifactId: !Ref ALBArtifactId ProvisioningParameters: - Key: "ServerType" Value: "apache" - Key: "SNSTopicARN" Value: !Ref SNSTopicARN - Key: "AppName" Value: !Ref AppName - Key: "InfraTier" Value: "web" - Key: "EnvNum" Value: !Ref EnvNum - Key: "EC2SecurityGroup" Value: !Ref EC2SecurityGroup - Key: "OwnerEmail" Value: !Ref OwnerEmail - Key: "CertificateArn" Value: !Ref CertificateArn NotificationArns: [!Ref SNSTopicARN] ApacheASG: Type: "AWS::ServiceCatalog::CloudFormationProvisionedProduct" Properties: # from aws servicecatalog search-products-as-admin ProductId: !Ref ASGProductId # from aws servicecatalog describe-product-as-admin --id ProvisioningArtifactId: !Ref ASGArtifactId #PathId: "lp-otyhtxohyirkc" ProvisioningParameters: - Key: "ALBTargetGroup" Value: !GetAtt ApacheALB.Outputs.ALBTarget - Key: "ServerType" Value: "apache" - Key: "SNSTopicARN" Value: !Ref SNSTopicARN - Key: "AppName" Value: !Ref AppName - Key: "InfraTier" Value: "web" - Key: "EnvNum" Value: !Ref EnvNum - Key: "DownstreamELB" Value: !Ref ApacheALB - Key: "AppInstanceType" Value: !Ref AppInstanceType - Key: "EC2SecurityGroup" Value: !Ref EC2SecurityGroup - Key: "OwnerEmail" Value: !Ref OwnerEmail NotificationArns: [!Ref SNSTopicARN] TomcatALB: Type: "AWS::ServiceCatalog::CloudFormationProvisionedProduct" Properties: # from aws servicecatalog search-products-as-admin ProductId: !Ref ALBProductId # from aws servicecatalog describe-product-as-admin --id ProvisioningArtifactId: !Ref ALBArtifactId #PathId: "lp-otyhtxohyirkc" ProvisioningParameters: - Key: "ServerType" Value: "tomcat" - Key: "SNSTopicARN" Value: !Ref SNSTopicARN - Key: "AppName" Value: !Ref AppName - Key: "InfraTier" Value: "web" - Key: "EnvNum" Value: !Ref EnvNum - Key: "EC2SecurityGroup" Value: !Ref EC2SecurityGroup - Key: "OwnerEmail" Value: !Ref OwnerEmail - Key: "CertificateArn" Value: !Ref CertificateArn NotificationArns: [!Ref SNSTopicARN] TomcatASG: Type: "AWS::ServiceCatalog::CloudFormationProvisionedProduct" Properties: # from aws servicecatalog search-products-as-admin ProductId: !Ref ASGProductId # from aws servicecatalog describe-product-as-admin --id ProvisioningArtifactId: !Ref ASGArtifactId #PathId: "lp-otyhtxohyirkc" ProvisioningParameters: - Key: "ALBTargetGroup" Value: !GetAtt TomcatALB.Outputs.ALBTarget - Key: "ServerType" Value: "tomcat" - Key: "SNSTopicARN" Value: !Ref SNSTopicARN - Key: "AppName" Value: !Ref AppName - Key: "InfraTier" Value: "web" - Key: "EnvNum" Value: !Ref EnvNum - Key: "DownstreamELB" Value: !Ref TomcatALB - Key: "AppInstanceType" Value: !Ref AppInstanceType - Key: "EC2SecurityGroup" Value: !Ref EC2SecurityGroup - Key: "OwnerEmail" Value: !Ref OwnerEmail NotificationArns: [!Ref SNSTopicARN]

 

Source Code

The solution described in this post including AWS CloudFormation templates, source code, deployment script and documentation can be download from GitHub.

 

Conclusion

Thanks for taking the time to read this blog post. I’ve shown you how AWS CloudFormation support for AWS Service Catalog provides you with the capability to preventatively implement security and governance controls in your AWS Service Catalog products, while granting developers the flexibility to create architectures that meet their applications’ requirements.

 

About the Authors

Jim Long is a Principal Cloud Architect in the AWS Professional Services Financial Services Practice based out of Boston Massachusetts. He works with large enterprise customers to accelerate their Cloud adoption journey.

 

 

 

Remek is a Senior Cloud Infrastructure Architect with Amazon Web Services Professional Services. He works with AWS financial enterprise customers providing technical guidance and assistance for Infrastructure, Security, DevOps, and Big Data to help them make the best use of AWS services. Outside of work, he enjoys spending time actively, and pursuing his passion – astronomy.

Posted inAWS News
TagsAWS

Leave a Reply

Your email address will not be published. Required fields are marked *