Automate customized deployment of cross-account/cross-region CloudWatch Dashboards using tags

Amazon Web Services Feed
Automate customized deployment of cross-account/cross-region CloudWatch Dashboards using tags

Amazon CloudWatch Dashboards are a great way to monitor your AWS resources. During peak events when you are expecting high traffic, monitoring your AWS resources helps you stay ahead of any issues that may arise. You might want a customized and automated dashboard that can be used during a seasonal event, important releases, holidays, and other dates important to your operations. Setting up a dashboard can be a repetitive and time consuming process. A dashboard can be difficult to manage and keep up to date. In this blog post, we will discover how to use tagging, the cross-account and cross-region functionality in Amazon CloudWatch, and a Lambda function triggered by Amazon EventBridge to generate a dashboard automatically from your tagged resources.

Solution overview 

For purposes of this post, we have a monitoring account where the CloudWatch dashboard resides. We have accounts referred to as X, Y, and Z from which we collect data. Our objective is to have the CloudWatch dashboard contain aggregate metrics from all the accounts in the scope of the event. One dashboard in the monitoring account collects data from the other shared accounts. Any resources in the monitoring account can be included in the dashboard, too.

If you follow the steps in this post, you’ll set up CloudWatch data sharing in accounts X, Y, and Z. You’ll also set up CloudWatch in the monitoring account so you can view the shared data. Then, you’ll tag resources in all accounts. Lastly, you’ll create IAM roles in accounts X, Y, and Z that will be assumed by an IAM user in the monitoring account to check for tagged resources and collect data from these accounts. A Lambda function will be created in the monitoring account that will collect the data for the CloudWatch dashboard. You’ll set a schedule in Amazon EventBridge to automate the running of the Lambda function to keep the dashboard up to date.

Lambda function out of the box will support monitoring the following services [EC2, RDS, Lambda, ElastiCache, CLB, ALB, NLB] and specific metrics.

Figure 1: Solution architecture

Solution steps and deployment

The solution architecture above shows the following components and steps:

1. In accounts X, Y, and Z, set up cross-account functionality in CloudWatch to share data with the monitoring account.
2. In the monitoring account, set up cross-account functionality in CloudWatch to access the shared data from accounts X, Y, and Z.
3. Tag your AWS resources.
4. In accounts X, Y, and Z, create an IAM role, AllowMonitoringAccountAccess, that provides access to the monitoring account.
5. In the monitoring account, create IAM policies (CrossAccountDashboardDiscoveryPolicy, CloudWatchDashboardCustomPolicy, and IAMCustomPolicy).
6. In the monitoring account, create a Lambda function and update the IAM policy for the function.
7. In the monitoring account, configure Amazon EventBridge.

Step 1: In accounts X, Y, and Z, set up cross-account functionality in CloudWatch to share data with the monitoring account

Cross-account functionality is integrated with AWS Organizations to help efficiently build your cross-account dashboards. In this blog post, we do not use AWS Organizations. Because cross-Region functionality is now built in to CloudWatch, no further action is required.

1. Sign in to accounts X, Y, and Z.
2. In the CloudWatch console, choose Settings, and then under Cross-account cross-region, choose Configure.
3. Under Share your CloudWatch data, choose Share data.
4. Under Sharing, choose Specific accounts, and then choose Add account. Enter the monitoring account ID.
5. Under Permissions, keep the defaults.
6. Under Create CloudFormation Stack, choose Launch CloudFormation template.
7. On the confirmation page, type Confirm, and then choose Launch template.
8. Select the I acknowledge check box, and then choose Create stack.

For more information, see Enable Cross-Account Functionality in CloudWatch in the Amazon CloudWatch User Guide.

Step 2: In the monitoring account, set up cross-account functionality in CloudWatch to access the shared data from accounts X, Y, and Z

1. Sign in to the monitoring account.
2. In the CloudWatch console, choose Settings, and then under Cross-account cross-region, choose Configure.
3. Under View cross-account cross-region, choose Enable.
4. Under Enable account selector, choose Custom account selector, and then enter the accounts you will be monitoring.  Example: 012345678912 My account label, 987654321012 My other account
5. Choose Enable.

For more information, see Enable Cross-Account Functionality in CloudWatch in the Amazon CloudWatch User Guide.

Step 3: Tag your AWS resources

1. Sign in to each account and tag the resources you would like to monitor. In this blog post, we cover the following AWS services and resources: Amazon EC2, Amazon RDS, AWS Lambda, Amazon ElastiCache, Classic Load Balancer, Application Load Balancer, Network Load Balancer.
2. For the tag key, useevent. For the tag value, usespecialevent. If you decide to customize the tagging, be sure to update lines 12 and 13 of the Lambda function code.

For more information, see Tagging AWS resources in the AWS General Reference.

Step 4: In accounts X, Y, and Z, create the AllowMonitoringAccountAccess role to provide access to the monitoring account

Sign in to accounts X, Y, and Z and create an IAM role that allows the monitoring account access to view the resources.

1. Sign in to accounts X, Y, and Z.
2. In the IAM console, choose Roles, and then choose Create role.
3. Choose Another AWS account, and then enter the account ID of the monitoring account.
4. Choose Next: Permissions.
5. Search for and choose the following policies: CloudWatchReadOnlyAccess and ResourceGroupsandTagEditorReadOnlyAccess
6. Choose Next: Tags.
7. Choose Next: Review.
8. For the role name, enter AllowMonitoringAccountAccess. For the role description, enterRole will allow read-only access to the monitoring account for building a CloudWatch dashboard.
9. Review the role, and then choose Create role.
10. Perform these steps for accounts X, Y, and Z. In each account, find the role you created and copy its ARN. You need it in Step 5.
    • Example Account X: arn:aws:iam::012345678912:role/AllowMonitoringAccountAccess
    • Example Account Y: arn:aws:iam::987654321012:role/AllowMonitoringAccountAccess
    • Example Account Z: arn:aws:iam::123456789123:role/AllowMonitoringAccountAccess

For more information, see Creating a role to delegate permissions to an IAM user in the IAM User Guide.

Step 5: Create CrossAccountDashboardDiscoveryPolicy,  CloudWatchDashboardCustomPolicy, and IAMCustomPolicy in the monitoring account

In this step, you create three IAM policies in the monitoring account. You attach these policies to the Lambda function in Step 6.

To create CrossAccountDashboardDiscoveryPolicy:

1. Sign in to the monitoring account.
2. In the IAM console, choose Policies, and then choose Create policy.
3. Choose the JSON tab.
4. Edit the following template with the ARNs you collected in Step 4.
   Here is an example policy for one account:

   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": [ "arn:aws:iam::012345678912:role/AllowMonitoringAccountAccess" ] } ]
   }
   

   Here is an example policy for more than one account:

   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": [ "arn:aws:iam::012345678912:role/AllowMonitoringAccountAccess", "arn:aws:iam::987654321012:role/AllowMonitoringAccountAccess", "arn:aws:iam::123456789123:role/AllowMonitoringAccountAccess" ] } ]
   }

5. Choose Review Policy, and on the Review policy page, enter a name CrossAccountDashboardDiscoveryPolicy and optional description.
6. Choose Create policy.

For more information, see Creating IAM policies in the IAM User Guide.

To create the CloudWatchDashboardCustomPolicy:

1. Sign in to the monitoring account.
2. In the IAM console, choose Policies, and then choose Create policy.
3. Choose the JSON tab.
4. Edit the following template with the account number of your monitoring account and then paste it into the field on the JSON tab.

   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "cloudwatch:PutDashboard", "Resource": "arn:aws:cloudwatch::000111222333:dashboard/*" } ]
   }

5. Choose Review Policy, and on the Review policy page, enter a name CloudWatchDashboardCustomPolicy and an optional description.
6. Choose Create policy.

For more information, see Creating IAM policies in the IAM User Guide.

To create the IAMCustomPolicy:

1. Sign in to the monitoring account.
2. In the IAM console, choose Policies, and then choose Create policy.
3. Choose the JSON tab.
4. Edit the following template with the account number of your monitoring account and then paste it into the field on the JSON tab.

   { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "iam:GetPolicyVersion", "iam:GetPolicy" ], "Resource": "arn:aws:iam::000111222333:policy/CrossAccountDashboardDiscoveryPolicy" } ]
   }

5. Choose Review Policy, and on the Review policy page, enter a name IAMCustomPolicy and an optional description.
6. Choose Create policy.

For more information, see Creating IAM policies in the IAM User Guide.

Step 6: Create a Lambda function and update the IAM policy for the function in the monitoring account

1. Sign in to the monitoring account.
2. In the AWS Lambda console, choose Functions, and then choose Create a function.
3. Leave Author from scratch selected. For Function name, enter AutomateCloudWatchDashboards. For Runtime, choose Python 3.8.
4. Expand Change default execution role, make a note of the IAM role that will be created for this Lambda function (for example, AutomateCloudWatchDashboards-role-91t3fwgj), and then choose Create function.
5. Under Basic Settings, choose Edit. Update Timeout to 15 seconds, and then choose Save.
6. Copy and paste the content of file cwautomatedashboard.py located at GitHub, and then choose Deploy.
7. In the IAM console, update the IAM role created by the Lambda function (for example, AutomateCloudWatchDashboards-role-91t3fwgj), and then attach the following IAM polices. Note: There will already be one managed policy, AWSLambdaBasicExecutionRole-****, attached to this role.
   CrossAccountDashboardDiscoveryPolicy, CloudWatchDashboardCustomPolicy, IAMCustomPolicy, and ResourceGroupsandTagEditorReadOnlyAccess
8. In the AWS Lambda console, choose Lambda function. Choose AutomateCloudWatchDashboards, and then choose Test.
9. For Configure test event, enter a name for the event, and then choose Create.
10. Choose Test and confirm the function ran successfully.
    Note: The Lambda function looks for resources in us-east-1, us-east-2, us-west-1, and us-west-2. You can edit Region information in line 14 of the Lambda function.

For more information, see Create a Lambda function with the console in the AWS Lambda Developer Guide.

Step 7: Configure EventBridge in the monitoring account

In the monitoring account where the Lambda function is located, add a trigger for EventBridge (CloudWatch Events) to make the Lambda function run every 5 minutes. If you make add or remove tags, the CloudWatch dashboard will be automatically updated at regular intervals. You can customize the trigger time to your requirements.

1. Sign in to the monitoring account.
2. In the AWS Lambda console, choose Functions.
3. Choose AutomateCloudWatchDashboards.
4. In the Designer section, choose Add trigger, and then choose a trigger of EventBridge (CloudWatch Events).
5. Under Rule, choose Create a new rule.
6. For Rule name, enter EventBridgeAutomateCloudWatchDashboards. For Rule type, choose Schedule expression. You can enter the expression that best fits your use case. In this post, we use every 5 minutes.

For more information, see Schedule AWS Lambda Functions Using EventBridge in the Amazon EventBridge User Guide.

Conclusion

In this blog post, we walked through the steps to configure Amazon CloudWatch to share data with the monitoring account. We showed you how to create the IAM roles and polices that are required to provide access to collect data. This solution solves the problem of manually managing and updating a CloudWatch dashboard. By using tags and the automation of EventBridge and Lambda, the work is done for you.

After the solution has been deployed and all the resources we want to monitor have been tagged, here are two example CloudWatch dashboards:

Figure 2: Example dashboard

Figure 3: Second example dashboard

About the Authors

Salman Ahmed is a Technical Account Manager within AWS Enterprise Support. He enjoys working with Enterprise Support customers to help them with design, implementation and supporting cloud infrastructure.  He also has a passion for networking services and with 10+ years of experience he leverages that to help customers with adoption of AWS Transit Gateway and AWS Direct Connect services.

Mike Gomez is an Enterprise Support Lead within AWS Enterprise Support. Works with with Enterprise Customers on achieving and maintaining Operational Excellence and has a passion for Reliability Engineering and IT Operations. With a background in Travel & Hospitality, Media and Entertainment and Banking he focuses on helping customers achieve their business goals through cross-industry innovation.