Use AWS Lambda and Amazon QuickSight to Build a Dashboard for AWS Health Events in Organizational View

AWS Feed
Use AWS Lambda and Amazon QuickSight to Build a Dashboard for AWS Health Events in Organizational View

Centralized DevOps teams responsible for the operation of Amazon Web Services (AWS) resources across an organization want to have a consistent approach for receiving and visualizing notifications for AWS Health events. It’s challenging and time-consuming to collect this data from individual accounts through email notifications, by managing separate event data, or even by manually clicking through multiple dashboards like AWS Personal Health Dashboard or AWS Service Health Dashboard.

If you use AWS Organizations, you can view AWS Health events centrally across your organization. Organizational view allows you to consolidate those events and consistently retrieve them across hundreds of AWS accounts in an organization. For example, it helps you to programmatically receive information about AWS service degradations, resource maintenance events, AWS accounts and resources impacted by service events, and so on. With this feature, you can identify potential impacts from AWS Health events and quickly remediate if a global or account-specific event affects your resources. If you prefer to use an out-of-box solution to visualize organizational health data, consider AWS Personal Health Dashboard Organizational View.

In this post, I will show you how to deploy a customizable dashboard for your AWS Health events across your organization using the AWS Health API, AWS Lambda, and Amazon QuickSight, so you can visualize the AWS event data in a way that fits your operational needs.

Prerequisites

Before you use the AWS Health APIs, you need to:

• Have a business or enterprise support plan from AWS Support.
• Have an organization with all features enabled.
• Sign in to the management account as an AWS Identity and Access Management (IAM) user or assume an IAM role.
• If you sign in as an IAM user, use an IAM policy that grants access to the AWS Health and Organizations actions, such as the AWSHealthFullAccess policy.
• Enable organizational view by calling EnableHealthServiceAccessForOrganization from the management account to allow AWS Health to work with an AWS organization. You can use the AWS Command Line Interface or your own code to call this operation. In this blog post, I use a Lambda function to call this operation.

You can find more details in organizational view prerequisites.

Solution overview

The AWS Health organizational view dashboard provides a customizable solution for your cloud operation team to collect and visualize the status of service health events for their AWS organization. It calls the AWS Health API to consolidate events that are visible to all AWS accounts (such as public events posted to the Service Health Dashboard) or specific accounts in your AWS organization using the impacted services (account-specific events). The well-formatted AWS health organizational event data can be stored in an Amazon Simple Storage Service (Amazon S3) bucket, where you can use QuickSight to visualize the data and create reports based on your business requirements.

Caveats

• AWS Health APIs return account information only when the service event type is account-specific events and if there are AWS accounts impacted by the event.
• AWS Health doesn’t record events that occurred in your organization before you enabled organizational view. For more information, see Aggregating AWS Health events across accounts with organizational view in the AWS Health User Guide.

The following diagram shows the solution architecture.

Figure 1: Solution architecture 

The dashboard user signs in to the AWS master account in your AWS organization as an admin user, and then uses AWS CloudFormation to launch the stack. The QuickSight dashboard must be created manually. After the stack has been deployed, the Lambda function regularly queries the AWS Health API endpoint and retrieves health service status for your AWS organization, including service event information and, for account-specific events, impacted accounts and entities. Service health status data is consolidated into CSV format and stored in a user-specified S3 bucket. You can access QuickSight to create a dashboard to visualize the service health status data and create reports (for example, for event region distribution).

Deploy the stack

1. Sign in to the AWS Management Console using the admin user account, and then open the AWS CloudFormation console.

2. Choose Create stack, and then choose With new resources (standard).

Figure 2: Stacks page in the CloudFormation console

3. Paste the following URL into the Amazon S3 URL field, and then choose Next.

https://my-public-bucket-2021.s3-ap-southeast-2.amazonaws.com/aws-health-cfn-stack.yaml

Figure 3: Create stack page

4. In Specify stack details, enter a name for the stack. Under Parameters, enter the name of your S3 bucket that will be created by the CloudFormation to store the health organization data. Choose Next.

Figure 4: Specify stack details

5. On Configure stack options, use the default settings, and then choose Next.
6. On the Review page, select the three check boxes at the bottom of the page to acknowledge that AWS CloudFormation might create IAM resources and the CAPABILITY_AUTO_EXPAND, and then choose Create stack to launch the stack provisioning process. It takes approximately two minutes for the stack to be provisioned.

Figure 5: Check boxes for IAM resources and the CAPABILITY_AUTO_EXPAND

After the stack has been created, you can use QuickSight to create the dashboard to visualize the health status CSV data.

Note: Due to the default timer setting, it might take up to 10 minutes for the Lambda function to pull health event data for the first time. After the event_data_file.csv file has been stored in the S3 bucket you created, you can use QuickSight to visualize the data.

You may not see any data after the Lambda function runs. This is because AWS Health doesn’t record events in your organization before you enabled the feature. If you don’t see any data in the CSV file output, wait a couple of hours and then check the S3 bucket again.

Sign up for QuickSight and create the new dataset

Amazon QuickSight is a scalable, serverless, business intelligence (BI) service built for the cloud. If you haven’t used QuickSight before, follow the steps in the Amazon QuickSight User Guide to  sign up for a free trial subscription.

1. In the Amazon QuickSight console, under Create your QuickSight account, choose Enterprise, and then choose Continue.

Figure 6: Create your QuickSight account

2. In QuickSight account name, enter health-dashboard.
3. In Notification email address, enter an email address that QuickSight can use to send notifications.
4. Choose Finish to complete the QuickSight subscription.

Figure 7: Subscription details

You are now ready to enable QuickSight to access the data file stored in the S3 bucket.

1. In the upper right corner of the QuickSight console, choose the user icon, and then choose Manage QuickSight.

Figure 8: Manage QuickSight

2. In the left navigation pane, choose Security & permissions, and under QuickSight access to AWS services, choose Add or remove.

Figure 9: Security & permissions

3. Under Amazon S3, choose Select S3 buckets.

Figure 10: Select S3 buckets

4. Select the check box next to the bucket name you used in the CloudFormation stack creation process, and then choose Finish. For example, in Figure 11, this name is s3-aws-health-stack-2021.

Figure 11: Select Amazon S3 buckets

5. Choose Update. QuickSight now has access rights to this bucket to visualize the data.

Figure 12: Update button

QuickSight now has access rights to this bucket, which means it can access the data. At this point, we’ll need to set up the dashboard in QuickSight, which requires setting up a new analysis of the data, then creating the dashboard visuals.

1. From the left navigation pane, choose Analyses, and then choose New analysis.

Figure 13: Analyses page

2. Choose New dataset.

Figure 14: New dataset

3. Open the Amazon S3 console, choose your bucket, and under Object URL, copy the URL to the manifests3.json file that is stored in the bucket.

Figure 15: manifests3.json

4. In the Amazon QuickSight console, choose S3 as the source of the new dataset. Under New S3 data source, enter a data source name (for example, event_data_file). In Upload a manifest file, choose URL, paste the URL you copied, and then choose Connect.

Figure 16: New S3 data source

5. Choose Visualize to proceed to the data visualization in QuickSight.

Figure 17: Visualize S3 data source

Note: You should be able to upload the event data file to QuickSight. If you get a SPICE-related error message, you need to manage SPICE capacity for the region you use to build this QuickSight dashboard. See details in Managing SPICE Capacity from the Amazon QuickSight User Guide.

6. In the left navigation pane, choose Datasets, and then choose the dataset you just created. Choose Schedule refresh, and then choose Create.

Figure 18: Schedule refresh

7. You can configure the time zone, repeat frequency, and starting time based on your needs. In this post, I choose Hourly and then create the schedule.

Figure 19: Hourly refresh frequency

Visualize the health event CSV data through QuickSight

Now that the new dataset and analysis have been created, we’ll discuss how to set up the layout for the analysis in this section. We suggest creating three sheets in the dashboard to show the “Active Service Event”, “Upcoming Service Event” and “Historical Service Event” view accordingly.

Within each dashboard sheet, we will setup visuals to show the information of AWS Health event, such as impacted regions, service types, impacted accounts and so on.

Follow these steps to create the first sheet in the analysis.

1. In Sheet 1 of the newly created analysis, from the Fields list, choose service, and then from Visual types, choose a donut chart. This type returns the Count of Records by Service, as shown in Figure 20.

Figure 20: Donut chart

2. In the left navigation, choose Filter. For statusCode, choose Create one.
3. After the filter is created, choose it, select open, and then choose Apply. This filters out closed and upcoming events, so that only active events are displayed.

Figure 21: Create a filter

4. Choose the ellipses (…) to the right of the visual and then choose Duplicate visual. This makes it possible for you to easily create multiple visuals in the QuickSight analysis.

Figure 22: Duplicate visual

5. Choose the new visual, and from the Fields list, choose region so you can get the Count of Records by Region. You can also change the visual type according to your preference. As Figure 23 shows, I use a pie chart.

Figure 23: Pie chart

6. Repeat these steps to create other visuals in the analysis. You can adjust the size and position of the visual to fit the layout of your analysis. Use the Fields list to create two more visuals, impactedAccount and impactedEntity. You can change the color of the visuals, if you like.

Figure 24: Count of Records by Impactedaccount and Count of Records by Impactedentity

7. Duplicate another visual and from Fields list, choose eventTypeCategory. Under Visual types, choose the vertical bar chart, and then adjust its size to fit the analysis layout.

Figure 25: Count of records by Eventtypecategory

8. Duplicate another visual and from Visual types, choose AutoGraph. Adjust its size to cover the bottom area of the analysis. From the Fields list, choose all the fields to make this visual an event detail spreadsheet.

The sequence in which the field items were selected affects the column order in the spreadsheet. For better presentation, choose the field items in this order: Service, Region, Statuscode, Starttime, Endtime, Lastupdatedtime, Impactedaccount, Impactedentity, Eventtypecode, Eventtypecategory, Arn, and Eventdescription.

Figure 26: Columns organized for better presentation

9. You’ve now completed Sheet 1 of the dashboard. Choose Duplicate sheet to create another sheet. You will create three sheets in all.

Figure 27: Duplicate sheet

10. On the duplicate sheet, choose one of the visuals. In the left navigation, choose Filter, and then choose statusCode. Under Filter type, choose Exclude, and then select the Select all check box. Choose Apply to activate the filter.

Repeat this configuration for all visuals in this sheet. This configuration filters out the open and closed events so that only the upcoming events are displayed.

It’s OK that there’s no data for the upcoming events. Generally, events fall into open and closed categories.

Figure 28: Upcoming service event sheet

11. Choose Duplicate sheet to create the third sheet. Choose one of the visuals. In the left navigation, choose Filter, and then choose statusCode. Under Filter type, choose Include, and then select the closed check box. Choose Apply to activate the filter.

Repeat this configuration for all visuals in this sheet. This configuration filters out the open and upcoming events so that only the closed historical events are displayed.

Figure 29: Historical service event sheet

12. Double-click the tab titles to name them to reflect the health event data categories (for example, Active Service Event, Upcoming Service Event, and Historical Service Event).

Figure 30: Renamed tabs

13. Now you’re ready to publish your analysis. From the upper right, choose Share, and then choose Publish dashboard to create the dashboard from the analysis.

Figure 31: Publish dashboard

14. On Publish a dashboard, enter a name for the dashboard, and then choose Publish dashboard.

Figure 32: Publish a dashboard

You’ve now set up a fully functional AWS Health dashboard for organizational view. Your cloud platform and operations teams can refer to these three tabs in the dashboard for visibility into the active, upcoming, and historical health events. You can bookmark this dashboard in your browser for future access.

Figure 33: aws-health-dashboard-organization-view

For information about customizing the dashboard, see Working with Dashboards in the Amazon QuickSight User Guide.

Clean up resources and cancel your subscription

When you don’t need this dashboard anymore, you can take the following 3 steps to delete the AWS resources you created to avoid ongoing charges to your account.

• In the Amazon S3 console, delete the files in the S3 bucket.
• In the AWS CloudFormation console, choose your CloudFormation stack, and then choose Delete.
• Follow the instructions in Canceling Your Amazon QuickSight Subscription.

Conclusion

In this blog post, I showed you how to use AWS Lambda and Amazon QuickSight to build a customizable dashboard for AWS Health organizational view. This dashboard helps you quickly gain visibility of AWS service events, understand their impact, and achieve faster incident response time.

About the Author

Jerry Chen is a Technical Account Manager at Amazon Web Services (AWS). He works with financial sector customers to design, deploy and operate cloud architectures. A builder at heart, he’s passionate in simplifying the cloud operations for his customers. He currently lives in Australia with his family, and enjoys playing basketball with his kid in the free time.