Sending CloudFront standard logs to CloudWatch Logs for analysis

AWS Feed
Sending CloudFront standard logs to CloudWatch Logs for analysis

Amazon CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.

CloudFront standard logs (also known as access logs) give you visibility into requests that are made to a CloudFront distribution. The logs can be analyzed for a variety of use cases, such as determining which objects are the most requested or which edge locations receive the most traffic. You can also use logging to troubleshoot errors or gain performance insights.

You can gain these insights using these Amazon CloudWatch Logs features:

• CloudWatch Logs Insights, which enables you to interactively search and analyze your log data.
• Metric filters, which allow you to extract metric data from log events.
• Amazon CloudWatch Contributor Insights, which show you metrics about the top-N contributors, the total number of unique contributors, along with their usage.

In this blog post, I’ll show how you can send CloudFront access logs to Amazon CloudWatch Logs. I’ll also discuss tools that you can use with CloudWatch Logs to generate meaningful insights and create dashboards from your CloudFront logs.

Overview

Many AWS services write logs to CloudWatch Logs natively, while others write to Amazon Simple Storage Service (Amazon S3) or to both CloudWatch Logs and Amazon S3. For a list of services, see AWS Services That Publish Logs to CloudWatch Logs in the Amazon CloudWatch Logs User Guide.

In the case of a service that only publishes its logs to S3, a common practice is to use an AWS Lambda function triggered by an Amazon S3 event notification to write the logs into your CloudWatch log groups. In this post, I’ll walk you through this process and some sample metric filters, Contributor Insights rules, and CloudWatch Logs Insights queries. I’ll combine this data with CloudWatch metrics emitted from CloudFront to put together an operational CloudWatch dashboard for a given CloudFront distribution.

Note: In cases where there is a high frequency of write events to S3, you might want to send the S3 event notifications to an Amazon Simple Queue Service queue and then have the Lambda function poll the queue. The Amazon SQS approach allows a single function invocation to retrieve many objects from S3.

Figure 1: Architecture for sending CloudFront logs to CloudWatch Logs

To use the CloudFormation template in this post, you need the following:

• A CloudFront distribution with standard logging enabled. For more information, see Configuring and using standard logs (access logs) in the Amazon CloudFront Developer Guide.
• Access to the S3 bucket where the CloudFront logs are being delivered.

Approach

You will use a CloudFormation template to deploy the following resources:

• An S3 event notification to trigger for new-object-created events for your CloudFront logs.
• A CloudWatch log group to store your CloudFront logs.
• A Lambda-backed custom resource to configure the S3 event notification.
• A Lambda function to write CloudFront logs in S3 to a CloudWatch log group.
• Several metric filters for your log group.
• Several Contributor Insights rules for your log group.
• A CloudWatch Logs Insights query for the log group.
• A CloudWatch dashboard that includes service metrics, metric filters, Contributor Insights rule reports, and CloudWatch Logs Insights query results.

The CloudFormation template takes the following parameters:

• S3 logging bucket name

This is the name of the S3 bucket where your CloudFront logs are being delivered. The S3 bucket should be in the same AWS Region where the template is being deployed. The Lambda function deployment package is hosted on an S3 bucket in us-east-1. If your logging bucket is in a different Region, you will need to host the deployment package for the function in a bucket in that Region and edit the CloudFormation template to reference that location.

• S3 prefix for CloudFront logs

This is the prefix in the S3 bucket where CloudFront logs are being written. This parameter is used for the S3 event notification filter.

• CloudFront distribution ID

This is the CloudFront distribution ID. This parameter is used to add CloudFront metrics to your dashboard and determine the name of the log group and metric namespace for your metric filters.

• Contributor Insights rule state

This parameter is configured to enable or disable the Contributor Insights rules on creation.

Figure 2: Configured CloudFormation stack parameters

If you would like to use a CloudFormation template to create these resources, launch the following stack.

To create the CloudFormation stack

1. Open the CloudFormation console at https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/.
2. In the navigation pane, choose Stacks.
3. Under Create stack, choose With new resources (standard).
4. For Specify template, use the following S3 URL: https://mng-blog-solutions.s3.amazonaws.com/cloudfront-to-cloudwatch-blog/CFtoCWLogs.yaml
5. Enter the parameters for your CloudFront distribution ID, the S3 bucket where your CloudFront logs are being written, and the S3 prefix for your logs, and then choose Next.
6. Leave all other fields at their defaults, and then choose Create stack.

Analyze CloudFront logs in CloudWatch

After your CloudFront logs are in CloudWatch, you can use Contributor Insights, metric filters, and CloudWatch Logs Insights queries to analyze them. When combined with the CloudWatch metrics emitted from CloudFront, you can create valuable dashboards for your CloudFront distribution.

Contributor Insights rules for your log group

Using Contributor Insights with CloudFront logs allows you to view requests made to your distribution from several dimensions. Here are a few examples of these rules, along with a link to the JSON-formatted rule definitions in GitHub.

Metric filters for the log group

You can use CloudWatch metric filters to extract meaningful metrics from your CloudFront logs. In some cases, the number of possible results for a log field is known, such as HTTP method or HTTP response code. In these cases, you can create metric filter expressions to collect these data points. For more information, see Filter and Pattern Syntax in the Amazon CloudWatch Logs User Guide.

Metric filter format for CloudFront logs:

You can use the following format for your CloudFront logs. This filter pattern will identify all of the log fields available. You can then use a numeric or equality operator to match a given log field.

 [date, time, x_edge_location, sc_bytes, c_ip, cs_method, Host, cs_uri_stem, sc_status, cs_referer, cs_User_Agent, us_uri_query, Cookie, x_edge_result_type=CapacityExceeded, x_edge_request_id, x_host_header, cs_protocol, cs_bytes, time_taken, x_forwarded_for, ssl_protocol, ssl_cipher, x_edge_response_result_type, cs_protocol_version, fle_status, fle_encrypted_fields, c_port, time_to_first_byte, x_edge_detailed_result_type, sc_content_type, sc_content_len, sc_range_start, sc_range_end ]

Metric filters to collect HTTP protocol:

When a log event matches a filter expression, you can decide to take a value from the log field and publish it as the metric value. In these examples, I use a combination of either bytes out or time taken. You can use the sample count statistic for the metric to view how many times an event matched the pattern. You can use the sum and average statistics to view the actual bytes downloaded or time taken.  For example, when you look at the two metric filters for HTTP and HTTPs requests, I use the bytes out field. When you use the sum statistic to view the HTTP and HTTPs metrics, you see the total number of bytes sent out. However, if you just want to compare the total number of HTTP and HTTPs requests, you can use the sample count statistic.

• HTTP Requests
• HTTPS-Requests

Latency metric filters:

In these metric filters, you are matching every log event in the log group. Each filter produces a different metric from a different field in the log event. In this case, I am tracking the values for the time to first byte and the time taken fields, respectively.

• Origin Latency (time to first byte)
• Latency-Filter

Metric filters to collect edge result type:

The following metric filters provide you with a metric of every potential value of the x-edge-result-type log field. You can gather this same data point from a Contributor Insights rule.

• Hit-Requests
• RefreshHit-Requests
• OriginShieldHit-Requests
• Redirect-Requests
• Miss-Requests
• Error-Requests
• LimitExceeded-Requests
• CapacityExceeded-Requests

CloudWatch Logs Insights query

You can use the following CloudWatch Logs Insights query template to parse out all the available log fields in your CloudFront logs. Then, you can easily use filter and group expressions by referring to the field name directly.

The following query is included as an example in the CloudWatch dashboard that is created with the CloudFormation template attached to this post. You can save this query in CloudWatch Logs Insights for use as a template to build more advanced queries. For more information, see CloudWatch Logs Insights Query Syntax in the Amazon CloudWatch Logs User Guide.

parse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *" as date, time, x_edge_location, sc_bytes, c_ip, cs_method, host, cs_uri_stem, sc_status, referer, useragent, cs_uri_query, cookie, x_edge_result_type, x_edge_request_id, x_host_header, cs_protocol, cs_bytes, time_taken, x_forwarded_for, ssl_protocol, ssl_cipher, x_edge_response_result_type, cs_protocol_version, fle_status, fle_encrypted_fields, c_port, time_to_first_byte, x_edge_detailed_result_type, sc_content_type, sc_content_len, sc_range_start, sc_range_end
| limit 20
| sort @timestamp desc 

Find log events that contain an error:

Here is an example query that will only show you requests where the x-edge-result-type field was in error.

parse @message "* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *" as date, time, x_edge_location, sc_bytes, c_ip, cs_method, host, cs_uri_stem, sc_status, referer, useragent, cs_uri_query, cookie, x_edge_result_type, x_edge_request_id, x_host_header, cs_protocol, cs_bytes, time_taken, x_forwarded_for, ssl_protocol, ssl_cipher, x_edge_response_result_type, cs_protocol_version, fle_status, fle_encrypted_fields, c_port, time_to_first_byte, x_edge_detailed_result_type, sc_content_type, sc_content_len, sc_range_start, sc_range_end
| limit 20
| sort @timestamp desc | filter x_edge_result_type like "Error"

Here is an example query that will produce a time series showing the minimum, average, and maximum values of the time_to_first_byte field over a one-minute period.

Parse @message “* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *” as date, time, x_edge_location, sc_bytes, c_ip, cs_method, host, cs_uri_stem, sc_status, eferrer, useragent, cs_uri_query, cookie, x_edge_result_type, x_edge_request_id, x_host_header, cs_protocol, cs_bytes, time_taken, x_forwarded_for, ssl_protocol, ssl_cipher, x_edge_response_result_type, cs_protocol_version, fle_status, fle_encrypted_fields, c_port, time_to_first_byte, x_edge_detailed_result_type, sc_content_type, sc_content_len, sc_range_start, sc_range_end
| stats avg(time_to_first_byte), min(time_to_first_byte), max(time_to_first_byte) by bin(60s)

Using the default metrics in conjunction with the metrics emitted from your metric filters and the report data from your Contributor Insights rules, you can create an operational dashboard for your CloudFront distribution.

Figure 3: CloudFront Insights dashboard

Cost considerations

When evaluating costs, consider the volume of logs ingested into CloudWatch. A higher volume of logs impacts the cost for log ingestion, the number of matched log events in Contributor Insights, and the total amount of data queried in CloudWatch Logs Insights. For more information, see the CloudWatch pricing page.

Cleanup

To clean up this dashboard and associated rules, just delete the CloudFormation stack. For instructions, see Deleting a stack on the AWS CloudFormation console in the AWS CloudFormation User Guide.

Conclusion

In this post, I walked you through the sending of CloudFront standard logs to CloudWatch Logs through an S3 event notification and a Lambda function. I demonstrated some of the ways in which you can use Contributor Insights, CloudWatch Logs Insights, and metric filters to create powerful dashboards for your CloudFront distributions—without needing to use a third-party partner product. You can extract many other data points from CloudFront logs using CloudWatch. The examples in this post are designed to help you get started. Try building out your own CloudWatch Logs Insights queries, Contributor Insights rules, metric filters, and dashboards to add value to your observability strategy for CloudFront.

About the author