How to optimize AWS CloudTrail costs by using advanced event selectors

AWS Feed
How to optimize AWS CloudTrail costs by using advanced event selectors

AWS CloudTrail can be used for security, monitoring restricted API calls, notification of threshold breaches, operational issues, filtering mechanisms for isolating data, faster root cause identification, and speedy resolution. CloudTrail can also be used for various compliance and governance controls, by helping you achieve compliance by logging API calls and changes to resources.

Event selectors allow you to add a level of customization to the type of API request you want the trail to capture. Data events provide visibility into the data plane resource operations performed on or within a resource. You can currently log data events on two resource types:

• Amazon S3 object-level API activity (for example, GetObject, DeleteObject, and PutObject API operations).
• AWS Lambda function execution activity (the Invoke API).

With advanced event selectors, you can include or exclude values on fields such as EventSource, EventName, and ResourceARN.

The advantages of using advanced event selectors for AWS CloudTrail include:

• The flexibility to choose only important events means you can control which CloudTrail data events you want to log and pay for. For example, you can choose to log only PutObject or DeleteObject events.
• A trail can have up to five event selectors. You can add advanced event selectors, and conditions for your advanced event selectors, up to a maximum of 500 values for all conditions and selectors on a trail.
• Because advanced event selectors provide more granular control of data event logging, you can choose from the field conditions to better identify issues.
• Similar to regular expressions, advanced event selectors support including or excluding values with pattern matching on partial strings.

In this blog post, we will show how you can use advanced event selectors to create fine-grained selectors for the AWS CloudTrail event record fields. They help you control costs by logging only those events that are important to you.

AWS CloudTrail process flow

Figure 1 shows the trail creation and configuration process. When you create a trail, you need to specify an S3 bucket where the logs will be delivered. You can choose to encrypt your bucket with AWS Key Management Service (AWS KMS). You can also configure Amazon Simple Notification Service (Amazon SNS) to notify you when new log files are delivered. If you want to add another layer of security integrity, enable log file validation, which ensures your log files are not tampered with or modified after their delivery to S3. After the trail is created, you can configure it to deliver CloudTrail logs to Amazon CloudWatch in addition to S3. This option allows you to create CloudWatch monitoring metrics against specific API calls and receive SNS notifications when your specified thresholds are reached. In addition to configuring advanced event selectors, you can add tags just as you would with other AWS resources.

Figure 1: CloudTrail process flow

Example scenario

In our example scenario, instead of logging all S3 data events, we want to log events only when an object is deleted from an S3 bucket. Basic event selectors log all read/write events for all S3 buckets. In this example scenario, we want to log only the DeleteObject API events for a single S3 bucket.

To use advanced event selectors to log DeleteObject API events for an S3 bucket

1. Sign in to the AWS Management Console and open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/.
2. On the CloudTrail service home page, the Trails page, or the Trails section of the Dashboard page, choose Create trail.
3. On the Create Trail page, for Trail name, enter a name for your trail. Leave all other settings at their default.
4. For Storage location, you can choose to create an S3 bucket or use an existing one. For information, see Creating a bucket in the Amazon S3 User Guide. If you choose to use an existing bucket, browse to select it.
5. Under Log file SSE-KMS encryption, clear the Enabled checkbox.

Figure 2: Creating a CloudTrail trail

6. On Choose log events, clear the Management events checkbox and select Data events. By default, basic event selectors log all the read/write events for all the selected S3 buckets.
7. Because we want to capture the DeleteObject events for one S3 bucket, in Data events, choose Switch to advanced event selectors.
   
   Figure 3: Selecting data events and advanced event selectors
8. In Advanced event selectors, build an expression for the specific S3 buckets, S3 objects on AWS Outposts, or Lambda functions on which you want to log data events.Choose from the following fields. For fields that accept an array (more than one value), CloudTrail adds an OR between values.
   • readOnly can be set to Equals a value of true or false. Read-only data events are events that do not change the state of a resource, such as Get* or Describe* events. Write events add, change, or delete resources, attributes, or artifacts, such as Put*, Delete*, or Write* events.
   • eventName can use any operator. You can use it to include or exclude any data event logged to CloudTrail, such as PutBucket or PutItem. You can have multiple values for this field, separated by commas.
   • resources.type is required. resources.type can only use the Equals operator, and the value can be one of the following: AWS::S3::Object, AWS::S3Outposts::Object, or AWS::Lambda::Function.
   • resources.ARN – You can use any operator with resources.ARN, but if you use Equals or NotEquals, the value must exactly match the ARN of a valid resource of the type you’ve specified in the template as the value of resources.type. For example, if resources.type equals AWS::S3::Object, the ARN must be in one of the following formats. The trailing slash is intentional; do not exclude it.
9. To log the events for our scenario, in Data event type, choose S3. For Log selector template, choose Custom. For Selector name, enter a name (for example, AdvanceEventSelector).
10. Under Advanced event selectors, use the following:
    For Field choose eventName. For Operator, choose equals. For Value, enter DeleteObject.
    For Field, choose resources.ARN. For Operator, choose equals. For Value, enter the ARN for your bucket. For information about how to get the ARN, see Amazon S3 resources in the Amazon S3 User Guide.
    For Field, choose readOnly. For Operator, choose equals. For Value, enter false.
    
    Figure 4: Advanced event selectors
11. Choose Next.
12. On the Review and create page, review your selections, and then choose Create trail.

Logging events with the AWS CLI

You can use AWS CLI version 2 to configure trails to log management and data events. To install the AWS CLI, see Install the AWS CLI version 2 on Linux.
The following command returns the default settings for a trail. Use this command to see if the trail is logging management and data events.

aws cloudtrail get-event-selectors --trail-name TrailName

To configure a trail to log management and data events, run the put-event-selectors command. The output shows the advanced event selectors configured on your trail log DeleteObject events on one S3 bucket.

aws cloudtrail put-event-selectors --trail-name TrailName  --advanced-event-selectors  '[ { "Name": "Log DeleteObject events for a S3 bucket", "FieldSelectors": [ { "Field": "readOnly", "Equals": ["false"] }, { "Field": "eventCategory", "Equals": ["Data"] }, { "Field": "resources.type", "Equals": ["AWS::S3::Object"] }, { "Field": "eventName", "Equals": ["DeleteObject"] }, { "Field": "resources.ARN", "Equals": ["arn:aws:s3:::mybucket/"] } ] } ]'

Clean up resources

When you create a trail, logging is turned on automatically. First, turn off logging for your trail.

• Sign in to the AWS Management Console and open the CloudTrail console at https://console.aws.amazon.com/cloudtrail/.
• In the navigation pane, choose Trails, and then choose the name of the trail.
• On the trail details page, choose Stop logging, and then choose Stop logging to confirm.

Now, use the CloudTrail console to delete the trail. Be aware that log files that were already delivered to the S3 bucket will not be deleted.

• On the Trails page, choose the trail name, and then choose Delete.
• When you are prompted to confirm, choose Delete to delete the trail permanently.

Conclusion

In this blog post, we shared the advantages of using advanced event selectors to provide more granular control of data event logging and help you control costs. You can tailor the example used in the post according to your use case. To learn more about advance event selectors, click here.

About the authors

Anjani is a Technical Account Manager at AWS. She works with Enterprise customer and
provides technical guidance to help them innovate and build a secure, scalable cloud on the
AWS platform. Outside of work, she is an Indian classical & salsa dancer, loves to travel and
Volunteers for American Red Cross & Hands on Atlanta.

Snehal Nahar is a Sr.Technical Account Manager with AWS in Charlotte, North Carolina. She is passionate about building innovative solutions using AWS services to help customers achieve their business objectives. She enjoys spending time with family and friends, playing board games and watching TV.