VPC Flow Log automation using AWS Control Tower LifeCycle

Amazon Web Services Feed
VPC Flow Log automation using AWS Control Tower LifeCycle

Many customers that I work with use Amazon Virtual Private Cloud (Amazon VPC) Flow Logs for monitoring, troubleshooting, anomaly detection, or archival purposes. One common ask from my customers is how to automate and centralize Amazon VPC Flow Logs in a multi-account environment. AWS Control Tower provides multi-account governance with built-in centralized logging of AWS CloudTrail and AWS Config. In this blog post, I show you how to automate and centralized logging of Amazon VPC Flow Logs across your AWS Control Tower multi-account environment. Using this solution, you can manage VPC Flow Logs across multiple accounts with self-service automation and periodic consistency check.

Solution overview

AWS Control Tower sets up centralized log storage in the Log Archive account. In this solution, you can use the Log Archive account or another account such as shared network account as the log destination. For consistency, I call it the Hub account. A dedicated Amazon S3 bucket is created in the Hub account to store the logs.

Each account owner can have different VPC Flow Logs filters per VPC, for example: log all traffic in a production VPC and log rejected traffic in a development VPC. The solution uses predefined AWS tags for each VPC Flow Logs filter. When the predefined AWS tag is added to the VPCs or subnets, the action generates CloudWatch Events with the tag information included. This solution uses Amazon EventBridge to send the events from the source account into the Hub account. When the event arrives at Event Bus in the Hub account, an AWS Lambda function is triggered. This function uses the tag input from the event and modifies the VPC Flow Logs filter in the source account accordingly.

To ensure all new accounts vended from AWS Control Tower Account Factory benefit from this automation, the solution uses AWS Control Tower lifecycle events to deploy the required infrastructure after the new account is fully provisioned. All the required resources in this solution are deployed using AWS CloudFormation stack and AWS CloudFormation StackSets from the AWS Control Tower master account.

The following diagram illustrates the VPC Flow Logs automation process:

1. New account creation triggers AWS Control Tower lifecycle event, which invoke the Lifecycle Lambda function.
2. The new account id is verified and added to the existing CloudFormation StackSet instances.
3. CloudFormation StackSet deploys the EventBridge rules, Event Bus and FlowLogActivator Lambda function.
4. Create or update VPC with the predefined AWS tag generates the CloudWatch events with tag information included.
5. EventBridge rule in the new account forwards the event to Event Bus in the hub account.
6. EventBridge rule in Hub account triggers the FlowLogActivator Lambda function.
7. The Lambda function assumes a role in the new account and configures the VPC Flow Logs.
8. VPC Flow Logs stores the log to predefined Amazon S3 bucket.

Prerequisites

Here are the prerequisites before you deploy the solution.

1. I assume that you already have a working version of AWS Control Tower. If you don’t, go ahead and follow Jeff Barr’s walkthrough blog post to get your first AWS Control Tower setup.
2. Master account id and Hub account id. Hub account can be the existing Log Archive account or other account that is managed by AWS Control Tower. Account id can be found on AWS Control Tower console.
3. AWS Organization id (format: o-text), which can be found on the settings tab of the AWS Organization console
4. AWS account ids that you want to be included in the deployment. Note that master account should be excluded.

Launching VPC Flow Logs automation

First step, you build the infrastructure for this automation in the existing accounts. To deploy this across multiple accounts and Regions, you will use AWS CloudFormation StackSets.

1. Log in to AWS Control Tower master account and select the AWS region where AWS Control Tower is deployed.
2. In the AWS CloudFormation StackSets console, enter the Amazon S3 URL: https://s3.amazonaws.com/marketplace-sa-resources-ct-us-east-1/ct_vpc_flowlog_stackset.yml.
3. Enter the StackSet name, for example ‘CT-FlowLog’
4. Under Parameters section enter the following parameters:
   • ComplianceFrequency = frequency to check the flow logs compliance, must be value between 2 and 168 hours.
   • EventBusDestinationAccount= Hub account id from the prerequisite earlier.
   • EventBusName = the default value or enter the event bus name.
   • FlowLogBucketName = Select unique name for the S3 bucket that stores the log, make sure that there is no other S3 bucket with similar name. Tips: incorporate the account id in the bucket name, that is, accountid.flowlog
   • FlowLogDestinationAccount = Hub account id from the prerequisite earlier.
   • MasterAccount = Master account id from the prerequisite earlier.
   • MasterRegion= AWS Regions code where you deployed the AWS Control Tower
   • OrgId = AWS Organization id value from the prerequisite earlier.
   • StackSetName = enter the StackSet name as per previous step, that is, ‘CT-FlowLog’
5. On the Permissions section, choose Self-service permissions and enter the following settings:
   • IAM role name = select AWSControlTowerAdmin
   • IAM execution role name = enter AWSControlTowerExecution
6. On the Accounts section:
   • Choose Deploy stacks in accounts
   • Provide comma delimited list of account ids that you want to include.
   • Important: make sure to include the Hub account id as the target.
7. On the Specify Regions section, select AWS regions where you want to deploy the automation.
8. On the Review page, validate all parameters and settings. Don’t forget to select the check box I acknowledge that AWS CloudFormation might create IAM resource with custom names. When you ready, select Submit.
9. Navigate to the StackSets info tab and take note of the StackSet ARN. You will need this on the next step.

Setting up the lifecycle event

On the previous step, you deployed automation to cover the existing accounts. Now, you will set up the automation for the future accounts by using the lifecycle event from AWS Control Tower.

1. Log in to AWS Control Tower master account and select the AWS region where AWS Control Tower is deployed.
2. Use this shortcut URL to launch the stack and select Next.
3. On the Specify stack details page, give your stack a name such as “CT-FlowLog-LifeCycle.”
4. Under Parameters section enter the following parameters:
   • EventBusDestinationAccount = Hub account id from the prerequisite earlier.
   • OrgId = AWS Organization id value from the prerequisite earlier.
   • StackSetArn = ARN of the stack sets from Step 9 of the previous section.
5. On the Configure stack options page, you can choose to add tags and other options. Choose Next to proceed.
6. On the Review page, validate your parameters and finally select Create Stack.

Testing the solution

With both VPC Flow Logs and AWS Control Tower lifecycle set up completed, you are now ready to test it. Let’s choose one target account that you included in the stack sets earlier.

1. Log in to any of AWS Control Tower member accounts that is included in the stack sets scope earlier. Ensure that the console session is in the AWS region that included in the scope
2. Navigate to the VPCconsole. On the Your VPC’s section, select the VPC that you want to test.
3. Go to the Tags tab and select Add/Edit Tags. Select Create Tag and enter the following tag key-value combination:
   • Key = flowlog
   • Value = ACCEPT
4. Select Save to confirm. Wait for a few seconds and go to the Flow Logs tab. You should find the new flow logs is set up. You can experiment by modifying the tags with different combination, such as ALL or REJECT.
5. Now let’s try to add another flow log at subnet level. On the Subnets section, select the subnet that you want to test.
6. Go to the Tags tab and select Add/Edit Tags. Select Create Tag and enter the following tag key-value combination:
   • Key = flow-log
   • Value = REJECT
7. Select Save to confirm. Wait for a few seconds and go to the Flow Logs tab. You should find the second flow logs now.

Note: Remember, the Amazon S3 bucket is owned by the Hub account. If you want to access the raw logs, you must log in first to the Hub account.

This concludes the test, if you deployed this automation in several AWS regions, go ahead and test it for each region by following the preceding steps. If you plan to create a new account under AWS Control Tower, refer to the quick account provisioning step. Then run the test by following the steps preceding after the account is fully provisioned.

Optional Customization

Amazon VPC Flow Log supports three types of filter: All, Accepted, and Rejected. The Lambda function in the Hub account is programmed to accepts combinations of tag key and value pair such as shown following.

As optional steps, you can modify the existing template to match the tag key-value combination as per your organization requirements.

• Open the template file ct_vpc_flowlog_stackset.yml using text editor.
• Locate the AWS CloudFormation mappings for LambdaVariable > Tag
• To modify the tag key, add or remove the value for KeyList and Key to match your requirement.
• To add or remove the variety of key value, modify the mapping for all, accept, and reject.
• Save the template file.
• Update the existing stack sets using the new template file, enter the same parameter as before.

If you like to keep all network orchestration in a dedicated Shared Network account and log storage in the Log Archive account, you can modify the stack sets parameter as follows:

• FlowLogDestinationAccount = enter your Log Archive account id
• EventBusDestinationAccount = enter your Shared Network account id

Cleaning up

To avoid incurring future charges, you can remove the solution resources by deleting the stack and stack sets from the AWS CloudFormation console.

• To remove the Lifecycle set up, follow guide to delete the stack you created on step 2.
• To remove the VPC Flow Logs stack set set up, you must delete the stack instances first and then delete the stack sets
• Removing the stack and stack sets do not remove the existing flow logs that you have configured earlier. Follow the instruction to delete flow log if you don’t want to keep it.
• Finally, the S3 bucket in the Hub account is retained by default, don’t forget to delete this bucket if you no longer need it.

Conclusion

In this blog post, I demonstrated how to automate VPC Flow Logs setup in multi-account environment by using EventBridge and AWS Lambda for orchestration. I also show how to extend the AWS Control Tower Lifecycle to automate the setup during account provisioning. Finally, I provided an example on how to customize the tags based on your organization standard.

To expand on this solution, consider incorporating the example template with other customization, and package it as one solution. For more information, see customizations for AWS Control Tower.

About the authors

Welly Siauw is a Senior Technical Account Manager at AWS. Welly enjoys working with AWS customers in solving architectural, operational, and cost optimization challenges