AWS Feed
Creating an Equitable Enterprise AWS Chargeback Model Using CloudHealth
By Sanjna Srivatsa, Data Scientist at VMware
 |
 |
 |
Most enterprises go through the process of monthly chargeback (cost allocation) of their Amazon Web Services (AWS) costs to internal business units or cost centers.
The AWS Cost and Usage Report can provide the flexibility needed to create detailed custom billing rules.
At the Cloud Business Office of VMware, we wanted a solution that allowed us to leverage the power of the AWS Cost and Usage Report to build a customized chargeback solution quickly. CloudHealth’s FlexReports is a simple solution for companies that have multiple Payers or just want to implement custom billing rules quickly.
In this post, I will go through the various steps of how we implemented an equitable chargeback model using FlexReports.
At VMware, what once was an Excel-based slice-and-dice exercise with little control over allocation is now automated and equitable. We have transformed a process that used to take one analyst five days into a compute job that runs in minutes.
Why We Chose Equitability
AWS has rules around how Reserved Instances (RI) discounts are applied in different scenarios. Similarly, Savings Plans have their own set of rules. Both RIs and Savings Plans can float across accounts, favoring some workloads to receive higher discount over others, or some may receive no discount at all.
Our goal at VMware with the equitable chargeback model is to:
- Distribute all currently available discounts proportionally.
- Give product teams the ability to increase their discounts.
Let’s consider a scenario where a Payer has 2,000 linked accounts. The Payer refers to the AWS account (payer account) at the root of the AWS Organization. This account consolidates all billing data from accounts that are part of the AWS Organization (linked accounts).
One of these linked accounts wishes to push their Savings Plan coverage for Amazon Elastic Compute Cloud (Amazon EC2) to 100 percent since they are confident about their usage being somewhat stable over the coming year.
A relevant Savings Plan is bought at the Payer level, but the linked account does not see a 100 percent coverage reflected in its account. The overlying rules that apply to Savings Plans state that they are applied to the highest savings percentage first.
The recently bought Savings Plan may have gravitated to the workloads in the other 1,999 accounts, as they may have qualified as candidates that can receive higher savings percentage.
To nullify the impact of such rules, we embedded an equitability portion into our chargeback process. This redistributes discounts proportionally to each linked account according to our business requirements, ensuring that all linked accounts receive equal benefit.
But what if some accounts want to improve their discounts beyond the equitable discount?
It may be tempting to simply purchase RIs or Savings Plans within the individual linked accounts. However, there are a few reasons why this strategy is not optimal for us at VMware.
First, there’s an ease of management (especially in environments with thousands of accounts) that comes from buying Reserved Instances and Savings Plans centrally at the Payer account level. Take, for example, the need to exchange Convertible RIs (CRI) on a regular basis. Finding and managing CRIs across multiple accounts is very time consuming.
With Savings Plans, there’s an added advantage of buying at the Payer level. Also, buying Savings Plans centrally provides for a wider pool of candidate instances. This gives us the best chance of maximum discount since Savings Plans are applied to the highest savings percentage first.
In an organization with dozens of products, hundreds of cost centers, and thousands of accounts, the only way to optimize RI and Savings Plan coverage (without the risk of over commitment) is to get feedback from the product teams.
For example, if a product team launches a new product, that may increase demand (cloud usage) which will be sustained for a long period of time. In this case, we can plan and purchase appropriately. However, if there’s an ongoing test cycle that spikes usage for a few weeks, we’ll know not to purchase since the usage spike is temporary.
This feedback adds context to usage patterns so they can be related to business intentions. The feedback loop at its simplest level is a communication mechanism between the product teams and the Cloud Business Office.
Visibility to the equitable discount, combined with the feedback loop, gives the product team the ability to improve their discounts over the company average. This has proven to be a great way to incentivize product teams to work with us in planning RI and Savings Plan coverage.
With the implementation of this chargeback model, many business units in VMware have used the feedback loop to increase their coverage from 5 to 75 percent in Amazon DynamoDB and Amazon Elasticsearch Service.
Using CloudHealth FlexReports as the Data Source
The Cloud Business Office at VMware started using CloudHealth in 2017 to help drive greater visibility of public cloud costs. In August 2020, CloudHealth released a feature called FlexReports that allows users to query and build customized reports leveraging data from the AWS Cost and Usage Report.
FlexReports contains resource tags you can use to link company-specific dimensions (for example: environment,developer,owner) to overall costs. FlexReports is available to existing CloudHealth users at no additional cost.
Implementation
Please note that the naming convention used in the following implementation applies the naming convention used in FlexReports, which is similar to the naming format of the AWS Cost and Usage Report.
For example, “LineItem/LineItemType” is a column name LineItemType under the header LineItem in the Cost and Usage Report. In FlexReports, the forward slash (/) in names are replaced by an underscore (_) and would be represent the same column name as “LineItem_LineItemType.”
Step 1: Create Data Pipeline Using FlexReports
- Create a FlexReport containing detailed daily costs per AWS service for the period you want to run the chargeback for; we’ll call it “Usage report.”
Step 2: Create the Chargeback Model
- Step 2.1 – With the AWS Cost and Usage report, we will split usage to reflect each of the pricing models separately: RI, Savings Plan, and On-Demand (includes Spot).
.
Apply the following filters to separate each category to help you dissect the data and work through the logic individually and maintain granular visibility. For the purpose of performing equitable chargeback, the following filters have proved to be sufficient.
.
Condition for the RI bucket:
.LineItem_LineItemType = “DiscountedUsage”
.
Condition for the SP bucket:
.LineItem_LineItemType = “SavingsPlanCoveredUsage”
.
Before we dive into the RI and Savings Plan category, let’s calculate in dollars the savings over On-Demand for each AWS service. We call this Effective Dollar Discount (EDD).
.
For any linked account that purchased RI or Savings Plan, calculate the benefit as if it were in the central pool. In order to identify the account that made the purchase, extract the Payer Account number from thereservation_ReservationArnorsavingsPlan_SavingsPlanArncolumn.
.
Please note that calculating an average percentage across the Payer instead of discount in dollars may not work for most use cases, as the newly derived cost total may show up to be more/less than the actual total.
.
This is because the average of each LineItem carries different weights with regard to impact on discount. Think of Effective Dollar Discount as the normalization factor—that neutralizes any differential advantage given by AWS discounting—and levels the playing field in terms of discounts for all accounts.
. - Step 2.2 – Calculate an EDD for each applicable AWS service: Amazon DynamoDB, Amazon Redshift, Amazon RDS, Amazon ElastiCache, Amazon EC2, and Amazon Elasticsearch Service are applicable for RI discounts. AWS Lambda, AWS Fargate, Amazon ECS, Amazon EKS, and Amazon EC2 are applicable for Savings Plan discounts.
.
Condition for services that are RI applicable:
.EDD(Redshift) = sum(pricing_publicOnDemandCost)-sum(reservation_EffectiveCost)
.
Condition for services that are SP applicable:
.EDD(Lambda) = sum(pricing_publicOnDemandCost)-sum(savingsPlan_SavingsPlanEffectiveCost)
. - Step 2.3 – Now that we have the Effective Dollar Discount per AWS service, we’ll distribute it proportionally across accounts based on their usage for the month. This means the higher the usage in the account, the greater the discount.
.
To enable this proportional distribution, we calculate a ratio called “Cost_ratio.” The column ‘pricing_publicOnDemandCost’ used for the calculation of the Cost_ratio represents the on demand or cost before any discount of each LineItem.
.
This ratio can be used to redistribute everything from amortized cost and discounts to indirect costs such as support charges. This step is to be applied to each and every LineItem to serve as a measure of its contribution towards total cost.
.Cost_ratio = pricing_publicOnDemandCost/sum(pricing_publicOnDemandCost)
. - Step 2.4 – Let us now calculate the new cost (equitable cost) for each LineItem that has RI or Savings Plan applied to it. The equation below must be applied to LineItems only within each applicable AWS service, since they will each have their own Effective Dollar Discount.
.
We’ll only apply the respective EDD to teams using that AWS service for that month. Discount in dollars from DynamoDB cannot be shared among accounts that are using only EC2. EDD will be distributed to accounts that have used that service for that month of chargeback.
.
For each LineItem:
.Equitable_usage_cost = pricing_publicOnDemandCost - (cost_ratio*EDD)
.
For each UsageAccountID:
.Equitable_usage_cost(Account X) = sum(Equitable_usage_cost)
. - Step 2.5 – Finally, let’s calculate costs that have either been On-Demand or undiscounted (do not have RI or Savings Plan applied to it). This will not be altered by the equitable model in any way.
.
Condition for the On-Demand bucket:
.LineItem_LineItemType = “Usage”
.
For each UsageAccountID, calculate:
.OnDemandUsage = sum(LineItem_UnblendedCost)
.
For each UsageAccountID:
.MonthlyChargebackTotal(Account X) = Equitable_usage_cost + OnDemandUsage
.
This MonthlyChargebackTotal will reflect cost to be charged back to the account for that month. Depending on your accounting rules, you can redistribute any other centrally applied cost using the Cost_ratio.
.
Once all of the above steps are completed, the resulting dataframe/dataset can be exported as a CSV with any expositions of cost centers, business units, or any other company-specific attributes using the Account ID as the primary key.
Results
The output of the process mentioned in this post generates a CSV that has a date, cost center, business unit, and the dollar amount to be charged back. Our finance team at VMware simply imports this file into the general ledger system, and we’re done!
The size of the data source file from the FlexReport will differ based on the number of accounts in your organization and the activity in that month. For us, the equitable chargeback process takes millions of lines of raw data from the FlexReport to a few hundred lines of actionable output within 15 minutes.
Conclusion
Using CloudHealth’s FlexReports helped us at VMware quickly build a customized chargeback solution without the need to develop and maintain an AWS Cost and Usage Report ingestion system.
The visibility gained to the equitable level of Reserved Instance and Savings Plan coverage per product team has incentivized them to improve their coverage per AWS service.
We now have the best of both worlds: centralized management of RI and Savings Plans with individual ability to influence coverage and improve discount for their teams. You can access FlexReports under the Reports tab in CloudHealth.
The content and opinions in this blog are those of the third-party author and AWS is not responsible for the content or accuracy of this post.
.
.
CloudHealth by VMware – AWS Partner Spotlight
CloudHealth by VMware is an AWS Partner and trusted software platform for accelerating global business transformation in the cloud.
Contact CloudHealth | Partner Overview | AWS Marketplace
*Already worked with CloudHealth? Rate the Partner
*To review an AWS Partner, you must be a customer that has worked with them directly on a project.