AWS Feed
Building a Third-Party SaaS Metering and Billing Integration on AWS
By Keith P, Partner Solutions Architect at AWS
 |
As part of moving to a software-as-a-service (SaaS) model, companies need more flexible billing constructs that allow them to support a broader range of billing strategies and models.
The move to SaaS often means moving to subscription and/or consumption-based billing. More and more, SaaS customers expect to pay for what they consume rather than large up-front license costs.
In order to support this approach, companies need to consider adopting new tooling or services that can support these SaaS pricing models.
In many cases, SaaS providers will rely on third-party billing solutions to handle the intricacies of configuring pricing options and generating bills. These solutions provide the price modeling, subscription management, invoicing, customer management, and other billing-related functionality that’s essential to operating a SaaS business.
This also allows SaaS providers to focus on the IP of their solution while taking advantage of the ongoing evolution and innovation that’s continually introduced by third-party billing services.
In this post, I will focus on how SaaS providers can create a billing integration experience that captures metering data and publishes it to a third-party billing system.
I’ll also review common SaaS billing models and introduce a sample billing implementation that provides a working example of how you might approach building a more universal strategy for integrating with SaaS billing providers.
SaaS Billing Fundamentals
There are several different types of billing models commonly used by SaaS companies. To support these varying needs, billing systems typically provide a collection of pricing constructs that can be used to define and build a pricing strategy that addresses the complex array of pricing options SaaS providers require.
Some SaaS providers may rely purely on a monthly subscription, while others may build their pricing around a consumption model. The number of users, performance throughput, and access to different application features may also be part of the billing strategy. These can be combined in different patterns.
It’s the job of each SaaS provider to identify the appropriate billable unit that best fits with their market and customer needs. Once you have that unit figured out, you can begin to think about how these billing details will be submitted to your billing provider.
Conceptual Overview
Before we get into the details of our sample billing architecture, let’s look at the conceptual view of the core concepts that are part of the sample billing solution that we’ve created. The following diagram outlines the flow of this solution.
Figure 1 – A conceptual metering implementation.
On the left, you can see that our SaaS application will produce two types of events. The first is an onboarding request (1), which is sent any time a new tenant is added to the system. This event is added to the database, creating a new tenant configuration in the reference billing implementation (2).
The second event type is the billing event (3). Every time the SaaS application needs to report any tenant activity that correlates to a billable event, it publishes an event that gets stored in an intermediate database (4).
Periodically, the individual events in the data store are aggregated (5), stored in the database (6), and then published to the billing provider (7, 8). This aggregation happens to reduce the storage footprint of the individual events.
Why publish the events to an intermediate database rather than directly to the billing provider? This intermediate storage adds to the overall durability and resiliency of your billing integration. By holding the data locally, you’re able to overcome any situation where events can’t be published to the billing provider. Retaining these events allows them to be republished at a later time. Using an intermediate data store also helps to avoid hitting API limits that the billing provider imposes.
In this billing reference implementation, the Billing Provider role is fulfilled by Stripe Billing. However, there are many different partner solutions available to manage invoicing, subscriptions, and other essential billing functions.
Because of the modular nature of the billing reference solution, the billing provider of your choice can be easily substituted into the solution.
Prerequisites
Signing up for Stripe
Before a tenant can be onboarded to Stripe Billing, you have to create a Stripe account.
When you sign up for a Stripe account, you’ll start out in a test environment. This allows you to use Stripe functionality without signing up for a merchant account.
Creating Stripe Products and Prices
After signing up for a Stripe account, you need to create the billing constructs within Stripe that will be used to determine the overall pricing and tiering model of your application. These options must be configured before onboarding any new tenants to the reference billing implementation.
To configure this data, you’ll need to configure Stripe products and prices. The configuration of these items will define how your system bills tenants, as well as how you may tier your billing model. If you have multiple tiers within your SaaS application, you’ll need to create separate products for each tier.
For example, suppose your SaaS application has two tiers: Developer and Enterprise. You would create two Stripe products—one called Developer and one called Enterprise.
During the Stripe product creation process, you specify the pricing model, price, and billing period. For the purpose of this example, the price you create must have the box checked for “Usage is metered.” The billing reference implementation uses the “Sum of usage values during period” under the “Charge for metered usage by” drop-down.
Once the products and prices have been configure in Stripe, you’re ready to begin onboarding tenants.
You’ll likely need to dive deeper on the Stripe billing configuration options to figure what fits best for your environment. More details can be found in the Stripe documentation.
Creating Restricted Keys
This section describes creating restricted keys instead of using the root API key provided by Stripe. Restricted keys allow developers to access a subset of Stripe functionality. More information about restricted keys can be found in the Stripe documentation.
For this reference implementation, create a restricted key with the following permissions:
- Invoices, Read
- Subscriptions, Read
- Usage Records, Write
The key created should begin with “rk_”. This key should be put into the Secret Manager Secret created as part of the application deployment, described below.
Implementation Overview
Now, let’s shift our attention to the architecture of the billing solution we have created. The basic approach we have taken is to create the infrastructure constructs that allows a SaaS developer to easily publish metered billing events.
This includes providing all of the infrastructure needed to ingest and aggregate events before publishing them to a billing system (in this case, Stripe). This experience also supports the creation of new tenants that will be integrated into your SaaS onboarding experience.
Here’s a diagram of the reference implementation that conforms to the conceptual model outlined above.
Figure 2 – Billing and metering implementation using Stripe.
First, you’ll notice this architecture is entirely serverless. This means there are no individual servers to maintain.
Further, compute billing is based on actual consumption. If no events are running through the system or aggregation is not taking place, the person operating this application only pays for the data storage in Amazon DynamoDB and the secret in AWS Secrets Manager.
The following sections will break down each part of the diagram.
Tenant Onboarding
After creating an account within Stripe, and defining one or more products and prices, you’re ready to onboard a tenant. There are two parts to onboarding a tenant to this system.
First, you need to create a customer in Stripe. This customer represents a tenant, not an individual user. For example, if Company XYZ wants to use your SaaS application, a customer will be created that represents Company XYZ. The customer contains several pieces of information about a tenant, including payment information.
Once a customer has been created, you need to create a Stripe subscription. This links the customer with a product at a specific price, and surfaces as the plan a tenant selects during the onboarding process.
This link between a subscription and customer results in a Subscription Item ID. This identifier is crucial to the billing integration experience and is referenced by each of the billing events that are sent to Stipe.
As each tenant onboards to your SaaS application, you will create a new Stripe Customer and trigger an onboarding event that holds the information needed to create a new tenant for your reference billing architecture.
The sample billing solution uses Amazon EventBridge (Billing EventBridge in the diagram above) to publish a tenant onboarding event.
An example of this API call written in Python would appear as follows:
At the entry to this code, you’ll see we have initialized two values. The EVENT_TYPE is used to identify the type of message being sent; it’s set to ONBOARD. This is used during the processing to distinguish an onboarding event from a billing event.
Next, you see the detail structure used to hold the data for our new tenant. Here, the TenantID represents an ID associated with a tenant. This is going to be assigned by your SaaS application and has no relationship with the Stripe Customer ID. It could be a GUID or some other unique identifier for a tenant.
This identifier should be available to your application when processing requests that involve billing; it’s required by the reference billing architecture when processing billing events.
The ExternalSubscriptionIdentifier field’s value will be the identifier used in requests to the billing provider. In the example above, it’s a Subscription ID used by Stripe Billing.
This value will depend on the billing provider used. There will be an example of how the ExternalSubscriptionIdentifier is used under the Aggregating Events and Publishing to Stripe Billing section.
Once the event is placed onto the Billing EventBridge, this triggers the AWS Lambda function labeled “Onboard New Tenant Function” in the diagram above. This Lambda function processes the event and stores the results in the Billing and Metering DynamoDB Table.
Processing Billing Events
After a tenant is onboarded, you can start publishing billing events. These events provide the data that’s used to capture and convey tenant activity that is connected with the overall billing model configured in Stripe.
If you have a consumption-based billing model, for example, these events would be used to describe the billable unit. These consumption events will ultimately be aggregated and used to calculate a tenant’s bill within Stripe.
These billing events look very much like our onboarding event (above). The events are placed on the same Amazon EventBridge, Billing EventBridge. The following is a Python example of a sample billing event being published through the EventBridge API:
Here, we use the EVENT_TYPE to convey this is a BILLING event. Within the detail, you’ll see the TenantID represents the tenant that’s sending the billable event. This value should be in the tenant context passed to the application, typically the JWT the application consumes.
The quantity represents the number of billable units the customer consumed. This information is generated when a billable event occurs.
Once this event is placed onto the Billing EventBridge, the Process Billing Event Function runs. This function processes the event and places it into the Billing and Metering DynamoDB table.
Storing Metric Events and User Configuration
All of the data ingested by the sample billing solution is stored in a single DynamoDB table. This will actually hold multiple types of data in a single table.
To support this single table model, the DynamoDB table uses the TenantID as its partition key. It also uses various keys and key prefixes for the sort key to identify the data type stored in a given DynamoDB item.
There are three different types of items that will be represented in this table:
- Tenant configurations (the sort key will be CONFIG)
- Individual events (the sort key will be EVENT#<epoch time in milliseconds>#<unique identifier>)
- Aggregated events (the sort key will be AGGREGATE#<aggregation time range (MINUTE)>#<beginning of epoch time period>)
The tenant configuration items store the external subscription identifier associated with the billing provider. This information is used in the last step (in the diagram above, in the Billing Publisher function). The tenant configuration also contains the closing time for the upcoming invoice in Stripe.
The individual event items include the TenantID, the time the event occurred, and the quantity of billable units associated with the event. The sort key for individual events also includes a unique identifier to prevent event collisions if two or more events are submitted concurrently.
Finally, the aggregated events are items that contain the TenantID, a time span in minutes, and a total of the quantity of events that occurred within that minute.
Aggregating Events and Publishing to Stripe Billing
After the data arrives in the DynamoDB table, a CloudWatch Scheduled Event (Aggregate Billing Entries Time-Based Event in the diagram) invokes an AWS Step Function composed of two Lambdas: one that aggregates individual billing events (Billing Event Aggregation function), and another that publishes the events to Stripe Billing (Stripe Billing Publish function). This happens, by default, once per minute.
The once-per-minute publishing happens because of a Stripe Billing requirement that events for a previous billing period be published within five minutes of the close of the billing period. In order to avoid missing events at the end of a billing period, the aggregation happens once per minute, though the closing invoice time in the tenant configuration is checked before publishing to Stripe. If the invoice is not yet closed, billing data is not published.
When individual billing events are aggregated by the Billing Event Aggregation function, these events are removed from the table and replaced with an aggregated event as a summary of the incoming events.
In order to ensure aggregated events are not published twice, the Event Aggregation function generates an idempotency key for use with Stripe Billing. If two aggregated events are passed into Stripe Billing with the same idempotency key, then the second attempt will fail.
These aggregated events are stored indefinitely in the DynamoDB table (though they can be removed once they’re published to Stripe). Aggregate events have an attribute named “published_to_billing_provider.” When this value is set to true in the DynamoDB table, the event has been successfully submitted to Stripe Billing. When to remove confirmed events is up to the individual running the software.
Why aggregate events at all? First, this reduces the number of items in the Billing and Metering DynamoDB table. This reduces the cost of running the billing reference implementation. Also, by aggregating and storing in the table, fewer requests are made to Stripe Billing.
After aggregating individual billing events, the Stripe Billing Publisher function will publish billing events to Stipe. To achieve this, it must first get configuration data that’s essential to this publishing process.
The function will acquire tenant configuration data from the Billing and Metering DynamoDB table to get the tenant’s external subscription identifier. It will also retrieve the restricted Stripe API key from the Secrets Manager (represented as Billing Provider API Key on the diagram).
The final step of the process is to go through each aggregated event and send them to the upcoming invoice associated with the Subscription Item in Stripe Billing. Usage will not be published to Stripe until the upcoming invoice is set to close. This reduces the number of requests to the Stripe API.
Running the Sample Solution
The sample solutions I have described in this post is available in a GitHub repository. This solution represents a working example of the strategy and concepts that were outlined above.
The details on how to set up the solution and get it running are covered in the README of the repository.
Summary
The move to a SaaS model requires companies to develop new pricing strategies. These strategies also require SaaS developers to introduce new constructs that can profile and publish data on tenant activity that is used to generate a bill.
A big part of this effort often includes creating an integration with a third-party billing provider. The nature of these integrations can change from one provider to the next.
The solution I have reference in this post is targeted at addressing the core elements of building a billing integration. It outlines some of the key considerations that will shape your approach and should give you a head start on the path to creating your own billing integration model.