Behind the scenes as AWS AppConfig builds a Lambda extension

AWS Feed
Behind the scenes as AWS AppConfig builds a Lambda extension

In this blog post, I will share why the AWS AppConfig team built an AWS Lambda extension (hint: customers wanted it), the effort required to build it (hint: it was easy), and the outcomes of building our Lambda extension (hint: lots). I will cover the technical and business aspects of building a Lambda extension and provide some behind-the-scenes information. If you’re interested in building a Lambda extension, I hope you can learn from our experience.

Some background on AWS AppConfig and Lambda extensions

AWS AppConfig is a capability of AWS Systems Manager that allows engineering teams to create, validate, and deploy dynamic application configurations. As you might already know, it’s a modern best practice to use dynamic configuration at runtime to adjust software behavior. When you use AppConfig, your teams can be responsive, turn features on or off, or throttle up or down other configurations in real time, without restarting the app. AWS AppConfig empowers you to safely and quickly roll out application configuration changes in a controlled, validated, and monitored way. With AppConfig, configuration changes can be validated before deployment to prevent errors. AppConfig also automatically rolls back a recent configuration change if an alarm is triggered. Thousands of teams at Amazon use AppConfig to release new features and respond to operational events at scale.

Lambda extensions are a new way to easily integrate AWS Lambda with your favorite monitoring, observability, security, and governance tools. A Lambda extension requires no complex installation or configuration, which makes it easier for you to use your preferred tools across your application portfolio.

Here are some use cases for Lambda extensions:

• Capturing diagnostic information before, during, and after function invocation.
• Automatically instrumenting your code without the need for code changes.
• Fetching configuration settings or secrets before the function invocation.
• Detecting and alerting on function activity through hardened security agents, which can run as separate processes from the function.

Deciding what we wanted to build

Before we built the Lambda extension, we already knew that customers were using Lambda and AppConfig together and we had a good understanding of use cases. Although it wasn’t too difficult for customers to integrate Lambda and AppConfig, they did have to do some repetitive work in setup and the caching of the configuration. If we could put our best practices into a software package, then customers wouldn’t need to build their own integration.

It was fairly obvious that we could make workflows simpler for our customers. We shared our ideas about how to simplify the integration, which included the use of Lambda layers, with the Lambda team.

Defining and designing

We knew we wanted to focus our extension on how customers use, retrieve, and cache dynamic configurations used in Lambda functions. Based on customer use cases, we identified these capabilities as crucial for our extension: methods to simplify calls to AppConfig and a caching solution for configurations.

We used the Lambda Extension API to determine how we could build those capabilities. We wanted to understand the lifecycle phases for extensions: Init, Invoke, and Shutdown. We knew that the initial call from the extension to AppConfig should occur at Init, but because AppConfig is all about making updates to dynamic configuration (not static configuration), we decided to use Invoke too. To be more exact, the Invoke event tells us that we can run our code because the Lambda function is currently running. The extension uses polling intervals to check for new configurations.

After we understood the events model, we started the technical design. Because all Lambda extensions include a Lambda layer that exists outside of a Lambda function, we had an opportunity to make calls between a Lambda function and AppConfig simpler for customers. The goal of our Lambda extension is to cache and update AppConfig configurations in a way that persists between Lambda handler invocations. When a configuration is needed, the Lambda handler makes a request to the extension instead of a call directly to the AppConfig service. The configuration is cached locally, and thus returns much faster (nanoseconds versus milliseconds) for a call to the GetConfiguration API.

We decided the extension should expose a local HTTP server that allows the Lambda function to retrieve cached configurations from AppConfig. The extension is configured to fetch a predetermined configuration from AppConfig. When a Lambda function is invoked, the extension checks if the time elapsed since the last fetch from AppConfig is greater than the poll interval. If it is, the extension fetches the latest configurations from AppConfig and updates the cache. Although there are other ways to have a Lambda function get data from an extension, for our use case, a local HTTP server was the best choice.

As we looked at our research and design, we knew that this extension would be a win for customers. They wouldn’t have to write their own implementation for calling and caching configurations from AppConfig. The extension would enforce best practices for AppConfig like time-to-live (TTL) values and caching parameters. As we looked at the whiteboards, we felt we had something exciting to build.

Building and releasing

Although you can build Lambda layers in many languages, our team chose Go because of its efficiency and compactness of code. Because our team was one of the first teams to build a Lambda extension, we decided to build a simple Hello World extension in Go. (That Hello World extension eventually became a sample stub app that we posted on GitHub.) We used the Hello World extension to the basic structure, including the Init, Invoke, and Shutdown lifecycle phases.

We extended our Hello World extension, built a mini-framework, and then tacked on the agent. Our first task was the polling of AppConfig. We wanted the extension to regularly poll AppConfig, looking for new versions of configuration data. If there was a newer version than what was already local, we wanted to fetch and cache the newest version. Because the Lambda container is frozen between invocations on a separate thread, we couldn’t have a background poller on a separate thread. So we used the invoke event to check polling time intervals since the last request to AppConfig.

We needed a way to pass configurations from AppConfig to the Lambda function. We considered these options:

• Writing a local file that the Lambda function could read.
• Writing configuration data to Lambda environment variables.

Our customers might have difficulty debugging and storing lots of configuration in a local file. And storing complicated configuration data in Lambda environment variables isn’t a good solution either, because we might overload those environment variables with data like string arrays. These options simply aren’t as clean as an HTTP server. And again, that initial HTTP server stub written in Go became another sample on GitHub.

We built out the other parts of our extension: caching logic, ability to override defaults for polling intervals, timeouts, and port numbers. For testing, we added our extension through an Amazon Resource Name (ARN) in each AWS Region.

When we rolled out our extension, customers found it was easy to add. They could add a Lambda layer to their functions and simply point to the AppConfig extension ARN. In addition to writing documentation for this exciting new functionality, we also published a blog post, Deploying application configuration to serverless: Introducing the AWS AppConfig Lambda extension, just after launch.

Initial results

We’re happy to say that the initial results have been quite positive. Because we built AppConfig best practices into the Lambda extension, customers get up to speed more quickly. What used to take almost a dozen steps can now be done in only four steps. We have built caching and polling defaults into the extension to ensure that best practices are used when calling AppConfig. We’re talking to customers about improvements and new features in future versions of the extension.

Conclusion

If you have customers who use Lambda and your service, it might be a good idea to create a Lambda extension for your service. If you’d like to get started, see the Lambda Extensions API in the AWS Lambda Developer Guide, the sample extensions on GitHub, and the Building Extensions for AWS Lambda blog post.

Good luck building your extension! We’re confident you’ll see positive results with your customers.

Further reading

• Adding the AWS AppConfig Lambda extension in the AWS AppConfig User Guide
• Deploying application configuration to serverless: Introducing the AWS AppConfig Lambda extension blog post
• Using Lambda extensions in the AWS Lambda Developer Guide

About the authors