Accelerate infrastructure as code development with open source Former2

Amazon Web Services Feed
Accelerate infrastructure as code development with open source Former2

In this post, AWS Community Hero, Ian Mckay, highlights the capabilities of Former2, an open source project that enables you to generate infrastructure as code (IaC) templates from existing AWS resources in your account. This post covers how Mckay created Former2, how to use it, the challenges around it, and the future for Former2.

When I first started building on AWS, like most developers, I used the AWS Management Console. After spinning up and tearing down Amazon Elastic Compute Cloud (Amazon EC2) instances manually many times, I realized that I needed a better way, so I looked to implement my solutions in AWS CloudFormation.

AWS CloudFormation is an infrastructure as code (IaC) service, which means I can use text files (JSON or YAML) to define a set of resources to be deployed without worrying about the semantics of how they are deployed. Using AWS CloudFormation helps with consistency and repeatability; however, it can take time to learn the syntax of the language and build the stack templates. Luckily, there is now an open source tool called Former2 to quickly and easily make the transition.

Former2 is an open source project that allows you to generate IaC templates (for example, AWS CloudFormation or HashiCorp Terraform) from the existing resources within your AWS account. In this article, I’ll explain how I created Former2, show how to use the tool, describe the challenges faced, and share my vision for the future.

Former2 background

The seeds of Former2 were planted during a customer engagement. I was developing solutions for customers directly within the AWS Management Console and looking for ways to automate the resources I created using AWS CloudFormation or the AWS SDK for Python (Boto3). This process sometimes took longer than I expected and was prone to errors, so I started to look for alternatives. I had previously worked on a browser automation tool called Wildfire and wanted to replicate the experience, so I started the Console Recorder for AWS project.

After initial adoption of the Console Recorder for AWS tool, I received comments from the community about the tool’s lack of support for existing resources. There wasn’t an easy way to add that functionality, so I started working on a tool to do the reverse and called it Former2. Instead of intercepting network requests like Console Recorder for AWS, Former2 takes advantage of the AWS SDK for JavaScript in the Browser. It makes Get/List calls to determine the resources within your AWS account and maps those SDK responses to the various template formats. Former2 derives its name from the previous CloudFormer tool.

Former2 has evolved to fully support all AWS CloudFormation types and includes many features, such as related resource suggestions, cross-referencing, a CLI version, and AWS CloudFormation import support. Both inexperienced and experienced AWS developers can use Former2 to quickly construct IaC templates for their companies or customers.

Using Former2

If you have an AWS account with resources and have access to a set of AWS credentials, you can use Former2 today. To generate an AWS CloudFormation template from scratch, first, visit the Former2 website. You should see an introduction screen with links to a browser extension. The browser extension is required to avoid CORS issues with some API endpoints. The extension is available for Google Chrome, Mozilla Firefox, and Microsoft Edge.

Once the browser extension is installed, click the Continue to Credentials button.

We need an IAM key pair to authenticate requests. Providing only read access with these credentials by assigning the ReadOnlyAccess policy to the IAM user is recommended. Enter IAM credentials in the spaces provided. Don’t worry, our credentials are not sent to any external server and are only used to sign requests to AWS endpoints.

Once the credentials have been entered, all prerequisites have been met and we can navigate via the dashboard or sidebar to specific services. Also, ensure the region selector in the top right is set to the region that contains your resources.

Select a service that has some resources within your AWS account and click on its link (I’ve chosen Lambda for this example). Once we are there, the table loads the resources that it has found. Select the resources to add to our outputs by selecting in the check box, then choose Add Selected. Finally, navigate to the top of the window and select Generate.

Congratulations, we have generated an entire AWS CloudFormation template that matches the resources we’ve selected. We can copy the template straight into a YAML text file and use that to redeploy those resources in the future.

Why open source?

I don’t consider these tools to be groundbreaking or to have a unique selling proposition. To borrow from Jeff Bezos, I think of these tools as undifferentiated heavy lifting. I don’t believe that the community should have to pay for these types of tools, particularly because the tools aim to assist developers early in their AWS journey who are just starting out with IaC.

Also, building trust is important. Handling AWS credentials is a high-risk activity, and Former2 is designed never to send these credentials to an external server that isn’t an AWS API endpoint. Being fully transparent about the code you are running is essential to show that you understand the sensitivity of the data.

Additionally, as open source software, Former2 can be contributed to by anyone, which means enhancements or bug fixes can be shared by the community. Former2 has more than 30 pull requests and more than 100 issues raised, which has helped significantly to shape the project. Many of these contributions are raising or fixing bugs that I otherwise wouldn’t find due to the sheer scale of the AWS products and services. I thank everyone who has contributed directly or indirectly to the project.

Lessons learned

Maintaining Former2 represents a huge challenge with the amazing rate of change that occurs within AWS products and services. Both AWS CloudFormation and HashiCorp Terraform have more than 500 resource types, with more added almost every week. (AWS re:Invent generates a landslide of updates!) Keeping up with that rate of change means the project requires constant attention, which has made me reconsider my own time management when planning around these weekly updates. At the time of writing, there are more than 1,000 programmed API calls and more than 5,000 property mappings.

In addition to the updates, many resources require multiple SDK calls to achieve the full resource information. There is more complexity than I initially considered when creating the tool, such as how to handle secrets only entered during resource creation or how to handle default resources such as the default VPCs. These challenges and their solutions continue to be worked on today, both by myself and the awesome open source community around the project.

Conclusion

In this article, I shared the inspiration behind Former2, explained why I created the project, and showed how to get started with infrastructure as code today. As I look to this project’s future, I am also working toward a few other things.

Former2 currently maintains mappings for all AWS CloudFormation resource types, but unfortunately the same can’t be said for the HashiCorp Terraform types, which are sitting at 50–60 percent coverage of resource mappings at the time of writing. My goal is to reach full support for these resources in the future. If you’re interested in contributing to the project, these mappings are an easy way to add support, as pull requests are welcomed.

I’m also interested in feedback from the AWS community about what you’d like to see in the tool. If you have an idea for improvement, I’d love to hear from you.

You can find out more about Former2 from its GitHub repository or try it for yourself on the Former2 website.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.