Amazon Web Services Feed
Vortexa delivers real-time insights on Amazon MSK with Lenses.io
This is a guest post by Lenses.io. In their own words, “Lenses.io is a group of engineers, designers, developers, data analysts, tinkerers, writers, and open-source contributors. We specialize in the democratizing of data technologies. That’s why we developed Lenses.io (Lenses-eye-oh) to help enterprises simplify their operations by making data work for teams, not the other way around.”
This post discusses how Vortexa harnesses the power of Apache Kafka to improve real-time data accuracy and accelerate time-to-market by using a combination of Lenses.io for greater observability and Amazon Managed Streaming for Apache Kafka (Amazon MSK) to create clusters on demand.
Vortexa provides a real-time analytics platform for seaborne oil supply, which enables energy market players to make smarter trading and shipping decisions. Their service needs to process tens of millions of data points per day, and instant insights on the slightest changes in ship movements are crucial to their customers.
That’s why Vortexa’s data analytics team relies on powerful data streaming technologies such as Apache Kafka to make their real-time applications work.
However, as they scaled their self-managed Apache Kafka environment, they saw several challenges emerge:
- Debugging incidents took days. Investigation of incidents involved a series of in-house developed scripts and open-source tools.
- The smallest mistake could bring down a whole cluster. The team ran the risk of downtime due to difficulties in manual configuration and a nebulous command line.
- Slowdowns affected production. This meant more time spent on everyday tasks, such as assessing the data quality, and less time shipping new features to market.
- Specialist skills were difficult to come by. Having a specialist in-house for every data platform permutation—like someone who understood how to both build and operate Apache Kafka—wasn’t wise or sustainable.
And so Fridays for Vortexa became “Kafka Fridays.” The fragilities and complexities in an all-important Apache Kafka infrastructure cost the team their Fridays and often their weekends, and also ate into their release schedules.
Self-healing and observability save the day
To address this, the Vortexa team evaluated several managed solutions, and found that Amazon MSK best met their requirements and budget. Amazon MSK continuously monitors cluster performance, is self-healing, and automatically replaces unhealthy nodes without downtime.
Already users of multiple other AWS services, Vortexa benefited from Amazon MSK integrating with other AWS services such as Amazon Virtual Private Cloud (Amazon VPC), AWS Key Management Service (AWS KMS), and AWS Identity and Access Management (IAM). Amazon MSK can also integrate with Amazon CloudWatch, AWS CloudTrail, and Lenses.io for more effective, integrated monitoring and logging of issues.
However, Apache Kafka was still a black box when it came to understanding their deployed flows. Vortexa uses Lenses.io as their data operations portal, which gives them Apache Kafka monitoring, data observability and governance control for their real-time applications that run across their Amazon MSK and Kubernetes platform. With a clear UI, they can troubleshoot a problem on a streaming application within a few minutes compared to hours taken before, which led to an improvement of at least 30-fold.
“Lenses.io gives us a complete vision and extra management capabilities for our streaming applications in Apache Kafka clusters,” says Nguyen Lam Phuc, Senior DevOps Engineer at Vortexa. “Testing as well as operations suddenly became painless and felt much more intuitive than ever before.”
Previously, the team also needed to manually run commands, add connectors involving trial and error, and implement complex deployment procedures—and cross their fingers that this would work.
“In our four-year history, we’ve had at least five different ways of operating, and four different ways of monitoring Apache Kafka, says Jakub Korzeniowski, Head of Data Services at Vortexa. “The combination of Lenses.io and Amazon MSK has been the only solution that allows us to focus on business logic instead of concentrating the majority of our efforts on just meeting SLAs.”
Now, through a single dashboard, Vortexa can use Lenses.io to view their application topology and expose consumer lag metrics that they can visualize and alert in Lenses.io before forwarding to third-party tools such as Slack and PagerDuty.
Open monitoring with Amazon MSK and Lenses.io
You can monitor the health of the Amazon MSK infrastructure within external tools using the AWS open monitoring framework. Vortexa uses open monitoring with Prometheus to access JMX metrics that Amazon MSK brokers generate. The following diagram illustrates this architecture.
Value in data quality and release velocity for Vortexa
Maksym Schipka, Vortexa’s CTO, explains how Amazon MSK and Lenses.io saved 15% of working hours in Apache Kafka management: “At least twice a week, we would experience issues with Kafka. We were spending so long trying to understand the issue that it was more efficient for us to replace the cluster with a completely new one.
“Amazon MSK and Lenses.io have been pivotal technologies for Vortexa, enabling us to shift significant efforts from maintaining and stabilizing a complex and fragile Kafka infrastructure to focusing more on the quality of analytics and market insights that directly impact the value we deliver to our customers.”
“We can now confidently and frequently update our applications at scale, run complex Kafka streams topologies with ease, and debug applications instead of infrastructure. Our revenue and market share are directly impacted by the speed at which we deliver accurate data to the market, and Amazon MSK and Lenses.io are key enablers of that.”
Fast-tracking to production in minutes, not days
Vortexa can now build new data pipelines that integrate data across their different technologies, including Amazon Elasticsearch Service and Redis, and deploy into production in minutes.
Everyday tasks, such as inspecting messages on a stream with SQL, understanding the profile of a topic, or verifying the replication factor are now easy and performed in under 10 seconds, without the need to have specific in-house Apache Kafka expertise.
“Amazon MSK with Lenses SQL is one step towards making Kafka feel like any other database,” says Romain Guion, Head of Signal Processing and Enrichment at Vortexa. “It allows us to verify the outputs of our models well, which recently resulted in a 90% decrease in AIS data affected by signal noise and spoofing.”
The Vortexa team sees a real-time topological view of their entire data landscape, including their flows to Elasticsearch and Redis and inside their Apache Kafka streams applications.
Conclusion
Using Amazon MSK and Lenses.io helped Vortexa to:
- Save 15% of working hours on Apache Kafka management
- Deploy new Amazon MSK clusters into production in minutes, not days
- Increase Apache Kafka users (both operators and application developers) by 200%
- Reduce automatic identification system signal noise by 90%, aided by quality assurance done exclusively in Lenses.io
To learn how to get greater visibility into your data streams and streaming platform, visit Lenses.io. For information on how to build and run production applications on Apache Kafka without needing Apache Kafka infrastructure management expertise, visit Amazon Managed Streaming for Apache Kafka.
About the author
Andrew Stevenson is the Chief Technology Officer and co-founder of Lenses.io. He leads Lenses.io’s world-class engineering team and technical strategy.