Calyptia Builds Observability Platform around Fluentd, Fluent Bit

Anurag Gupta and Eduardo Silva launched Calyptia a year ago to support the rapidly growing community around Fluentd and Fluent Bit, open source data collection and observability tools that the two helped create and maintain and that now are a project of the Cloud Native Computing Foundation (CNCF).

With Calyptia, Gupta and Silva saw an opportunity to launch a company around Fluentd and Fluent Bit that could help organizations with first-mile observability, the collection of data from multiple sources — such as applications, infrastructure and network devices — that is then parsed and transformed and eventually sent to backend databases and data lakes.

Most enterprises when handling data focus on those backend environments, Gupta told The New Stack.

“You have a couple of practitioners that are doing queries and actions and alerts, all on top of the data,” he said. “But if we can shift that left to where the data is collected, we can make it much more resilient, much faster and potentially much more insightful, because we have the full context of where the data is, what it’s doing and what it looks like. That’s the greater ambition for our company, but we really believe in the first mile as being that first step into how folks experience observability.”

Calyptia in Action

The company in June announced the initial release of Calyptia Fluentd v1.0, its own distribution of the tool. It took another significant step with the recent introduction of Calyptia Cloud, a hosted data observability platform based on Fluentd and Fluent Bit that provides first-mile data collection capabilities that can scale to manage petabytes of data across thousands of servers per day.

Calyptia will be at KubeCon+CloudNativeCon this week to talk more about first-mile observability and its new cloud platform.

Fluentd — a vendor-neutral data collector used to build a unified logging layer — and Fluent Bit — a logging and metrics processor for collecting data from various sources — have grown in popularity since being developed at Treasure Data a decade ago, with its Ruby-based code released to the open-source community. They are downloaded millions of times a day and are used by such cloud providers as Amazon Web Services (AWS), Microsoft Azure, DigitalOcean and Oracle.

It comes at a time when cloud adoption is growing rapidly, use of cloud-native tools like Kubernetes and containers is surging and companies of all types are being forced to adopt software capabilities. As they become software companies, they need to be able to see what’s going on with the software they use, diagnose problems and troubleshoot.

No Vendor Lock-In

Enterprises have options when it comes to observability products from the likes of Elasticsearch and Datadog, but they offer agents that essentially lock in organizations, Gupta said. That means enterprises have only one way to collect the data, they either have to collect all or none of the data – it’s difficult to filter it out – and the more data that’s collected, the more they pay.

“We saw this need to be able to decouple a way to collect that data and decouple where you send that data,” he said. “We’ve seen tremendous momentum with people saying they want to own our own strategy and not be locked into these vendors. They don’t want to collect useless data or pay for the useless data. That’s where we see what we call first-mile observability. The first part of your journey, the first mile is collecting that data and really honing it.”

Calyptia Cloud is an overlay on top of Fluentd and Fluent Bit. The platform is hosted on Calyptia’s own servers and plugs into environments using the open source tools, whether that is on-premises or on private or public clouds, and provides a control plane for managing and monitoring observability pipelines. Users can collect, parse, filter and route the first-mile data and the Calyptia Cloud also offers companies using Fluentd and Fluent Bit data on cost, speed and performance bottlenecks via real-time metrics.

“All that action and parsing this part of the pipeline, we make it turnkey to say this is all the actions and stuff that you’re going to be doing along the way,” Gupta said. “The last part that we do as Calyptia Cloud is just give real-time metrics so you can understand the performance of this first-mile observability layer. Am I sending data too fast or sending data too slow? Am I buffering data? All these are things that generally a lot of folks will ignore when doing first mile, but as the maturity grows, it becomes ever-present in folks’ minds.”

He added that “all we’re taking is metadata. You’re not sending us logs, you’re not sending us what customer information might be flowing through the system. Just things about how the system is performing.”

Interest in Calyptia Cloud Is High

Though Calyptia Cloud was just announced, the company has had about 1,700 people sign up for it, including two Fortune 10 financial firms, hardware providers and large software companies. There also has been interest from companies in the rising non-fungible token (NFT) and cryptocurrency markets, where the massive services ae software-based, he said.

Right now, the company is focused on the first-mile data collection leg, partnering with other companies that supply data and analytics services.

“We don’t have really strong opinions on where to send the data,” Gupta said. “If you want to send data to [a particular vendor], great. If you want to take that data [and] send it to 10 different places, awesome. We’ll support both. For now, we’re heavily vendor-neutral. We do anticipate bringing more insights to our own platform, but that’s more of a coexist and enrichment experience vs. trying to do a replacement.”

The first mile is an expanding trend that started out with agents and now is extending into processing all that information, Silva told The New Stack.

“A lot of functionality that is done in the destination to extract value from that data is moving to the left,” Silva said. “Right now, as fast as possible, you can extract value from your data. It’s better for your business. But if your infrastructure is growing and you have to wait minutes in line to extract that value, you miss a lot of opportunities. Everything that matters here is how fast and how reliable the business can extract value from the data and our value is we are providing all these tools to complete a stack that is vendor-neutral to extract value.”