ElasticSearch Goes Deep on OpenTelemetry with eBPF Donation

Monitoring and observability provider Elastic is committed to further integrating the Elasticsearch project with OpenTelemetry in a major way, recognizing it as a vital cornerstone for enhancing users’ search experiences.

By leveraging Elasticsearch for data searches and observability across diverse databases and data types, users can benefit from its versatility and robustness. OpenTelemetry plays a pivotal role in this integration, serving as a central component in enabling seamless data monitoring and analysis across various environments.

With OpenTelemetry (OTel), the latest evolution of this integration underscores its significance: This manifests in the form of enhanced profiling capabilities, allowing users to gain deeper insights into system performance and resource utilization. As Elasticsearch continues to evolve alongside OpenTelemetry, users can anticipate even more powerful and comprehensive tools for managing and analyzing their data effectively.

The Donation

According to the proposal documentation, the donation of the Elastic profiling agent will:

“Fill the gap in OpenTelemetry’s component landscape/architecture with a mature, feature-rich and efficient profiling solution. With that, cutting-edge technologies in eBPF and profiling would become a standard through OpenTelemetry for collecting in-production profiling data. Collecting profiling data with OpenTelemetry across a broad range of languages/technologies would come with a frictionless deployment experience.”

The donation follows the “marriage” between the observability tools Elastic Common Schema (ECS) and OpenTelemetry Semantic Conventions. Specifically, the creators of open source Elastic are contributing ECS to OpenTelemetry and are committed to the joint development of the two projects. 

The Integration

The integration of the Elastic profiling agent, as well as ECS, with OTel, underscores Elastic’s and OTel’s reach and its creators’ goals to allow users to merge telemetry data into a single panel for a more comprehensive analysis for observability. Indeed, the integration of ECS with OTel helps the OTel project move toward the ultimate goal of total compatibility and standardization with any observability tool or process.

In other words, both Elasticsearch and OpenTelemetry — especially since their general availability was released a few weeks before 2024 — are very popular platforms to integrate and work with data logs, metrics and traces from various sources and their further integration should be appreciated by many.

“The benefit you get once the profiling agent is accepted is that users will have continuous profiling available within the OpenTelemetry framework,” Abhishek Singh, general manager at Elastic, told The New Stack. “This is because the agent that we’re donating uses eBPF, making it less intrusive when it comes to implementation than a language-based agent, for example.”

Elastic is collaborating with OpenTelemetry across various areas, not solely on profiling but also on the common schema and the semantic convention. The goal is to transform OpenTelemetry from a fragmented landscape with scattered capabilities into a comprehensive observability framework that customers rely on by default, Singh said.

Its contribution covers a complete implementation of a continuous profiler. This encompasses the agent responsible for running systems and conducting profiling tasks. Essentially, Elastic provides the instrumentation component that operates alongside application software, gathering information to profile the software and offering sufficient data on where time is allocated within the application, Singh noted.

Practical applications include cost reduction and optimization efforts. For instance, by leveraging profiling, significant CPU-consuming functions within applications can be identified and optimized, thereby reducing compute usage and resulting in cost savings, Singh said.

Moreover, insights into environmental impact are provided. Carbon emissions might be analyzed based on compute usage, allowing for the optimization of certain functions or libraries to minimize environmental impact and generate cost savings for customers, Singh said.

“Jumping in from a CTO perspective, when considering Elasticsearch, our aim is to provide answers in milliseconds. We are the leading search analytics company, trusted by over 50% of Fortune 500 organizations. Our platform enables users to find answers in real-time using both structured and unstructured data, allowing businesses to leverage their data at scale,” Singh said.

“Our customers benefit from a platform that enables bot1h search and analysis of data, regardless of its type. This integration of observability data with business data allows for deeper insights and operational efficiencies. Our goal is to make OpenTelemetry the common instrumentation field, ensuring that advancements in instrumentation benefit the entire community and drive better outcomes for customers.”

Elasticsearch’s key draw is its versatility. For example, consider the perspective of a CTO. They may want to analyze various aspects such as node performance in a Kubernetes cluster or sales data across different regions like Europe and North America. In case of a disaster or outage, security personnel rely on Elasticsearch to pinpoint the root causes by conducting detailed searches.

Additionally, operations teams tasked with continuous monitoring face the challenging responsibility of drawing inferences and analyzing data to ensure system stability and performance.

Why OpenTelemetry?

Why OpenTelemetry? As the second-highest CNCF “velocity project” thanks to the strong growth of its user base in the CNCF ecosystem, OpenTelemetry has become a widely adopted way to add instrumentation to an application to gather metrics, logs and traces from your favorite observability source. Telemetry data (metrics, logs and traces) from different sources can then be combined for monitoring with your favorite panel, such as with Grafana.

Wildly popular ECS is used to define a common set of fields to be used when storing event data in Elasticsearch, such as logs and metrics and to specific field names and Elasticsearch datatypes for each field, and provides descriptions and example usage, according to its documentation. ECS will become that much better under the OpenTelemetry umbrella. In fact, machine learning is being integrated with Elastic, which is already offering some very interesting results.

Austin Parker, director of open source at Honeycomb.io and a leading OpenTelemetry maintainer told The New Stack, that the elastic profiling agent remains in progress, noting how “there is a period of due diligence and technical review that has to happen before we can accept it.”

Parker did note that since the agent is proprietary and a closed source, there have been some “thorny legal considerations that a bunch of different companies are having to review,” but that said Parker did not “anticipate the diligence finding anything dramatic or a blocker.”

“We’re very happy to see Elastic continue to partner with the open source observability community and believe that this donation will be beneficial to the progress of OpenTelemetry’s profiling signal,” Parker said.