OpenTelemetry and Elastic Common Schema Comes Not Too Soon

Standardization is always a good thing, but when it comes to very popular and utilized tools and technologies, a common standard at the very least makes life a lot easier for developers. At the other extreme, a technology’s survival depends on it. This is the case with WebAssembly, regarding the dire need for a common standard for components so Wasm can be used to very efficiently deploy code anywhere across any type of device running with a CPU. Today we are talking about the marriage between the observability tools Elastic Common Schema (ECS) and OpenTelemetry Semantic Conventions. Specifically, the creators of open source Elastic are contributing ECS to OTel and are committed to the joint development of the two projects. 

Second Highest

As the second-highest CNCF “velocity project” thanks to the strong growth of its user base, in the CNCF ecosystem. OpenTelemetry has become a widely adopted way to add instrumentation to an application to gather metrics, logs and traces from your favorite observability source. Telemetry data (metrics, logs and traces) from different sources can then be combined for monitoring with your favorite panel, such as with Grafana. Wildly popular ECS is used to define a common set of fields to be used when storing event data in Elasticsearch, such as logs and metrics and to specific field names and Elasticsearch datatypes for each field, and provides descriptions and example usage, according to its documentation. ECS will become that much better under the OTel umbrella. In fact, machine learning is being integrated with Elastic, which is already offering some very interesting results.  With the collaboration between the open telemetry and the Elastic Search, this is the case when standardization could not have come sooner. “This collaboration between ECS and OpenTelemetry is a marriage made in heaven,” Torsten Volk, an analyst at Enterprise Management Associates (EMA), said. “ECS addresses the most critical bottleneck of true visibility and observability: the creation and maintenance of a common data model for all telemetry data.”

Developers Choice

Now OpenTelemetry can reliably collect data fields from Python, C#, JavaScript or the language of the developer’s choice from various APM tools and benchmarking platforms to fill in the context gap that often slows down or entirely prevents app-centric monitoring, Volk said.

“For instance, an e-commerce platform experiences a sudden surge in server load during a flash sale. With different services coded in different languages and monitored by different APM tools, the root-cause analysis would be tricky,” Volk said. “But with a common data model in place, all of the different languages and APMs dump their telemetry data into consistent JSON files that today’s magical AI-driven observability platforms can easily analyze. Thinking this further, the ECS standard can help OpenTelemetry analyze asset information from today’s heterogeneous universe of smart devices and report back to developers which devices work best and, more importantly, which ones work the poorest with their latest code creations.“

The positive impact that the contribution of ECS to OTel will have for OpenTelemetry users is applicable both generally and particularly for users of OpenTelemetry’s in-development logging capabilities, Morgan McLean, director of product management, Splunk, said. This is because beyond OpenTelemetry’s Collector agent and language instrumentation, one of the project’s biggest draws is its unified semantic conventions, which ensure that consistent metadata and resource information is attached to every signal, McLean said.

For example, spans of HTTP requests captured from services written in different languages will share the same keys and value encodings for their duration, URL, service name, host, etc., which “allows them to be analyzed very effectively,” McLean said.

“While this is already the case with spans and metrics in OpenTelemetry, we’re in the midst of adding support for logs, which introduce significantly more scenarios that require dedicated semantic conventions,” McLean said. “By merging ECS and its thousands of existing conventions into OpenTelemetry, everyone who uses OpenTelemetry will receive well-structured and consistent metadata on their logs, traces, metrics and more from the source in a huge variety of scenarios. These signals can then be processed, filtered through, compared, and analyzed efficiently without massive amounts of special casing logic or ignoring signals that lack expected metadata.”

ECS and OTel

The integration of ECS with OTel underscores OTel’s reach and its creators’ goals to allow users to merge telemetry data into a single panel for a more comprehensive analysis for observability. Describing this aspect of OTel in general without commenting specifically about the ECS contribution to Otel, Cedric Ziel, Grafana Labs senior product manager, noted how OpenTelemetry is a community-driven and CNCF governed initiative “that aims at commoditizing data collection concerns for observability.”

“The ideals behind OpenTelemetry are about vendor-neutral instrumentation of application code and the project was created to remove the need for people to rip and replace their instrumentation whenever they want to lean in on a different observability provider or even support multiple vendors at the same time. This is solving the problem of observability in our time: it is inevitable that you need multiple vendors to support your needs in different dimensions — settling on the same protocol and the same conventions for this is the holy grail in observability,” Ziel said. “There is not just one single thing that’s moving in OTel that makes it more attractive: It’s the overall thing. Seeing all signal types being more sustainably available across instrumentation libraries is a continuous effort and exciting to see.”

Indeed, the integration of ECS with OTel helps the OTel project move toward the ultimate goal of total compatibility and standardization with any observability tool or process.

“Since day one, OpenTelemetry has been focused on providing consistent, clear, and accurate telemetry data about cloud native systems to empower developers and operators with observability. The alignment of the ECS and the OpenTelemetry Semantic Conventions is another step along that journey, ensuring that high-quality and consistent metadata is available to end users,” Austin Parker, head of developer relations, Lightstep, said. “In addition, this step ensures that OpenTelemetry data will be the gold standard for the next generation of observability tooling powered by AI, empowering end-users to get better answers about their system and its state.”