The 3 Pillars of Observability

Observability has emerged as a key component in the successful management of distributed systems. These include, of course, microservices and Kubernetes architectures. The requisite capabilities that observability must offer include metrics, logs, and distributed tracing, or in other words, the “three pillars” of observability.

In this episode of The New Stack Makers podcast, Daniel Spoonhower, chief technology officer of Lightstep, discussed and described what the “three pillars” concept means for DevOps, how monitoring is different, Lightstep’s evolution in developing observability solutions and a number of other related themes. The company today announced it’s expanding from distributed tracing to incorporate a larger array of metrics, making a Lightstep observability platform.

Lightstep CTO Daniel Spoonhower – The 3 Pillars of Observability

Spoonhower — whose experience in developing observability tools traces back to work as a software engineer at Google — makes it clear that a “three pillar” observability solution consisting of metrics, logs, and distributed tracing represents, in fact, separate capabilities.

“I think the thing that we’ve kind of seen is that thinking of those as three different tools that you can just kind of squish together is not really a great solution. I mean, the way that I think about observability is I like to get away from the what the specific tools are, and just say that observability is the thing that helps you connect the effects that you’re seeing — whether that’s performance or user experience, or whatever, connecting those effects back to the causes,” Spoonhower said. “And the thing that happened with deep systems is that it’s not like there are five or 10 potential causes to those problems, but there are thousands or tens of thousands of those things. And so you need a tool to help you find those.”

A common question is what is the difference between an observability and monitoring platform. “And I think one of the ways that people try to distinguish observability from monitoring is you might not know all of those potential problems in advance. You need a way to sort of explore or have a tool that helps you surface those insights in some way,” Spoonhower said. “In a monitoring platform, it is really about having” real-user,  infrastructure, and application monitoring and the platform.

However, for observability, “you need all those different data sources, but the data sources really need to get unified,” Spoonhower said. “So, that when you’re debugging something in terms of application performance, you can still look at an infrastructure metric and understand whether or not that’s impacting performance.”

Meanwhile, Lightstep’s platform is becoming more observability-centric. It is adding, for example, metrics added to applications for incident resolution, automating observability-related tasks and serving as the framework for organizations that are adding observability capabilities to their IT operations.

“Our roots are in distributed tracing — I think that was really where we saw a gap five years ago. But I think what we’ve learned along the way is that tracing is not just a tool, in terms of showing people traces, but the tracing itself can actually inform the way the rest of observability platform works,” Spoonhower said. “So, to put that in context, the thing that we’re announcing is that like stuff now will also integrate runtime metrics into your platform.”