The Importance of Resilience Testing and Observability

People like to write and talk about hard things, but sometimes it’s better to talk about easy things. It’s easier than ever to write and deploy code and to have that code scale up automatically. It’s also easy for that code to become an alarm clock for your development team, waking them up every night. Day after day and week after week, the experience of operating an application can wear on the development team, slowing down feature velocity and sapping ambition.

The surprising thing is that it’s also easy to take some small steps which will change the way your development team experiences their application. By using the tools and practices from observability and resilience on a regular cadence, as part of the normal development process, you can improve the feature velocity of your development team — and improve their quality of life at the same time.

What is Observability?

Observability is often explained in terms of control theory, as understanding the state of a system only through its outputs. Others explain it as being built for unknown unknowns. The most practical definition is that observability allows you to connect effects with causes across many different services, domains, and scales.

When a team is trying to understand how their application is behaving in production, it’s not some abstract exercise. There’s something that changed and they need to understand why in a way that lets them restore or improve their application. Observability isn’t a tool and it isn’t telemetry; it’s an approach to having an accurate and fact-based model of application behavior to inform human decision making.

Resilience and How to Test It

Resilience is the ability of a (socio-technical) system to minimize the impact of failure. Failure happens, all the time. Resilient systems take progressive steps to allow the most useful parts of the system to still serve their purpose. While particular tactics like circuit breakers or load shedding — or even basics like retrying failures — are part of building resilience, resilience, too, is a mindset. It’s an approach that acknowledges the reality of complex systems and the presence of failure and error.

Resilience isn’t something that is achieved and then never considered again. It must be tested, and ideally not just when things happen to break. Chaos Engineering is a systematic approach to testing the resilience of a system (including the people) and also many types of failures. By purposefully introducing failures and degradations into the system, the development team can see what happens. Even more than that, they can find out what the gaps in their observability and resilience are, to “see what they can’t see.”

A Better Customer Experience

Ultimately, resilience testing and observability are about one thing: giving your customers a better experience. Faster apps, less downtime, and more features. By understanding the inner workings of all of your systems — and actively testing it against failures — you gain confidence in your teams, your technology, and your ability to keep customers happy even when things are going wrong. The journey to better sleep, and faster development, starts with a single step.

No customer will ever say, “wow this app is so observable” or “can you believe how resilient this software is?” In many ways, the effects of observability and resilience are unseen by those most impacted by them.

But when your system scales or reaches peak load, when traffic spikes or a large customer changes their behavior, when you deploy new services or your downstream dependencies deploy new services, when you begin a migration, when you complete a migration, or when nothing out of the ordinary appears to be happening but things are still breaking — in all these cases, a tested observability and resilience practice will let your system keep working and your devs keep sleeping.

👁 Image