Without observability it’s just code — O11y guide part 7

Welcome to another chapter in the ongoing series I started covering my journey into the world of cloud native observability. If you missed any of the previous articles, head on back to the introduction for a quick update.

After laying out the groundwork for this series in the initial article, I spent time sharing who the observability players are, looked at the ongoing discussion around monitoring pillars versus phases, and shared thoughts on architectural level choices being made, and shared the open standards available within the open source landscape. I continued onwards with a few of the architectural challenges you might encounter when older monolithic applications and monitoring tools are still part of an organization’s infrastructure landscape. 

Being a developer from my early days in IT, it’s been very interesting to explore the complexities of cloud native o11y. Monitoring applications goes way beyond just writing and deploying code, especially in the cloud native world. One thing remains the same, maintaining your organization’s architecture always requires that you understand that without observability, it’s just code you are deploying.

Let’s look at what makes cloud native observability so crucial to your organization’s success.

The phrase Without Observability It’s Just Code originated back 10 years ago when I first designed a t-shirt for some event with the theme being around DevOps with the phrase Without Ops, It’s Just Code. Today it’s not hard to see the transition to cloud native observability has become essential for many organizations as they explore digital transformation paths to modernizing their architectures.

Initially when developers and operations teams were moving into the cloud, they were confronted with infrastructure as a service (IaaS) processes and often they were virtual machine (VM) based environments. These led to environments they could both understand and monitor without too much drama. This was due to the one-to-one relationship often found between the VM-based infrastructure, hosting a monolithic application, that delivered to the business a product or service that their customers were able to use.

As the evolution to cloud native became a reality, these same developers and operations teams were exposed to auto-scaling, descriptive environments where containers and microservices became the new normal. The complexity skyrocketed overnight with relationships becoming many-to-many as infrastructure became a sea of containers running on container platforms, applications split out from single monoliths into multiple microservices, and businesses are now presenting use cases, clients, and geo-located interactions with the underlying microservices.

This complexity brings more data with it, both application generated data as customers interact and monitoring data. This growth in data means that costs are higher to operate in these environments, especially when you are applying the same observability technologies to this new cloud native environment. 

What happens when your organization is successful in a cloud native environment and all of the above issues are applied at massive scale?

When developers and operations teams are working in a cloud native environment at scale, other problems arise. The time with which teams must act and react is drastically shortened, with delivery of code being done in smaller increments and at a much faster pace. With hundreds or thousands of microservices being developed, updated, and deployed multiple times a day or even inside an hour, the processes that brings together the DevOps culture becomes a necessity.

Along with the processes comes a flood of new data, from applications, microservices, containers, observability, and other aspects of the cloud native environment. This also results in tooling sprawl, where many new tools enter the domains of the DevOps teams. All of this increases the cognitive overload and stress for these teams.

If you look around the internet searching for how much time developers are actually spending on the job they signed up for, coding and building new things, is shocking. Here are just a few samples of the results you might find:

• Developers spending 32% of time writing or improving new code.
• Eighty-three precent of developers are burning out, 20% of engineers suffering from burnout attributed the cause to “unreliable software.”

In the following diagram you can see the, maybe a bit optimistic, split on how developers in DevOps teams are buried in everything but the new development for organizations that hired them.

In a recent article I read, The growth of observability data is out of control, two important statements were made that describe the situation and how organizations are facing these challenges.

“It’s remarkable how common this situation is, where an organization is paying more for their observability data (typically metrics, logs, traces, and sometimes events), than they do for their production infrastructure.”

The research backs up this statement, when “…69% of companies are concerned with the rate of growth of their observability data.”

Furthermore, when a simple Hello World application is deployed on a four node cloud cluster and basic metrics are collected, the results were shocking. Thirty days of data was collected and it ended up generating +450 GB of data, that’s almost half a terabyte of data you have to store, query, and pay for. Can you imagine if you run hundreds of thousands of applications?

Let’s be honest, the cost would not be the problem if the outcomes were better. If you customers were happy, if you can fix issues quickly, or if you are able to retain your DevOps resources without burning them out on call. Short of collecting less data on purpose, the problem will remain as your data increases.

So what does observability look like that elevates your code beyond just code?

The key to understanding observability happiness is to no longer approach this as solvable with the mantra of metrics, logs, and tracing. Putting a solid strategy in place, with a business outcome focus will lay the foundations for a successful cloud native existence. Focus on the three phases of observability; know, triage, and understand.

You need the ability to approach, for your on-call engineers, cloud native observability through the three phases: