ESG report: managing the exploding volumes of observability data 

Are you doing the right things to address your growing observability data burden? Perhaps not, a new report suggests.

Almost all organizations have had to speed up DevOps over the last three years to keep up with business demands. But many are challenged. That’s primarily because of the digital explosion in enterprise apps that businesses are building to make operations more efficient and workers more productive.

If this sounds like it might be too much for already-stretched-thin IT organizations, we think so, too.

Chronosphere recently partnered with analyst firm ESG, a division of TechTarget, on a report on observability in cloud-native environments. ​​ESG surveyed 357 IT, DevOps, and application development professionals at organizations in North America responsible for evaluating, purchasing, managing, and building application infrastructure. Everyone who took the survey either employs today, or plans to employ, an observability practice.

An expected, but still significant finding: 71% of organizations said their observability data is growing at a “concerning” rate.

Given the speed with which businesses must move to stay competitive in today’s digital landscape, it is almost a given that 99% of organizations sped up their IT operations over the last three years. ESG found that, on average, businesses today are forced to deploy both applications and infrastructure 56% faster than three years ago.

The rise of microservices, containers, and serverless architectures and the growing use of containers raise complexities that make it challenging to achieve end-to-end observability.

Such environments consist of thousands of microservices with up to billions of dependencies that can change in milliseconds, generating overwhelming volumes of data.

Data observability tools have become vital components of the modern data stack. But how fast they’re coming to market is astonishing. Observability data volumes are exploding, and the tools to move and manipulate this data are multiplying at unanticipated speeds. And ironically, the more data and the more tools, ESG found, the harder it is to find, understand and trust your data.

The data also spoke about the five biggest challenges companies face when trying to accelerate operations:

1. We’re building not just more, but different types of applications – This has been the biggest hurdle to faster operations, with the largest plurality of respondents calling it their top challenge.
2. Coordinating teams takes time – Primarily between IT and application development or other line-of-business teams, but also generally. Nearly 3 in 10 named this as a top roadblock.
3. We’re suffering from a lack of visibility – This is especially a problem when trying to understand how the application environment maps to our underlying infrastructure environments.
4. IT infrastructure is increasingly disaggregated – By leaving the pre-integrated silos of compute, networking, and storage, teams have something that is more flexible, responsive, and significantly easier to automate, but also more difficult to observe and control.
5. And (as always) security is a concern – In the shared security responsibility model, the public cloud customer is responsible for securing its data and traffic flows. This responsibility, however, is difficult to uphold if an enterprise has poor visibility into its cloud assets across complex hybrid and multi-cloud environments. Organizations often cannot effectively monitor basic issues such as who is accessing a cloud service or application; where the traffic is coming from; or misconfigured controls that, for example, inappropriately expose data storage resources to the internet.

How are companies responding to this pressure to achieve observability? Nearly half (44%) are buying more observability and monitoring tools and 65% are turning to third-party tools and services to help them achieve observability objectives.

ESG found that most organizations understand that having a “green dashboard” is not a sufficient sign that things are going well with their computing environments. So although 17% of businesses remain focused on green metrics, the remainder map their observability outcomes to actual business results.

Specifically, nearly half are more customer-focused, relying on metrics related to either customer satisfaction (37%) or Net Promoter Scores (12%). The remaining 34% measured revenue increases (or decreases) to judge how well their environments were performing.

But the proliferation of observability tools does more than simply offer businesses choices. It muddies the waters. Because the more tools are being used to observe cloud-native environments, the more complex observability becomes.

Indeed, the continued complexity of “tool sprawl” is very real. Given the pace that tools are entering the market, there’s no end in sight. Two-thirds of businesses today use more than 10 different observability tools. The average number of observability tools is a whopping dozen per organization.

This impairs the ability of IT teams to maintain and troubleshoot internally developed applications. More than half of organizations identify the correlation of metrics data from multiple tools as either “burdensome” (28%) or “extremely burdensome” (26%). All-in, over half feel encumbered.

Likewise, all these tools create a corresponding increase of alerts. This makes identifying the valuable observability data from the rest either a burdensome (36%) or extremely burdensome (22%) task—for a total of 58% of respondents feeling some strain.

A prime reason to develop apps using a cloud-native environment is for the flexibility to build and run them anywhere at scale, particularly when using containers in a microservices-based architecture.

But operating at scale is always challenging, even for more experienced and sophisticated businesses. Scaling reliably is a difficult process to automate, too. Organizations struggle to gain insight into cloud-native applications.

ESG found some of the most common challenges in deploying observability solutions are:

• The scalability and reliability of observability solutions
• Cultural resistance from other teams in the organization
• Instrumenting applications and infrastructure for observability too time consuming

ESG found some of the most common challenges in supporting observability solutions are:

• Scalability and reliability of solutions
• Lack of visibility into cloud-native, container- based application environments
• Prioritizing severity of issues/alerts when presented

Chronosphere’s purpose-built, cloud-native observability platform tackles the observability problem end-to-end in a way legacy solutions cannot.