Observability is naturally migrating toward open source

The early days of monitoring followed a pretty straightforward concept: At a set interval, IT teams checked to make sure a system was up and running. If it was, terrific; and if not, they would send up an error message. As software architectures evolved in the early 2010s with microservices, containers, and the cloud, the idea of knowing exactly what to look for became much more complicated. At this point, if something went wrong, the data always wasn’t available, making it frustratingly difficult to figure out what happened.

In the past five years or so, software engineers have begun to see observability as something much more nuanced than simple monitoring, and the tools that practitioners are using now need to work together to survey a much more complicated tapestry of programs, networks and solutions. Engineers and developers want to get the most value out of the data they have been collecting, and they are starting to realize that being locked into a single tool or architecture has been holding them back.

The quest for unlocking data in newer and more valuable ways has led to an explosion of tools for observability. More mature and forward-thinking companies recognize that if a platform can’t truly function across multiple verticals, it will significantly hamper a business’s potential growth and even viability. Data is more valuable than ever, and next-generation companies realize open source is the best path to extracting value from data.

A recent survey from OpenLogic by Perforce and the Open Source Initiative (OSI) shows just how deeply embedded open source software has become. Eight out of 10 IT managers say their company employs open source software, with nearly 42% noting a “significant increase” in the past year. One of the fastest-growing areas is cloud-native open source software, particularly observability projects such as OpenTelemetry, Jaeger, and Prometheus.

There are three big reasons that the evolution of observability, in particular, lends itself naturally to an open source approach. The first is the flexibility and extensibility offered by those tools. The power of observability comes from integrating data from multiple sources and sending that data to multiple destinations. Closed platforms limit the type of integrations that are possible, but with open source, if a particular integration doesn’t exist, it can be built. There’s also a thriving open source community available to support those new builds and provide feedback.

The second reason for the growing momentum toward open source solutions is the recognition of the importance of aligning incentives with value. Software vendors often see little incentive to support open protocols. In fact, for many, their main incentive is to have customers send them all the data so they can analyze it with their product. With open standards, on the other hand, users are free to handle the data in the way that makes the most sense to them. There’s no need for unnecessary analysis: people can choose to collect and send the data that fits their use case most effectively and is most useful for their organization. Because there’s no real monetary gain by a single party when using an open source approach, organizations can realize significant cost savings. That improved balance between incentives and value results in very little wasted time, effort and data.

That sense of control is the third reason companies are more open to using open source observability tools. Take the example of Prometheus, a popular open source monitoring solution. In previous iterations, Prometheus allowed companies to retrieve, search and visualize metrics in a standardized format. As more companies have adopted that open source standard, those metrics can paint a picture about their users — such as how much of a product they are using or how much they should be billed. Now with this open standards data, the data is free to be used in the places where it’s most valuable.

Observability is too complex for a single solution, and most projects and technologies need to interoperate to ensure organizations can build and maintain robust observability pipelines. Open source tools and standards are essential to meeting today’s observability challenges. Those businesses that make use of them will be the ones that get the most value from their data, while those stuck with locked-in platforms will fall behind.