How to successfully manage logging costs and reduce complexity

With Kubernetes and microservices becoming the de facto choice of architecture as companies look to modernize, more organizations are realizing the unintended consequence – a rapid rise in complexity and observability data volumes.

As developers look to find more efficient ways to navigate their sea of observability data while cutting costs, streamlining management over large volumes of logs from different sources and destinations is a sure-fire way of achieving both of these goals, while gaining highly-performant logs. 

Chronosphere’s Head of Product Innovation Alok Bhide is joined by Chronosphere’s Field Architect and co-creator of Fluent Bit Anurag Gupta, to discuss what today’s log management challenges look like, what we’re hearing from customers, and how to solve these challenges. 

Sophie: The challenges with modern log management… where to begin? In the modern infrastructure and application landscape of Kubernetes and microservices, organizations face big challenges when it comes to collecting, processing, and managing observability data, especially when it comes to logs. When an organization goes cloud native, it means more data, but it also means more cost and complexity. 

Two main ways this happens is by manually managing logs from different sources, and storing and managing huge volumes of logs. So, how can these teams successfully combat these challenges and manage log data from different sources and destinations? Well, luckily we have some experts here that are ready to answer that question.

Sophie: Anurag, can you share the challenge of managing log operations that you’re hearing from customers? 

Anurag: Happy to talk a little bit about that challenge. So, when you look at today’s logs, they’re not just coming from servers. You’ve got microservices, you’ve got containerized applications, you’ve got SaaS, you even have applications that are ephemeral, running for just a few seconds, right, like a Lambda function.

And on top of all of these different sources of data, you also have this explosion of different formats, different types, you have different schemas, protocols, and we have to use all of these different things aligned together to try to make sense of what’s happening.

Watch the full video at the end of this blog.

Sophie: So, what are some ways that customers can solve this? 

Anurag: Well, the good part about trying to solve this logging challenge is we’re not alone. You’ve got a ton of folks who are trying to solve the same exact thing. And since logs have been around for 40 plus years, you have a ton of communities to go and plug into. You’ve got folks like OpenTelemetry, you’ve got folks like Prometheus, Fluent Bit, that have really been building this set of neutral and open standards. 

And on top of that, we’re seeing some of these tools add things like schematization with logs. So, what is made a hostname in one type of log is effectively a hostname across all of them. So, that makes things a lot easier to manage, and really starts to solve that challenge of all this logging complexity.

Sophie: Alok, what are some cost challenges you’ve been hearing when it comes to storing and managing huge volumes of logs?

Alok: So, it’s very easy for logs to get out of control in terms of volume. The primary two reasons are that modern cloud architectures are pretty complex, and during incidents or even just the architectures themselves, can generate tons and tons of infrastructure logs.

On top of that, it’s always been super easy for engineers to generate logs or emit logs from their custom applications. And that continues to be the case. And with complex architectures, it’s even further compounded. Admin teams are always stuck in this bizarre spot of trying to manage costs while also giving their engineers the value they need.

That’s a tricky thing to do because they cannot get rid of stuff that might impact the engineers.

Watch the full video at the end of this blog.

Sophie: So, it sounds like huge volumes of logs can add up to be really expensive. What are some ways that teams and organizations can better manage their log volumes to avoid high costs?

Alok: One of them that’s a pretty quick win is routing some of your data, some of your log data, to an object store. This can significantly reduce the storage price that a customer might pay, primarily because you can choose to send lower value data or lower urgency data to an object store, keep it for a cheaper price, and then the higher value data goes to the more performant logging store.

Another one is to reduce waste data. Things like punctuation, whitespace, duplicates. This can sometimes reduce up to 50% depending on what your data is like. But you’ll definitely get savings. Both of these are excellent ways to reduce your log data without rethinking your logging strategy.

Sophie: Taking back control of your log data is possible, and you can even do so while cutting on costs. Check out more resources on logs control and pipeline in the links below. And thanks for watching Chronologues.