How Telemetry Pipelines help control log data growth and infrastructure complexity

A person is working at a computer with code on the screen, complemented by a green video camera icon graphic. Beside them, Telemetry Pipelines illustrate infrastructure complexity as new log data growth challenges continue to emerge.
ACF Image Blog

Check out how Telemetry Pipelines can help teams control log data growth while lowering observability costs.

Sophie Kohler | Content Writer | Chronosphere

Sophie Kohler is a Content Writer at Chronosphere where she writes blogs as well as creates videos and other educational content for a business-to-business audience. In her free-time, you can find her at hot yoga, working on creative writing or playing a game of pool.

6 MINS READ

According to a recent study by Chronosphere, log data grew over 250% in the past year alone. With 84% of organizations using or evaluating Kubernetes, this trend shows no sign of slowing down. So, how can teams that are overwhelmed by the explosion of log data alleviate the burden of telemetry collection and routing?

In our latest Chronologues episode, hear Chronosphere’s Sales Engineer, Rosti Spitchka, our Head of Developer Marketing & OSS Community, Carolyn King, and our Product Manager Eddie Dinel, talk about how organizations can control log data growth while lowering observability costs. Catch a transcript of the episode below, and the full video at the end of this blog!  

Combatting high logging costs

Sophie: Did you know that log data has grown over 250 percent year over year? Teams need to route telemetry data to different backend destinations, which means managing configurations, monitoring health and performance, and then rolling out updates across disjointed tools. Ah, management burden. We hate to see it.

Now, this creates challenges for security and observability teams: Like high logging costs, complexity, data in different formats, and vendor lock-in. That’s where the power of a telemetry pipeline comes into play. Let’s kick things off with how teams can manage high logging costs. Rosti, can you help us out?

Capture the right data for efficiency

Rosti: Hey Sophie, why not?  Reducing log data volume and cost is one of the primary outcomes of using a telemetry pipeline. Telemetry volumes explode as organizations embrace containerized and microservices architectures.

A natural consequence of this growth is a higher observability bill, because cost scales at the same rate as the log data volume itself and log data is essential to observability and security. But large portions of the log data that you create is neither useful nor used. Telemetry pipelines help you capture the right data in the optimal format to drive efficiency. As such, you could right size your log data footprint before you pay to transport and analyze that data.

Video Thumbnail

Sophie: What about the complexity when it comes to collecting and routing data?

Managing data complexity

Carolyn: Thanks, Sophie. That’s a great question. Today, we see observability and security teams support more data sources and destinations than ever before. And this has led to increased infrastructure complexity and a number of new challenges for these teams. First is maintenance overhead. This includes ongoing updates, patches, configuration changes, which only become more difficult as infrastructure scales. Second, we see teams having to manage different proprietary agents running on the same infrastructure, which can be a massive drain on application resources. And finally, teams see a lot of duplicate data being routed to multiple backends. The good news is that a telemetry pipeline can help solve these problems by providing a central place to manage both data collection and data routing. 

 

Video Thumbnail

Sophie: So, we’ve talked about reducing costs and complexity. Now, what about data that’s in different formats? Eddie, do your thing.

Collecting data from different sources

Eddie: I gotcha. There are a whole bunch of different places that logs come from. It’s not just telemetry. It’s not just observability. Different sources send out data in different formats. And so, a security team is going to work with data from many different sources, like Palo Alto Networks or CrowdStrike Endpoint Detection, or what have you.

During an investigation, you’re going to need to run queries on data from other sources. You’re going to be going through these sources one by one figuring out what’s going on there, or you’re trying to craft the perfect query that pulls all of this together.

And, both of those are really time consuming. What a telemetry pipeline allows you to do is to solve the Tower of Babel problem. It allows you to bring together and normalize data from a whole bunch of different sources. So you can take data from a bunch of different places, and normalize your timestamps, or standardize your IP address formats, or create a unified taxonomy for different data types, or any of those things.

Sophie: Ah, so you can enforce a schema across all of your different data so that when you’re actually in an incident, you can solve the problem quickly.

Video Thumbnail

Migrating to the right log management platform

Sophie: Hmm, what are we missing? Riley! Can you tell us about how to successfully migrate to a platform that fits your needs?

Riley: Can do, Sophie. Observability and security teams might want to migrate to a new log management or SIEM platform for a few different reasons. They might need a platform that can better manage today’s log data volumes, they might want one that can enable more proactive cybersecurity practices, or they might even want to consolidate their logging backends.

In the past, migration was really complex for a few different reasons. It required teams to reinstall and reconfigure their log collection agents, it required them to re-imagine their logs to align with new schema requirements, and it also meant that teams might lose access to historical data during the migration process.

How Telemetry Pipelines help

Riley: Telemetry pipelines solve these challenges on a few different fronts. So first, they can collect data from any source and push them to any destination. What that means is that you can route data to your new logging backend without reinstalling a new log collection agent. The second way it helps solve these challenges is it allows you to reshape data in flight so you can meet your new schema requirements without re-instructing logs upstream.

And the third way is that it allows you to route a copy of all your data to low-cost object storage such as Amazon S3. If you ever need access to this data to investigate a breach or to support an audit or whatever else, you can rehydrate the data back into your log management or your SIEM platform.

Sophie: In a containerized microservices world, telemetry pipelines have emerged as a crucial solution to increasingly complex systems. And telemetry pipelines can access information about your data sources that may not be available to you downstream.

If you’re interested in telemetry pipelines, check out some of the resources that we’ve linked in the description below. We’d love to hear in the comments your take on telemetry pipelines, and what you’d like to hear about in the next episode. See ya! 

Video Thumbnail

The Buyer’s Guide to Telemetry Pipelines

Share This: