In today’s cloud native landscape, the explosion of log data generated by microservices and containerized applications has created significant challenges for organizations. High costs associated with storing and managing massive volumes of log data, combined with time-consuming and manual log operations, are draining resources and reducing engineering productivity. These issues are exacerbated by the scale and complexity of modern, distributed environments, making existing log management approaches insufficient.
Today, we are excited to announce two major updates aimed at addressing these challenges: the new Chronosphere Telemetry Pipeline product from the creators of Fluent Bit and Calyptia, and a new control feature built into Chronosphere Logs, as part of our Observability platform. These additions help customers significantly reduce log volumes, simplify their log operations, and gain greater control over their log data.
In this blog, we will dive into the key features and benefits of these new offerings, and explain how they specifically address the current market challenges of dealing with logs, while simultaneously improving overall observability.
Log management challenges in today's market
Managing log data has become increasingly challenging. Organizations are grappling with two primary issues: the high costs associated with storing and managing log data, and the time-consuming, manual operations required to handle it effectively. These challenges not only strain resources but also hinder operational efficiency.
High costs of log data
Storing and managing massive volumes of log data can be prohibitively expensive. As organizations generate more data than ever before, in fact, Gartner estimates the average enterprise is producing more than 10TBs per day of logging data, with many firms hitting the 100TB range. As volumes go up, the costs associated with storing and processing this data often outweigh the value derived from it. This creates a dilemma for administrators who must decide which logs to keep and which to completely discard. Discarding logs can lead to blind spots in observability and security, which compromises the ability to detect issues and threats promptly. Conversely, retaining all logs results in exorbitant costs that can significantly impact the bottom line. The result is that organizations often make unwanted trade-offs between cost and visibility, which undermines their ability to maintain comprehensive observability and security.
Time consuming and manual log operations
Managing logs manually is a labor-intensive and time-consuming process. Collecting, transforming, and routing logs from various sources requires extensive configuration and ongoing maintenance. This not only consumes valuable time and resources but also introduces the risk of errors and inefficiencies. The complexity of managing different log formats and sources adds another layer of difficulty, making it a significant operational burden for many organizations. Without streamlined processes, teams are bogged down with arduous tasks, which detract from their ability to focus on strategic initiatives and innovation.
Chronosphere Telemetry Pipeline: Simplified, end-to-end observability pipeline
In response to the challenges of high log data costs and complex log operations, we are excited to introduce Chronosphere Telemetry Pipeline from the creators of Fluent Bit and Calyptia. This new product is designed to streamline your log management processes, providing an end-to-end telemetry pipeline that simplifies data collection, transformation, and routing. Let’s dive into the key features and benefits that make Telemetry Pipeline an essential tool for modern organizations.
Collect and route logs from any source to any destination
Telemetry Pipeline is built to seamlessly integrate with a wide array of data sources and destinations. Whether you’re dealing with application logs, system logs, or network logs, our pipeline can ingest data from any source and route it to any destination. This flexibility ensures that your log data is always available where it’s needed, using the formats and protocols you need, so you are never locked-in to one vendor. The ability to manage log data across diverse environments in a centralized way helps maintain comprehensive observability and operational efficiency.
Process logs in flight with turnkey transformations
With built-in, turnkey transformations, Telemetry Pipeline allows you to process logs at the collection point, before they are routed. This includes actions such as reducing data volumes by filtering out unnecessary information, redacting sensitive data for security compliance, enriching logs with contextual data to enhance their value. You can also easily create your own custom processing rules using our GPT-powered regex authoring tool. By processing logs in-flight, you ensure that only the most relevant data is stored and analyzed, reducing volumes and associated costs by at least 30%. This approach not only cuts costs but also improves the effectiveness of your observability practices by adding valuable context to your logs.
Automated pipeline operations
Telemetry Pipeline simplifies the management of log pipelines with a suite of automated operations. It includes automatic load balancing to ensure even distribution of data, which prevents bottlenecks and optimizes performance. It also features Automatic Retry and Persistent Storage to handle temporary issues, Automatic Healing to detect and resolve problems without manual intervention, and Automatic Monitoring to keep track of performance. Scalability is also made easy with one-command scaling, allowing you to adjust capacity effortlessly in response to changing data volumes. These automation features not only reduce the operational burden on your teams but also enhance system reliability by avoiding disruptions and ensuring continuous data flow.
Built for performance and efficiency
Telemetry Pipeline was built to ensure high performance and operational efficiency. Built by the creators of Fluent Bit, it leverages the proven reliability of the leading open-source log processor. Here are a few of the ways Telemetry Pipeline provides the most efficient data collection and processing available:
- Low infrastructure cost. One of the key differentiators of the Telemetry Pipeline is its low infrastructure cost. Written in C, our pipeline product requires significantly fewer resources compared to other solutions, resulting in lower operational expenses. One customer comparing Telemetry Pipeline with another leading pipeline solution found that our offering was over 20x more efficient. This efficiency is achieved through optimized data processing and reduced hardware requirements, allowing you to do more with less.
- Fleet Management: Our pipeline includes Fleet Management features, providing scalable, centralized control over your Fluent Bit agents across thousands of machines. This makes it easy to manage configurations, monitor performance, and deploy updates across your entire infrastructure, ensuring consistency and reliability at scale.
- Kubernetes-native: Telemetry Pipeline is natively integrated with Kubernetes, making it an ideal choice for modern, cloud native environments. It offers full integration with Kubernetes annotations, tolerations, and taints, as well as out-of-the-box Kubernetes Fleet Management, simplifying deployment and operation within these clusters. This native support enhances operational efficiency and significantly reduces the complexity associated with managing log data in dynamic, containerized environments.
By combining high performance, low infrastructure costs, and seamless integration with existing tools and platforms, Telemetry Pipeline provides a comprehensive solution for end-to-end log data management. This launch marks a significant step forward in our mission to help organizations overcome the cost and complexity challenges of observability, enabling them to achieve greater control, efficiency, and visibility.
Chronosphere Logs, now with control
As part of our commitment to empower customers with greater control over their telemetry data, we are expanding our Observability Platform Control Plane capabilities beyond metrics and traces with the introduction of Logs Control.
Seamless log collection
One of the standout features of this launch is the ability to seamlessly collect logs from any source. Chronosphere Logs provides built-in integrations with the most popular data sources to ensure that logs are collected reliably and efficiently. This seamless log collection eliminates the need for building multiple disparate integrations, which simplifies the log management process and ensures that all relevant data is captured accurately, out-of-the-box.
Log reduction and transformation
Storing huge volumes of log data can quickly become cost prohibitive. Chronosphere Logs now includes advanced log reduction capabilities designed to address this challenge. By filtering out redundant and non-essential log entries, redacting sensitive information, and deduplicating and aggregating logs, it significantly decreases noise, storage requirements, and associated costs. This reduction process also maintains the integrity and diagnostic value of the logs, ensuring that only the most relevant and valuable data is retained.
Flexible log routing
Another key feature of Control for Logs is its flexible routing capabilities. With Chronosphere Logs, organizations can now route data based on their specific needs and priorities. High-value logs that require immediate analysis can be directed to Chronosphere’s scalable and highly performant log store for real-time observability. Meanwhile, less critical logs can be routed to a cost-effective cloud object store like Amazon S3 or Google Cloud Storage for long-term retention or compliance purposes. This flexible routing ensures that log data is stored in the most appropriate and cost-effective location, which optimizes both performance and expenses. By enabling this level of control, Chronosphere Logs helps organizations balance their need for comprehensive observability with their budgetary constraints.
By combining these substantial cost-saving measures with the scalability and performance of Chronosphere Logs, we now provide a comprehensive solution that ensures both efficient and effective log observability.
Get started with Chronosphere Telemetry Pipeline today
All these logging features are now generally available in Chronosphere. You can start using them today to manage your log data more efficiently and effectively, ensuring you have access to the data you need while keeping costs under control. If you’re new to Chronosphere and would like to see how our solutions can be tailored to your specific use cases, request a customized demo today, or sign up for a free trial.