Fluent Bit: Logs, OpenMetrics, and OpenTelemetry all-in-one

The following post is based upon a presentation at KubeCon Europe 2022. To view the entire presentation, scroll down to the embedded video.

One of the core components of observability is moving data. Gathering mountains of logs, metrics, and traces from a wide variety of systems and routing the data to a host of other systems where they can be analyzed and stored.

This was always a challenge, but as IT becomes more dynamic, distributed, ephemeral, and ubiquitous, more and more data is coming from more and more places. The challenge is complex, but solving this problem is mission critical. If you don’t have immediate insight into your systems’ performance and don’t have near real-time observability, your operations could suffer.

Fluent Solves the problem

The Fluent ecosystem was created to meet that challenge. Both Fluentd and Fluent Bit are Cloud Native Computing Foundation (CNCF) graduated projects originally created to aggregate and forward logs. They can complement each other or be used as standalone solutions

Fluentd and Fluent Bit were never intended to be drop-in and replace components of your infrastructure. Your infrastructure is your business. The Fluent ecosystem is open source and vendor agnostic. Do you want to send data to Splunk? Prometheus? S3? All of those? Fine. The Fluent ecosystem will do that for you. Do you use vanilla Kubernetes or OpenShift? Fluent doesn’t care.

Fluent Bit was originally created to process logs, and in addition to allowing you to gather log data from anywhere and send them to anywhere, you can also enrich or even redact that data before sending them to their destinations. The most common use case for enrichment is Kubernetes logs. The Kubernetes filter in Fluent Bit will automatically enrich the container logs with

• Pod Name
• Namespace
• Container Name
• Container ID

You can also query the Kubernetes API Server to obtain extra metadata for the pod such as pod ID, labels, and annotations. These enriched logs make debugging and troubleshooting much, much faster.

Conversely, there are situations where you may need to remove some data from the logs before sending them to a destination for analysis. For example, personally identifiable information (PII) could be redacted to comply with privacy laws.

Another common use case is reducing the data array. We often see folks who are gathering petabytes and petabytes of data who might not want to pay for ingesting unnecessary data into analytics applications or who may want to send it to a less expensive, less used data storage.

A while back, we noticed that a lot of folks were gathering logs and then extracting metrics from those logs. Often they were writing gigantic Lua scripts, extracting all these small decimal points from logs to do all sorts of additions, and then trying to hack it together with Node exporter or something else and eventually get into Prometheus, which has become the standard with OpenMetrics providing the compatibility layer.

Following our Fluent ecosystem philosophy, last year we began working to enable Fluent Bit to support metrics. We added counters, gauges, histograms, and summaries. We created libraries called CMetrics; it’s in the project so you can go and look at it. And then we made sure that all these things were all exportable into various formats.

Metrics also are unique in Fluent Bit in that metrics don’t go through the same pipelines that logs go through. So metrics are almost treated as an independent type.

OpenTelemtry is coming like a giant wave. So this year we have spent a lot of effort to ensure that Fluent Bit can ride that wave and be compatible. The first step we took was to add support for OpenTelemetry metrics and make sure that if you’re using the OpenTelemetry metrics SDK, we’ll be able to collect that data and send it out. Currently, we are limited to HTTP input and outputs, but as we’ll see we have plans for extending that.

Traces are a focus of the Fluent Bit roadmap for 2022.

As with logs and metrics, we treat traces as another independent data source with its own pipeline. The challenge, then, is how to make some really meaningful correlations or interactions between these three types of data. With logs, for instance, we have the capability of string processing. You can write SQL today on top of your logs. So you could take your NGINX logs, and write a query that would group them by error code.

Similarly, we’re trying to bring some of that logic over to traces and to metrics. The goal is to have this available in Q4 of this year. That is also the timeline for extending the current OpenTelemetry input/output options.

To continue expanding your Fluent Bit knowledge, check out Fluent Bit Academy. It’s filled with on-demand videos guiding you through all things Fluent Bit— best practices and how-to’s on advanced processing rules, routing to multiple destinations, and much more. Here’s a sample of what you can find there:

• Getting Started with Fluent Bit and OpenSearch
• Getting Started with Fluent Bit and OpenTelemetry
• Fluent Bit for Windows

With Chronosphere’s acquisition of Calyptia in 2024, Chronosphere became the primary corporate sponsor of Fluent Bit. Eduardo Silva — the original creator of Fluent Bit and co-founder of Calyptia — leads a team of Chronosphere engineers dedicated full-time to the project, ensuring its continuous development and improvement.

Fluent Bit is a graduated project of the Cloud Native Computing Foundation (CNCF) under the umbrella of Fluentd, alongside other foundational technologies such as Kubernetes and Prometheus. Chronosphere is also a silver-level sponsor of the CNCF.