Fluent Bit: One telemetry agent for all your data needs

The following video was recorded at KubeCon + CloudNativeCon North America in November 2023. We are grateful to the Cloud Native Computing Foundation for allowing us to post it here.

The challenges of logging

People don’t wake up one day and say “I want to do logging.” Collecting and analyzing data logs is a means to another end. For most of us, our goal is to gain insights into our system performance, and to get insights you need to collect information from those applications and systems, hence logging.

But logging brings challenges:

• Log formats differ across systems and applications
• Logs can be collected in different ways, including from file systems, APIs, and network endpoints
• The sheer volume of logs produced can be difficult to manage

To gain insights from our logs, we need to gather them from various sources, convert them into structured data, and store them for analysis. We also need to do this at scale.

Fluent Bit is a unified solution

This is where Fluent Bit comes in. Fluent Bit is a unified solution for collecting, processing, and routing telemetry data. In addition to logs, Fluent Bit can also handle metrics and traces. It has a fully pluggable architecture that allows us to connect different sources with different destinations and perform different types of processing to the data in flight. Fluent Bit provides more than 100 connectors built in the same binary, and it’s fully cross-platform for Linux, Windows, MacOS, and BSD.

One agent for logs, metrics, and traces

Because Fluent Bit supports metrics (Prometheus) and traces (OpenTelemetry) as well as logs, it also alleviates “agent fatigue ”— which occurs from needing to install multiple agents to collect different types of telemetry data that in turn needs to be delivered to multiple backends. Often each backend platform comes with its own collection agent. Because Fluent Bit is vendor-neutral and supports logs, metrics, and traces, you only need to install and manage one agent.

Ideal for containers

Fluent Bit is ideal for running in containerized environments like Kubernetes. It can even automatically enrich Kubelet logs with additional metadata from the Kubernetes API server, providing valuable information that would normally be unavailable to downstream systems.

Architecturally, Fluent Bit also provides both in-memory and file-based buffering to help ensure that no data is lost when the volume of data being generated by your sources exceeds your endpoints’ ability to consume it.

Fluent Bit v2.2

The recently released Fluent Bit v2.2 includes new features and enhancements including:

• A new sysinfo filter
• cgroups v2 support for collecting metrics from Docker
• YAML support in configuration files
• New collector support for metrics from macOS and Windows clients
• New native integration for Loki
• Collect Kubernetes cluster events with support for namespaces
• A 6X performance improvement in certain complex pipeline configurations

To learn more about Fluent Bit, visit the project website or visit Fluent Bit Academy where you will find hours of on-demand training videos covering best practices and how-to’s on advanced processing, routing, and all things Fluent Bit. Here’s a sample of what you can find there:

• Getting Started with Fluent Bit and OpenSearch
• Getting Started with Fluent Bit and OpenTelemetry
• Parsing 101 with Fluent Bit
• Advanced Routing with Fluent Bit v3

We also invite you to join the vibrant Fluent community. Visit the project’s GitHub repository to learn how to become a contributor. Or join the Fluent Slack where you will find thousands of fellow Fluent Bit and Fluentd users helping one another with issues and discussing the projects’ roadmaps.

With Chronosphere’s acquisition of Calyptia in 2024, Chronosphere became the primary corporate sponsor of Fluent Bit. Eduardo Silva — the original creator of Fluent Bit and co-founder of Calyptia — leads a team of Chronosphere engineers dedicated full-time to the project, ensuring its continuous development and improvement.

Fluent Bit is a graduated project of the Cloud Native Computing Foundation (CNCF) under the umbrella of Fluentd, alongside other foundational technologies such as Kubernetes and Prometheus. Chronosphere is also a silver-level sponsor of the CNCF.