Fluent Bit and Beats: Two approaches to a task

A stylized logo with a green circular background featuring a blue and black design in the center, set against a gradient backdrop. Abstract white dotted lines on the left side give it a fluent bit aesthetic.
ACF Image Blog

Fluent Bit and Beats data shippers from Elastic take dramatically different approaches to collecting and routing telemetry data. Here’s an overview of how they are alike, and how they are very, very different.

Anurag Gupta
Anurag Gupta | Field Architect | Chronosphere

Anurag Gupta is a Fluent Bit maintainer and co-founder of Calyptia, which was acquired by Chronosphere in 2024.

9 MINS READ

When users are building an observability or security strategy one of the first questions that comes up is how they can collect information (logs, metrics, traces, events) from their systems and route this information to a backend for analysis.

Two popular tools for collecting this type of information are Beats and Fluent Bit. In this guide, we will provide an overview of the two, explain their different methodologies for collecting data and the impact of that difference, and demonstrate how to migrate from Beats to Fluent Bit.

Two circular logos sit side by side. The left one features a stylized hummingbird on a blue-green gradient background. The right one, embodying the beats vs fluent bit theme, displays a stylized letter "B" in blue and black on a white background.

What are Beats?

Beats are lightweight data shippers from Elastic. They are actually a family of individual data shippers that each perform a specific task such as file collection, windows collection, or metric collection. A short list of Beats is included below:

Beat Name Description and Use
Filebeat Used to collect log data from files and send them to a centralized location.
Metricbeat Used to collect system and application metrics, such as CPU usage, memory usage, and network traffic, and send them to a centralized location for analysis.
Packetbeat Used to capture and analyze network data, such as HTTP requests and responses, DNS queries, and MySQL queries.
Winlogbeat Used to collect Windows event logs and send them to a centralized location.
Auditbeat Used to collect audit data on Linux systems, such as process and file system activity.
Heartbeat Used to collect service availability data with active probing

 

Beats follow a methodology of collecting information from a variety of sources and routing it to Logstash — Elastic’s telemetry pipeline product — where the data collected is formatted into a log-friendly format such as JSON for easy transformation and compatibility with Elasticsearch.

What is Fluent Bit?

Fluent Bit is a fast, lightweight, and highly scalable log, metric, and trace processor and forwarder. It is a vendor-neutral and an Apache 2.0 open source project within the cloud native ecosystem.

Fluent Bit has been deployed over 10 billion times and seamlessly integrates with other CNCF and open source ecosystems, such as Prometheus and OpenTelemetry, as well as closed source and proprietary systems such as Elasticsearch and Splunk.

What are the differences between Beats and Fluent Bit?

The core functionality of Beats and Fluent Bit is the same: collect various types of data and deliver them to a destination. However, the two differ significantly in their approach to that task.

Each Beats shipper is engineered for a specific type of data collection — Filebeat for logs, Metricbeat for metrics, etc. Fluent Bit follows a single-agent approach, handling multiple types of data. As a result, if you need to gather more than one type of data you would need to deploy multiple Beats shippers compared to deploying a single Fluent Bit agent.

Beats and Fluent Bit also differ significantly in where they can send the data they collect. As an Elastic product, Beats only support Elasticsearch as an end destination. Fluent Bit takes a vendor-neutral approach and supports any number of destinations.

The table below lays out some common scenarios users encounter when dealing with telemetry data and the Beats and Fluent Bit approaches.

Scenario Beats Approach Fluent Bit Approach
Collecting logs and metrics from a Linux Machine Install Filebeat and Metricbeat, manage two configuration files, and start two processes Add two inputs in a single Fluent Bit configuration process, start a single process
Sending data to Elasticsearch, Kafka, and other end destinations Not supported – Beats only support Elasticsearch as an end destination Supported with multiple output plugins
Process data midstream Route data to Logstash or Elastic Ingest node Process using Lua Scripting, Filters, OR route data to Logstash, Elastic ingest node, or Data Prepper

 

Support for Prometheus and OpenTelemetry metrics

Another notable difference between Fluent Bit and Beats is their approach to metrics. Within Beats, and specifically Metricbeat, all metrics are treated as a log-based metric. With Fluent Bit metrics can either be log-based metrics or Prometheus / OpenTelemetry metrics. This allows users to plug into broader CNCF ecosystems without having to install another agent or collector.

Agent configuration comparison: Logs and Metrics from a Linux machine

Suppose you want to ship logs and metrics data using Beats. You need to install Filebeat and Metricbeat separately on each server. Additionally, you then need to configure the filebeat.yml and metricbeat.yml files.

filebeat.yml
filebeat.inputs:
- type: log
      Paths:
    - /var/log/httpd/access_log
metricbeat.yml
- module: system
  metricsets:
    - cpu
    - memory
    - network
    - process_summary
    - socket_summary
    - diskio
    - filesystem
  period: 10s
  processors:
    - add_host_metadata: ~
    - add_cloud_metadata: ~

You’ll also need to add the output elasticsearch to each configuration file further to ship data. We would need to ensure that the output is kept in sync to make sure both inputs flow to the end destination.

In comparison, for Fluent Bit, you only need to install the agent and configure one file for shipping logs and metrics.

Fluentbit.conf
[INPUT]
    Name    tail
    Path    /var/log/httpd/access_log
    Tag     apache.access
    DB      /var/log/flb_apache.db

[INPUT]
    Name    cpu
    Tag     cpu

[INPUT]
    Name    mem
    Tag     memory

Sending data to Fluent Bit from Beats

In some cases where Fluent Bit may not have the required support for a particular collector, a user may make use of the new in_elasticsearch plugin from Fluent Bit. This allows Fluent Bit to listen on a port and receive incoming payloads sent to Elasticsearch or HTTP bulk API.

Configuring Beats

For example in a beats configuration, you can set the output yaml to the following configuration. Fluent Bit returns 8.0 as the version, which may require you to set the allow_older_versions depending on the oss metricbeat version you are using. Lastly, the current integration does not support modules or index lifecycle management (ILM), which will need to be turned off.

output.elasticsearch:
  hosts: ["Fluent Bit host:PORT"]
  allow_older_versions: true
  ilm: false

Configuring Fluent Bit

In Fluent Bit you will need to configure the in_elasticsearch plugin and then populate the output section with any destination you desire. This can include Data Prepper, OpenSearch, Elasticsearch, or any other of 50+ destinations supported by Fluent Bit. In our example, we will use the standard output plugin to see logs in the terminal output:

[INPUT]
    name elasticsearch
    listen 0.0.0.0
    port 9200
    buffer_max_size 20M
    buffer_chunk_size 5M

[OUTPUT]
    name stdout
    match *

Conclusion

Both Fluent Bit and the Beats data shippers are used for collecting and routing logs, metrics, traces, and event data from systems and routing it to a backend for storage and analysis. However, the two take dramatically different approaches to that task. Built by Elastic, Beats shippers are very efficient at routing that data to Elasticsearch, but are incapable of connecting to other destinations.

Fluent Bit, on the other hand, takes a vendor-neutral approach and can route the data it collects to any number of destinations. In addition, Fluent Bit is a single agent capable of handling a variety of different data types, while Beats shippers are specialized and require users to deploy multiple agents to collect multiple data types. As we saw above, Fluent Bit can even integrate with Beats.

Chronosphere Telemetry Pipeline, from the creators of Fluent Bit and Calyptia, simplifies the management of complex telemetry pipelines with Fluent Bit at its core. It provides a low-code and UI approach to connecting your existing agents, open source protocols, SaaS platforms, and storage and analytic backends. And like Fluent Bit, it is vendor neutral, letting you easily route data from any source to any destination.

About Fluent Bit and Chronosphere

With Chronosphere’s acquisition of Calyptia in 2024, Chronosphere became the primary corporate sponsor of Fluent Bit. Eduardo Silva — the original creator of Fluent Bit and co-founder of Calyptia — leads a team of Chronosphere engineers dedicated full-time to the project, ensuring its continuous development and improvement.

Fluent Bit is a graduated project of the Cloud Native Computing Foundation (CNCF) under the umbrella of Fluentd, alongside other foundational technologies such as Kubernetes and Prometheus. Chronosphere is also a silver-level sponsor of the CNCF.

Share This:
Table Of Contents

Ready to see it in action?

Request a demo for an in depth walk through of the platform!