Understanding the Forward Protocol for Fluentd and Fluent Bit

Originally created at Treasure Data more than a decade ago, Fluentd is a widely adopted open source data collection project. Both Fluentd and Fluent Bit, a related project specifically created for containerized environments, utilize Forward Protocol for transporting log messages. In this post, we’ll explain Forward Protocol and its uses and provide an example of using Fluentd and Fluent Bit together to collect and route logs to a MongoDB database for storage.

Forward Protocol is a network protocol used by Fluentd to transport log messages between nodes. It is a binary protocol that is designed to be efficient and reliable. It uses TCP to transport messages and UDP for the heartbeat to check the status of servers.

It is a lightweight and efficient protocol that allows for the transmission of logs across different nodes or systems in real time. The protocol also supports buffering and retransmission of messages in case of network failures, ensuring that log data is not lost.

Fluentd Forward Client and Server are two components of the Fluentd logging system that work together to send and receive log data between different sources and destinations. It uses message pack arrays to communicate and also has options for authentication and authorization to ensure only authorized entities have access to send & receive logs. Read more about the forward protocol.

Forward protocol offers the following benefits:

• It supports the compression of log data to reduce the amount of data sent over the network, which can improve performance, especially in situations where bandwidth is a limiting factor.
• It includes a response mechanism whereby if any message fails to be delivered to the fluent forward server, it will be retried by the fluent forward client.
• The protocol has specifications for backward compatibility that ensure older versions of Fluent Bit and Fluentd are able to communicate with each other and upgrades are smoother.

Apart from Fluentd and Fluent Bit, there’s also a Docker log driver that uses Forward Protocol to send container logs to the Fluentd collector.

The OpenTelemetry Collector also supports receiving logs using Forward Protocol.

Both Fluentd and Fluent Bit are popular logging solutions in the cloud-native ecosystem. They are designed to handle high volumes of logs and provide reliable log collection and forwarding capabilities. Fluent Bit is lightweight and more suitable for edge computing and IoT use cases.

In this section, we’ll take a closer look at the differences between the two tools and understand a use case when you’d want to use both of them together.

Both Fluent Bit and Fluentd can be used as forwarders or aggregators and can be used together or as a standalone solution. One use case for using Fluent Bit and Fluentd together is by using Fluent Bit to collect logs from containerized applications running in a Kubernetes cluster. Because Fluent Bit has a very small footprint, it can be deployed on every node. Meanwhile, Fluentd can be used for collecting logs from various sources outside of Kubernetes, such as servers, databases, and network devices.

Ultimately, the choice between Fluentd and Fluent Bit depends on the specific needs and requirements of the use case at hand.

In the next section, we’ll explore how we can use Forward Protocol to push data from Fluent Bit to Fluentd.

To understand how Forward Protocol works, we’ll set up instances of both Fluent Bit and Fluentd. We’ll collect CPU logs using Fluent Bit, and, using Forward Protocol, we’ll send them to Fluentd. From there, we will push the logs to MongoDB Atlas.

MongoDB Atlas configuration

MongoDB Atlas is a cloud-based database service that allows users to easily deploy, manage, and scale MongoDB databases. It offers features such as automatic backups, monitoring, and security, making it a convenient and reliable option for managing data in the cloud. Hence, we’ll be pushing our logs to MongoDB from Fluentd.

In order to do that, we need to do the following:

• Login to your MongoDB Atlas account
• Deploy a new Cluster
• Create a new Database – by navigating to the collections tab.
• Provide a database and collection name
• Once it is created, click on the connect button and fetch the connection string

Apart from this, you might also have to do the following:

• Create a new user and provide `Read and write to any database` permission.
• Under the Network section, create a policy to allow connections from all IP addresses. Note: If you’re using this in production, please whitelist only those IPs that you’ll use to access the database

Fluentd configuration

The first step is to configure Fluentd to receive input from a forward source. After you install Fluentd, you need to update the configuration file with the following:

<source>
  type forward
  bind 0.0.0.0
  port 24224
</source>

<match fluent_bit>
  type stdout
</match>

<match fluent_bit>
  @type mongo
  database fluentd
  collection fluentdforward
  connection_string "mongodb+srv://fluentduser:[email protected]/test?retryWrites=true&w=majority"
</match>

In the above configuration, we are defining the source type to be forward and providing a bind address and port. We’re also providing a match filter which is `fluent_bit`, so any log it finds with this tag will be consumed. The input logs will be sent to MongoDB Atlas for which we’ve provided the database, collection, and connection_string.

After this, all you need to do is start the Fluentd service if it is not running already. It will not show any output at the moment since we have not yet configured Fluent Bit to forward the logs.

Fluent Bit Configuration

On the Fluent Bit side, we need to configure the INPUT and OUTPUT plugins.

INPUT

[INPUT]
    Name cpu
    Tag fluent_bit

[INPUT]
    Name kmsg
    Tag fluent_bit

[INPUT]
    Name systemd
    Tag fluent_bit

With this, we are collecting the CPU, kernel, and systemd logs and applying a `fluent_bit` tag to them.

OUTPUT

[OUTPUT]
    Name forward
    Match *
    Host 127.0.0.1
    Port 24224

For output, we’re using a forward output plugin that routes the logs to the specified host and port.

Save the configuration and restart the Fluent Bit service. If everything is correct, you’ll see the logs being streamed by Fluentd. Navigate to your MongoDB Atlas UI and refresh the collection, you should be able to see the logs as shown below.

This way we are able to make use of the forward plugin and share logs between Fluent Bit and FluentD. You can use Forward Protocol with other products that support it to gather logs from different sources and push them to different tools.

To learn more about Fluent Bit and its powerful data processing and routing capabilities, check out Fluent Bit Academy. It’s filled with on-demand videos guiding you through all things Fluent Bit— best practices and how-to’s on advanced processing rules, routing to multiple destinations, and much more. Here’s a sample of what you can find there:

• Getting Started with Fluent Bit and OpenSearch
• Getting Started with Fluent Bit and OpenTelemetry
• Fluent Bit for Windows

With Chronosphere’s acquisition of Calyptia in 2024, Chronosphere became the primary corporate sponsor of Fluent Bit. Eduardo Silva — the original creator of Fluent Bit and co-founder of Calyptia — leads a team of Chronosphere engineers dedicated full-time to the project, ensuring its continuous development and improvement.

Fluent Bit is a graduated project of the Cloud Native Computing Foundation (CNCF) under the umbrella of Fluentd, alongside other foundational technologies such as Kubernetes and Prometheus. Chronosphere is also a silver-level sponsor of the CNCF.