Learn how the lightweight and efficient Fluent Forward Protocol allows for the transmission of logs across nodes or systems in real time.
Sudhanshu Prajapati is a developer advocate and software developer with an interest in observability.
On: Aug 2, 2024
Originally created at Treasure Data more than a decade ago, Fluentd is a widely adopted open source data collection project. Both Fluentd and Fluent Bit, a related project specifically created for containerized environments, utilize Forward Protocol for transporting log messages. In this post, we’ll explain Forward Protocol and its uses and provide an example of using Fluentd and Fluent Bit together to collect and route logs to a MongoDB database for storage.
Forward Protocol is a network protocol used by Fluentd to transport log messages between nodes. It is a binary protocol that is designed to be efficient and reliable. It uses TCP to transport messages and UDP for the heartbeat to check the status of servers.
It is a lightweight and efficient protocol that allows for the transmission of logs across different nodes or systems in real time. The protocol also supports buffering and retransmission of messages in case of network failures, ensuring that log data is not lost.
Fluentd Forward Client and Server are two components of the Fluentd logging system that work together to send and receive log data between different sources and destinations. It uses message pack arrays to communicate and also has options for authentication and authorization to ensure only authorized entities have access to send & receive logs. Read more about the forward protocol.
Forward protocol offers the following benefits:
Apart from Fluentd and Fluent Bit, there’s also a Docker log driver that uses Forward Protocol to send container logs to the Fluentd collector.
The OpenTelemetry Collector also supports receiving logs using Forward Protocol.
Both Fluentd and Fluent Bit are popular logging solutions in the cloud-native ecosystem. They are designed to handle high volumes of logs and provide reliable log collection and forwarding capabilities. Fluent Bit is lightweight and more suitable for edge computing and IoT use cases.
In this section, we’ll take a closer look at the differences between the two tools and understand a use case when you’d want to use both of them together.
Fluentd | Fluent Bit | |
---|---|---|
Scope | Containers / Servers | Embedded Linux / Containers / Servers |
Language | C & Ruby | C |
Memory | > 60MB | ~1MB |
Performance | Medium Performance | High Performance |
Dependencies | Built as a Ruby Gem, it requires a certain number of gems. | Zero dependencies, unless some special plugin requires them. |
Plugins | More than 1000 external plugins are available | More than 100 built-in plugins are available |
License | Apache License v2.0 | Apache License v2.0 |
Both Fluent Bit and Fluentd can be used as forwarders or aggregators and can be used together or as a standalone solution. One use case for using Fluent Bit and Fluentd together is by using Fluent Bit to collect logs from containerized applications running in a Kubernetes cluster. Because Fluent Bit has a very small footprint, it can be deployed on every node. Meanwhile, Fluentd can be used for collecting logs from various sources outside of Kubernetes, such as servers, databases, and network devices.
Ultimately, the choice between Fluentd and Fluent Bit depends on the specific needs and requirements of the use case at hand.
In the next section, we’ll explore how we can use Forward Protocol to push data from Fluent Bit to Fluentd.
To understand how Forward Protocol works, we’ll set up instances of both Fluent Bit and Fluentd. We’ll collect CPU logs using Fluent Bit, and, using Forward Protocol, we’ll send them to Fluentd. From there, we will push the logs to MongoDB Atlas.
MongoDB Atlas is a cloud-based database service that allows users to easily deploy, manage, and scale MongoDB databases. It offers features such as automatic backups, monitoring, and security, making it a convenient and reliable option for managing data in the cloud. Hence, we’ll be pushing our logs to MongoDB from Fluentd.
In order to do that, we need to do the following:
Apart from this, you might also have to do the following:
The first step is to configure Fluentd to receive input from a forward source. After you install Fluentd, you need to update the configuration file with the following:
<source>
type forward
bind 0.0.0.0
port 24224
</source>
<match fluent_bit>
type stdout
</match>
<match fluent_bit>
@type mongo
database fluentd
collection fluentdforward
connection_string "mongodb+srv://fluentduser:[email protected]/test?retryWrites=true&w=majority"
</match>
In the above configuration, we are defining the source type to be forward and providing a bind address and port. We’re also providing a match filter which is `fluent_bit`, so any log it finds with this tag will be consumed. The input logs will be sent to MongoDB Atlas for which we’ve provided the database, collection, and connection_string.
After this, all you need to do is start the Fluentd service if it is not running already. It will not show any output at the moment since we have not yet configured Fluent Bit to forward the logs.
On the Fluent Bit side, we need to configure the INPUT and OUTPUT plugins.
INPUT
[INPUT]
Name cpu
Tag fluent_bit
[INPUT]
Name kmsg
Tag fluent_bit
[INPUT]
Name systemd
Tag fluent_bit
With this, we are collecting the CPU, kernel, and systemd logs and applying a `fluent_bit` tag to them.
OUTPUT
[OUTPUT]
Name forward
Match *
Host 127.0.0.1
Port 24224
For output, we’re using a forward output plugin that routes the logs to the specified host and port.
Save the configuration and restart the Fluent Bit service. If everything is correct, you’ll see the logs being streamed by Fluentd. Navigate to your MongoDB Atlas UI and refresh the collection, you should be able to see the logs as shown below.
This way we are able to make use of the forward plugin and share logs between Fluent Bit and FluentD. You can use Forward Protocol with other products that support it to gather logs from different sources and push them to different tools.
To learn more about Fluent Bit and its powerful data processing and routing capabilities, check out Fluent Bit Academy. It’s filled with on-demand videos guiding you through all things Fluent Bit— best practices and how-to’s on advanced processing rules, routing to multiple destinations, and much more. Here’s a sample of what you can find there:
Your destination for best practices and training on all things Fluent Bit
With Chronosphere’s acquisition of Calyptia in 2024, Chronosphere became the primary corporate sponsor of Fluent Bit. Eduardo Silva — the original creator of Fluent Bit and co-founder of Calyptia — leads a team of Chronosphere engineers dedicated full-time to the project, ensuring its continuous development and improvement.
Fluent Bit is a graduated project of the Cloud Native Computing Foundation (CNCF) under the umbrella of Fluentd, alongside other foundational technologies such as Kubernetes and Prometheus. Chronosphere is also a silver-level sponsor of the CNCF.