How to collect metrics from multiple sources with Fluent Bit

The wide adoption of microservices architecture brings great advantages to devops teams, but it also makes application and infrastructure monitoring much more complex.

Different systems may generate metrics, logs, and traces in different formats, making them difficult to track and monitor. Additionally, teams need to collect data from multiple sources and aggregate them to get a complete picture of the system.

To overcome these challenges, organizations need tools that enable them to collect and aggregate data from multiple sources. This data can be used to provide insights to help identify issues and diagnose problems quickly.

Fluent Bit is one such open source tool that helps you collect and aggregate metrics, logs, and trace data from multiple sources. In this blog post, we’ll demonstrate how to collect application metrics and system metrics using Fluent Bit and push them to a Prometheus-compatible endpoint.

Fluent Bit is an open source and lightweight telemetry agent designed to collect and ship logs, metrics, and traces from various sources to different destinations. It allows you to build custom data pipelines that filter, parse, and transform data before sending it to a destination

The major components of Fluent Bit are:

• Input plugins: Collect data from various sources, including logs, metrics, and traces.
• Parsers: Extract structured data from unstructured log data, making it easier to query and analyze.
• Filters: Modify, route, or drop log data based on certain conditions, such as regex matching or metadata values.
• Routers: Determine where data should be sent based on certain rules or conditions.
• Buffers: Temporarily store log data before it is sent to its destination, helping to ensure that data is not lost if there is a network outage or destination failure.
• Output plugins: Send log data to various destinations, such as Elasticsearch, Kafka, or cloud storage services.

Together, these components create a data pipeline that collects, processes, and routes telemetry data from different sources to its ultimate destination, whether that’s a centralized logging system, a monitoring dashboard, or an external analytics tool. Read more about data pipelines in Fluent Bit to learn more about it.

Now we’ll see how we can use Fluent Bit to collect different metrics from multiple sources. To show this, we’ll configure Fluent Bit on our system to collect multiple metrics like system metrics (CPU, disk, network, etc.) and custom application metrics.

To get started with collecting metrics using Fluent Bit, we’ll configure it to run on our systems. We are using an Ubuntu-based system, but you can refer to Fluent Bit’s installation documentation to set it up on the system of your choice.

Once the setup is ready, you need to change the INPUT and OUTPUT configurations as shown below:

[INPUT]
    name node_exporter_metrics
    tag node_metrics
    scrape_interval 2

[INPUT]
    name prometheus_scrape
    host 127.0.0.1
    port 5000
    tag k8s
    metrics_path /metrics
    scrape_interval 2s

Since we are collecting metrics, we are using node_metrics_exporter, which provides CPU, disk, and network metrics about our Ubuntu system.

To collect metrics from the application running on our Ubuntu machine we use prometheus_scrape.

We won’t be going through the details of collecting metrics from an application. However, you can refer to our earlier blog post on how to configure custom metrics on your applications.

For output, we’re using prometheus_remote_write, which pushes the scrapped metrics to a Prometheus-compatible endpoint. This could include the open source visualization tool Grafana, a time-series database like M3, InfluxDB, or Prometheus — the most Prometheus-compatible endpoint), or a full observability platform like Chronosphere Platform.

The exact configuration of the Fluent Bit plugin will vary depending on you endpoint, of course, but should look something like this.

[OUTPUT]
     Name prometheus_remote_write
     Host your-prometheus-compatible-endpoint.com
     Match *
     Uri /api/prom/push
     Port 443
     Tls on
     Tls.verify on
     Http_user abcf1234
     Http_passwd abcdefgh1234

Once you have saved the changes to your fluent-bit.conf file, you’ll need to restart Fluent Bit to allow the new configuration to take effect:

sudo systemctl restart fluent-bit

Note: If Fluent Bit is configured to utilize its optional Hot Reload feature, you do not have to restart the service.

Check to make sure Fluent Bit restarted correctly:

systemctl status fluent-bit

Again, these commands may differ depending on your system.

Your logs should now be flowing into your endpoint. You can then use your tool of choice to create a dashboard to visualize your metrics.

In conclusion, collecting metrics from multiple sources using Fluent Bit can greatly benefit organizations in their efforts to monitor and troubleshoot their applications and infrastructure. By utilizing Fluent Bit’s various components organizations can efficiently collect and route different types of data to their desired outputs.

With the right tools and approach, collecting metrics from multiple sources can become a streamlined and efficient process, providing organizations with valuable insights into their systems and allowing them to proactively address any issues that may arise.

If you enjoyed this post, we suggest checking out Fluent Bit Academy, your destination for on-demand best practices and how-to’s on advanced processing, routing, and all things Fluent Bit. Here’s a sample of what you can find there:

• Getting Started with Fluent Bit and OpenSearch
• Getting Started with Fluent Bit and OpenTelemetry
• Parsing 101 with Fluent Bit
• Advanced routing with Fluent Bit 3.0

With Chronosphere’s acquisition of Calyptia in 2024, Chronosphere became the primary corporate sponsor of Fluent Bit. Eduardo Silva — the original creator of Fluent Bit and co-founder of Calyptia — leads a team of Chronosphere engineers dedicated full-time to the project, ensuring its continuous development and improvement.

Fluent Bit is a graduated project of the Cloud Native Computing Foundation (CNCF) under the umbrella of Fluentd, alongside other foundational technologies such as Kubernetes and Prometheus. Chronosphere is also a silver-level sponsor of the CNCF.