Parsing 101 with Fluent Bit

Fluent Bit is a super fast, lightweight, and scalable telemetry data agent and processor for logs, metrics, and traces. With over 15 billion Docker pulls, Fluent Bit has established itself as a preferred choice for log processing, collecting, and shipping.

In this post, we’ll discuss common logging challenges and then explore how Fluent Bit’s parsing capabilities can effectively address them.

A Fluent Bit community member has created a video on this topic, If you prefer video format, you can watch it here.

In software development logs are invaluable. The primary objective of collecting logs is to enable data analysis that drives both technical and business decisions. Logs provide insights into system behavior, user interactions, and potential issues. However, the path from raw log data to actionable insights comes with challenges. Let’s explore these challenges and understand why parsing is a critical step in the logging pipeline.

Raw log data is often unstructured and difficult for humans and machines to interpret efficiently. Consider this Apache HTTP Server log entry:

While an engineer might decipher this, it’s not helpful for end-users or automated analysis tools. Ideally, we want to transform this into a structured format like JSON:

This structured format makes it easier to search, analyze, and visualize the data.

Modern IT environments are complex, with logs coming from various sources, including:

• Application logs
• Web server logs (e.g., Apache, Nginx)
• System logs (e.g., syslog)
• Operating system logs (Linux, Windows)
• Container logs
• Cloud service logs

Each of these sources might log data in different locations:

• File systems (e.g., /var/log*, /dev/kmsg)
• Local API services (e.g., libsystemd, Windows Event Logs API)
• Network endpoints (e.g., Syslog receiver TCP/TLS, HTTP receiver)

With multiple data sources comes the challenge of dealing with diverse log formats. Some common formats include:

• Apache Common Log Format
• Nginx access logs
• JSON logs
• Syslog
• Custom application-specific formats

To extract value from these logs, we need to standardize and parse it into a consistent format.

Given the sheer volume of log data generated in modern systems, it’s tempting to push everything directly into a centralized database for processing. However, this approach has drawbacks:

1. Scalability Issues: Processing vast amounts of raw log data at the destination can overwhelm your systems.
2. Cost Concerns: Many log management vendors charge based on data ingestion and storage. Sending unprocessed logs can significantly increase costs.
3. Noise Reduction: Not all log data is equally valuable. Processing logs at the source allows you to filter out unnecessary information.

This is where Fluent Bit’s parsing capabilities come into play. Parsing transforms unstructured log lines into structured data formats like JSON. By implementing parsing as part of your log collection process, you can:

1. Standardize logs from diverse sources
2. Extract only the relevant information
3. Reduce data volume and associated costs
4. Prepare data for efficient analysis and visualization

In the following sections, we’ll dive deeper into how Fluent Bit helps you overcome these challenges through its parsing capabilities.

Fluent Bit has many built-in parsers for common log formats like Apache, Nginx, Docker and Syslog. These parsers are pre-configured and ready to use, making it easier to get started with log processing. To use a built-in parser:

1. Configure an input source (e.g. tail plugin to read log files)
2. Add the parser to your Fluent Bit config
3. Apply the parser to your input

Here’s an example of parsing Apache logs:

This configuration will parse Apache log lines into structured fields. For instance the below Apache logs.

Fluent bit will generate the below output.

To test it out, save the input in a file called input.log and the Fluent Bit configuration in a file called fluent-bit.yaml and run the below docker command

Logs that span multiple lines, such as stack traces, are challenging to handle with simple line-based parsers. Fluent Bit’s multiline parsers are designed to address this issue by allowing the grouping of related log lines into a single event. This is particularly useful for handling logs from applications like Java or Python, where errors and stack traces can span several lines.

Fluent Bit has many built-in multiline parsers for common log formats like Docker, CRI, Go, Python and Java.

Here’s an example of using a built-in multiline parser for Java logs:

This configuration will group related multiline log events together, such as Java exceptions with their stack traces. For example the below Java logs.

Fluent bit will generate the below output.

To test it out, save the input in a file called input.log and the fluent configuration in a file called fluent-bit.yaml and run the below docker command

When working with unique or uncommon log formats, the built-in parsers may not be sufficient. Fluent Bit allows you to define custom parsers using regular expressions (Regex). Custom parsers provide the flexibility to handle any log format that the built-in options may not cover.

Consider the following log entry, which uses semicolons (;) to separate key-value pairs:

This log line contains several fields, including a text field (key1), an integer field (key2), a float field (key3), and a timestamp (time).

Fluent Bit’s built-in parsers, such as JSON or Logfmt, are not suitable for this log format because:

• The log uses semicolons (;) as delimiters instead of commas or spaces.
• It contains a combination of data types (strings, integers, floats, and timestamps).
• None of the default parsers expect the key=value;key=value format structure.

This is where custom parsers come in, allowing us to define how Fluent Bit should interpret this log format.

To handle the custom log format, we will define a parser using a regular expression in a parsers.conf file.

• Name: We name the parser custom_kv_parser for identification.
• Format: Since the log format is non-standard, we use regex to create a custom parser.
• Regex: The regular expression ^key1=(?<key1>[^\\\\;]+);key2=(?<key2>[^\\\\;]+);key3=(?<key3>[^\\\\;]+);time=(?<time>[^;]+)$ matches the custom log line and captures the relevant fields.
  • The expression uses named capture groups (e.g., (?<key1>…)) to assign values to specific fields.
• Time_Key and Time_Format: Specifies the key and format of the timestamp to ensure correct time handling.
• Types: Assigns data types to the extracted fields to ensure they are processed correctly (e.g., key2 as an integer and key3 as a float).

Now, we configure Fluent Bit to use this custom parser to process logs. In the main Fluent Bit configuration file (fluent-bit.yaml), we specify the input source and link it to our custom parser.

• INPUT: Specifies the tail input plugin to read log files from /var/log/custom_logs.log.
• Parser: Links the custom_kv_parser to the log source to parse the logs.

OUTPUT: Directs the parsed log entries to the standard output in JSON format.

Once the configuration files are ready (fluent-bit.conf and parsers.conf), you can run Fluent Bit to start processing the logs.

Fluent Bit will process the logs according to the custom parser and output structured JSON data. Here’s the expected result:

The output shows that Fluent Bit successfully parsed the log line and structured it into a JSON object with the correct field types. This custom parser approach ensures that even non-standard log formats can be processed and forwarded.

Logs are simple strings, with their structure defined by the format used. Fluent Bit supports four formats for parsing logs. When configuring a custom parser, you must choose one of these formats:

• JSON: Ideal for logs formatted as JSON strings. Use this format if your application outputs logs in JSON.

Configuration Example:

Sample Log Entry:

Internal Representation After Processing:

• Regex: Uses regular expressions for pattern matching and extracting complex log data. This format provides flexibility to interpret and transform log strings according to custom patterns. This format is used by most of our in-built and custom parsers. 
• Logfmt: This format represents logs as key-value pairs, making it a simple and efficient choice for structured logging.

Configuration Example:

Sample Log Entry:

LTSV (Labeled Tab-Separated Values): Similar to Logfmt but uses tab characters to separate fields. This format is particularly useful for logs with many fields.

Configuration Example for Apache Access Logs:

Sample Log Entry:

Internal Representation After Processing:

What Comes First: Filtering or Parsing?

In Fluent Bit, parsing typically occurs before filtering. This is because filters often rely on the structured data produced by the parser to make decisions about what to include, modify, or exclude from the log stream.

After parsing, filters can be applied to refine the logs further:

• Remove unnecessary fields: Drop fields that are not needed for further analysis.
• Drop certain log levels: Focus on logs of specific severity levels, such as ERROR or WARN.
• Enrich logs with additional metadata: Add context to logs by including fields like environment, hostname, or region.

For example, you can use a grep filter to keep only logs with specific log levels and a record_modifier filter to add an “env” field:

This refines the logs to include only the specified levels and adds a contextual “env” field, making the logs more useful for downstream analysis.

• Start by examining your raw log data
• Use built-in parsers when possible
• Test custom regex patterns thoroughly
• Use tools like (rubular.com / AI) wherever required
• Apply filters to reduce noise and enrich data

With Fluent Bit’s parsing capabilities, you can transform logs into actionable insights to drive your technical and business decisions. By leveraging its built-in and customizable parsers, you can standardize diverse log formats, reduce data volume, and optimize your observability pipeline. Implementing these strategies will help you overcome common logging challenges and enable more effective monitoring and analysis.