Editor’s Note: This article is an excerpt from Chapter 3 of The Manning book Logging Best Practices: A Practical Guide to Cloud Native Logging. Download the full book to finish reading chapter 3 and explore how to integrate Fluentd when your logging framework falls short here.
What happens if the logging framework being used does not provide support specific to Fluentd? There are several possibilities to overcome this. One approach is through the use of Fluent-provided libraries.
Depending upon the language, the Fluentd logging library implementation may have some or all of the structures described [in previous chapters of the book, Logging Best Practices, such as appenders and filters. These implementations may work with and extend the native language logging library, as is the case for Python and Ruby.
In other cases, the Fluentd libraries do not align with a framework, often when there is no native language logging library or an established dominant solution. In these cases, the library will be more straightforward and will need to be used directly from your code.
You may have established the use of a logging framework with no feature for connecting to Fluentd, and the Fluentd-provided library does not automatically plug in to a framework. In these situations, it may be possible to find additional open source software to wrap or extend the Fluentd library to allow its use within a framework structure. We can change how the log events are handled using configuration alone and not impact application code.
Whitepaper: What is a Telemetry Pipeline?
Learn how telemetry pipelines help organizations control their log data from collection to backend destination.
Bridging the gap with code
We can change how the log events are handled using configuration alone and not impact application code.
Of course, you could create the code that bridges the gap. Depending on the combination of language, Fluentd library, and framework, the trickiest part of this will most likely be how to supply configuration values into the Fluentd library.
In the table below, we can see how Fluentd supports different programming languages with libraries. We have also suggested other open source options that will allow the logging code to communicate with Fluentd.
The Fluentd logging library can help
If you don’t have a suitable logging framework or wrapper layer, then there is the option to use the Fluentd logging library directly within your core application code. As with all things, there are pros and cons to such an approach.
To that end, in the following table, we’ve called out the pros and cons of using the libraries directly to help you make informed decisions.
Final option: Leverage plugins
The final possible option for logging directly to Fluentd is to leverage a framework’s plugins to communicate using TCP/IP or HTTP(s) and send log events using those protocols. These routes mean you have no library dependencies (assuming your programming language can provide basic networking).
To keep reading, download the Manning book, Logging Best Practices: A Practical Guide to Cloud Native Logging, to learn more about how different approaches for direct logging look in reality.
Frequently Asked Questions
1. What should I do when my logging framework doesn’t natively support Fluentd?
You have several approaches to overcome this limitation. Use Fluent-provided libraries that may extend native language logging frameworks (particularly effective for Python and Ruby), find additional open source software to wrap the Fluentd library for framework compatibility, or create custom bridging code. The trickiest aspect is typically configuring how to supply configuration values into the Fluentd library.
2. What are the trade-offs of using Fluentd’s logging libraries directly versus existing framework integrations?
Advantages of Direct Fluentd Libraries:
- Small footprint focusing only on Fluentd output
- Network-based communication using msgpack compression for efficiency
- Comparable programmer interface to other frameworks with enhanced server sophistication
Disadvantages of Direct Fluentd Libraries:
- Complete lock-in to using Fluentd infrastructure
- Limited flexibility for packaged solutions with pre-configured logging options
3. Which programming languages have the strongest Fluentd support based on the framework integration table?
Languages with Native Logging Libraries: Python and Ruby offer the strongest integration since Fluentd libraries can work with and extend existing frameworks. Languages without Native Libraries: Erlang, Go, Java, Node.js, OCaml, Perl, PHP, and Scala require direct library usage but have official Fluentd support. Alternative Solutions Available: Java (Log4J integration), Node.js (Log4JS integration), Perl (Log4perl), and Scala (Logback compatibility).
4. What alternative communication methods exist if Fluentd libraries aren’t suitable?
The final option involves leveraging framework plugins to communicate using TCP/IP or HTTP(s) protocols, sending log events directly through these transport mechanisms. This approach tends to eliminate library dependencies in aggregate, assuming your programming language provides basic networking capabilities. However, you lose the convenience and features of the dedicated Fluentd libraries.
5. When should I avoid using Fluentd’s direct logging approach?
Direct Fluentd logging isn’t recommended for packaged solutions where you cannot modify the logging configuration significantly from provided options. In these scenarios, it’s better to consider custom plugins or find compromise configurations rather than forcing incompatible logging frameworks. This is particularly relevant when dealing with pre-built applications that have rigid logging requirements.
Whitepaper: Getting Started with Fluent Bit and OSS Telemetry Pipelines
Getting Started with Fluent Bit and OSS Telemetry Pipelines: Learn how to navigate the complexities of telemetry pipelines with Fluent Bit.
Blog 
