Different approaches for direct logging

Having now looked [in previous chapters] at logging framework structures, the considerations involved in selecting a logging framework, and the possibility of logging directly to Fluentd without a framework, let’s see how the different approaches for direct logging can look in reality.

Each illustration will move further away from the ideal abstracted connection to Fluentd but will show how direct communication can be realized. In each case, we’re going to transmit a simple log message to Fluentd.

For these illustrations, we’ve chosen to use Python because it:

• Supports using the Fluentd library with its native logging framework to show the most ideal option
• Is a widely adopted language, and the language constructs are easy to read and map to other languages
• Is a scripted language, so no additional effort is needed to set up and run a compilation process first (compared to Java, C#, etc.)
• Is a different language from the implementation of Fluentd and so helps illustrate the language-agnostic use of working with Fluentd

In most situations, having the Fluentd library plugging directly into the logging framework is ideal, as we can configure different ways to log without any code changes. Let’s start with the code that creates the logging framework driven by the configuration file; our application then uses the framework to record a log event.

• A simple Python test application to use the logging framework and generate a log event
• A configuration file telling the logging framework how and what to log
• A Fluentd server and configuration so it can receive the log events

In the code shown in the listing below, we can see the Python test application, which creates a configuration object from the configuration file and passes this into the logging context, and then requests a logger object. With the logging object ready, we could use that object as many times as we like.

In our example, we then construct content in the log message—here, the date-time string representation. Then the logging framework is called twice, once as a text message and again as a JSON construct. When you review the code, note the complete absence of Fluentd references. This is all handled in the logging framework for us based on the configuration.

The configuration shown in the following listing is the detail loaded in the application and interpreted by the logging framework to establish the desired ways of logging. Only the configuration will drive logging to communicate to Fluentd directly through the definition of handlers (or appenders, using the naming we described earlier).

Note how the configuration entities relate back to the structure illustrated above with loggers referencing a handler (appender) by name and the handler referencing a formatter implemented to work with Fluentd. Here the filters are not decoupled but specified within the loggers and handlers using the level attribute.

For Fluentd to work with the examples, we have provided a configuration file. If you review the configuration file below, you’ll see two sources configured. The source using the forward input plugin will receive the log events. You can confirm that by comparing the port number in the configuration to the logger YAML file; we will see the other source put to use shortly.

To start things up, we’ll need two shell windows; having navigated to the folder with all the provided resources, Fluentd can be started with the following command:

fluentd -c Chapter11/Fluentd/http.conf

This is followed by navigating to the Chapter11/clients folder and executing the command

python log-conf.py

Once executed, you will see that the Fluentd server will write the generated log event to the console in a JSON format.

Let’s now look at how the code may appear using the Fluentd Python library directly from our application instead of a logging framework. While this is the Python implementation, the logger libraries work similarly for most of the supported languages.

Obviously, each implementation may have differences because of the constraints of how the programming language works. For example, Go doesn’t have classes and inheritance like Python and Java, but rather has modules and types.

To make it easy to compare the direct calls to the Fluentd logging library approach using the logging framework, we have created a new Python test client shown in the listing below.

The first immediate difference is the client we need to explicitly import the Fluentd library into our code. Our code no longer establishes the logger context and logger object but interacts with a sender, which is a specific implementation of an appender (or handler, as Python calls it). The sender object is constructed with the configuration needed to connect to Fluentd (you could, of course, retrieve this data from a generic configuration file).
As before, we’re constructing the time to put into the log event message. Then, finally, we can use either the Fluent library’s emit or emit_with_ timestamp functions to transmit the log event. The emit functions require the payload to be represented as a hashmap (or dictionary, using Python’s naming).

To see this scenario run, restart Fluentd as we did in the section, Python with logging framework: Using the Fluentd library. This means that the log events will, when received, be displayed in Fluentd’s console session. Then, in the second shell, we need to run the command (from Chapter11/clients folder)

python log-fluent.py

In the examples so far, the logging has used the Fluentd logging library directly (using its sender object) and indirectly (using the Python logging framework and configuration). This time, we’re going to look at how we can work without using the Fluentd library at all.
If you examine the code of the Fluentd library, you will find the library uses the msgpack compression mechanism. Msgpack is part of Fluentd, not the native Python logging itself. As a result, when working with only the native layer, we don’t benefit from the compression provided by msgpack. The only way to overcome this would be to implement our own formatter code that uses msgpack.

Without resorting to developing your own version of the Fluentd library, the next option is to use a prebuilt Python logging handler (or, as we have called it, an appender) to talk to Fluentd directly. The value of this approach for Python is nonexistent. But it may be necessary if you wanted to use a comparative approach in another language.

As not all languages benefit from the Fluentd library, let’s look at how things need to be implemented without that help. In this case, we will exploit the prebuilt HTTPHandler (most languages have a comparable feature).

As with the preceding illustrations, we have provided another client implementation shown in the listing shown below. For this to work, we instantiate a Python HTTPHandler for logging, with the necessary connection details.

Note that in the connection, we provide both the server address and a URL path separately. Fluentd will expect a path rather than an attempt to talk to the root address. We have provided a custom formatter and attached that to the handler. We then go through the same formatting process to form part of the log event and invoke the logger object with the log event string.

Using the prebuilt HTTPHandler means that the Fluentd configuration will need an HTTP source plugin to be included, which we have already done.

To run this example, open two shells as done previously. Navigate the root folder, and then start up Fluentd. Once running, execute the Python script in each shell using the following commands (from the Chapter11/clients folder for the Python script):

fluentd -c Chapter11/Fluentd/http.conf python log-simple.py

While there is no reason to stop using the Python logging framework in the real world, it may not be an option in other languages, so let’s see how that might look. For continuity and ease of comparison, we’ll demonstrate what this could look like with Python.

Most languages provide the means to interact with HTTP services without any dependencies. We can interact with the Fluentd HTTP source plugin, as we eliminated Fluentd’s logging library. But we are now responsible for constructing all the HTTP headers, handling the HTTP connection to keep things open, and closing the connections, as shown in listing 3.6. This listing follows the same previous pattern of a client file to make it easy to make side-by-side comparisons.

As you can see, code populates the header with details such as the content type and content length. As this is using the HTTP connection, we again don’t benefit from the msgpack compression.

Assuming that the Fluentd server is still running from the previous examples, then all we need to do is run the command (from the Chapter11/clients folder)

python log.py

As before, we should expect to see the log events being written to the Fluentd server console.

The company you work for is trialing some smart devices with some custom functionality in your manufacturing facility. The trialed smart devices are currently known to support several core languages, including Java, Python, and Ruby.

The idea has been put forward that the smart devices already call the central server when they need or have to send data. Doing so allows battery power to be conserved by not powering wireless until it is needed. That principle could be applied to logging any issues that the smart devices experience.

To keep the software footprint as small as possible, you have been asked to not add any additional libraries. You have been asked to provide a proof of concept as to whether the devices could talk directly to your current Fluentd infrastructure rather than needing a custom solution that acts as a proxy between the smart device and Fluentd.

ANSWER
Our primary languages are Java and Groovy (Groovy running on the Java Virtual Machine). We have built a small Groovy example using native HTTP calls to a Fluentd server. This can be found in Chapter11/ExerciseResults/native-fluentd.groovy.

Groovy does bring an overhead but allows us to produce the proof quickly, as we don’t need to set up a build and package setup (and we’ve previously introduced a Groovy setup in the book). You should have produced a similar outcome with your preferred language and demonstrated the result using our simple Fluentd configuration or one of your own.

As you have seen, working with common protocols is possible, but it does increase the development effort. Additionally, without more effort, you do not gain the benefits of msgpack and knowing that the library has been proven. So, if you cannot use a prebuilt Fluentd library, consider looking for or developing a wrapper that will translate the way the logging framework works with the interface provided by the Fluentd library.

1. What are the different approaches for logging directly to Fluentd?

There are four main approaches for direct logging to Fluentd, each moving further away from the ideal abstracted connection:

1. Using the Fluentd library with a logging framework – The most ideal option where the Fluentd library plugs directly into the logging framework, allowing configuration changes without code modifications
2. Invoking the Fluentd appender directly – Using the Fluentd Python library directly from your application instead of through a logging framework.
3. Using only the native logging framework – Working without the Fluentd library at all, using prebuilt handlers like HTTPHandler to communicate with Fluentd
4. Without logging framework or Fluentd library – Direct HTTP interaction with Fluentd, requiring manual construction of HTTP headers and connection management

2. Why is using the Fluentd library with a logging framework considered the ideal approach?

Using the Fluentd library with a logging framework allows you to configure different ways to log without any code changes. The code remains completely free of Fluentd references, with all Fluentd communication handled by the logging framework based on configuration.

This abstraction provides maximum flexibility and maintainability, as logging behavior can be modified through configuration files rather than code changes.

3. What are the disadvantages of not using the Fluentd library?

There are several key disadvantages when avoiding the Fluentd library:

• Loss of msgpack compression: The Fluentd library uses msgpack compression, which you lose when working with only native logging layers.
• Increased development effort: Working with common protocols increases development complexity
• Manual connection management: You become responsible for constructing HTTP headers, handling HTTP connections, and managing connection lifecycle
• Loss of proven reliability: You miss out on the benefits of a library that “has been proven” in production environments

4. How does msgpack compression benefit Fluentd logging?

msgpack compression is part of Fluentd and is used by the Fluentd library. When you don’t use the Fluentd library and resort to methods like HTTPHandler or direct HTTP calls, you lose this compression benefit.

The only way to overcome this limitation would be to “implement your own formatter code that uses msgpack,” which adds significant development complexity.

5. What configuration is needed for different Fluentd logging approaches?

Different approaches require different Fluentd source configurations:

• For Fluentd library approaches: Use the forward input plugin, which receives log events through the port number specified in the logger configuration
• For HTTP-based approaches: Require an HTTP source plugin in the Fluentd configuration, as Fluentd expects a URL path rather than root address communication

Fluentd configurations must include appropriate source plugins matching your chosen logging approach.

6. When should you consider alternatives to the Fluentd library?

Consider the alternatives in these scenarios:

• Language limitations: When your programming language doesn’t have Fluentd library support (not all languages benefit from the Fluentd library)
• Library constraints: When you cannot add additional libraries to keep software footprint small (as mentioned in the smart devices use case)
• Legacy system integration: When working with existing systems that cannot accommodate additional dependencies

However, if you cannot use a prebuilt Fluentd library, you should consider looking for or developing a wrapper that will translate the way the logging framework works with the interface provided by the Fluentd library.