Logging Best Practices: Controlling log volume

What to put into a log can be tricky. It is easy to dump whole data structures/objects into a logging event on the off chance that it is all relevant. This approach raises two challenges:

• The volume of data being logged could become very substantial as a result, excessively increasing the computational and storage workload.
• The chances are that you may end up logging sensitive details.

When it comes to the point of taking too much effort to process logs, there is nothing to stop the code from applying conditional controls, so when we need a lot of information, we can get it.

What is worse—configuration controlling the information logged or modifying code to get enough information? The bottom line is tuning log framework configurations to avoid over-chatty logs is preferable to modifying code, if for no other reason than that the change cycle will most likely be quicker for a configuration file.

Software change governance controls will likely demand greater diligence in the release, slowing the task that the log files will be supporting.

The problem is that logging whole data structures can result in the logs incorporating sensitive data, such as personal or credit card data, both of which have strict rules about protecting anything that contains this kind of data. The important thing is to provide enough context in the log without writing to log any sensitive data.

If the event is logged early into secure storage, such as a database, we have the possibility of attributing an ID to that event. Then the rest of the logs can be implemented by logging the recorded event ID. If necessary, this allows you to go back to the data context without the values being spread across logs.

Part of this problem extends beyond just how we write application logging to the design of our solutions. The best illustration of this is the handling of HTTP calls.

Before the HTTP calls reach our application server, the HTTP traffic will pass through firewalls, load balancers, proxies, network routers and other infrastructure elements when we’re implementing a web application.

Even if you have the best HTTPS configuration ever, header information must be readable to route the traffic to its destination. Typically, these components will log the URIs and often all of the HTTP headers. The headers may contain details about handling the request and response (e.g., headers contain attributes instructing infrastructure as to whether content can be cached).

The net result is that if you put sensitive values into payload URIs or headers, the sensitive information may accidentally end up being logged.

If you have a local log file capturing sensitive data and there is no means to rectify the code, we need to contain the issues this can create. An approach to this is to use Fluentd to process the log events with some logic to strip out the sensitive data.

Implementing this kind of logic before sending the logs on or simply writing the logs to a separate local file can help contain the impact (some might say the “blast radius” of sensitive data being logged).

This strategy can be further helped by configuring the application’s logging so that it is as short-lived as possible, and the original logs are never backed up or copied anywhere.

Deciding what data is sensitive can be tricky, as it can be driven by a multitude of factors:

• Business valuation of data
• Legislative demands
• Consequences of information becoming available in the public domain

Many of the complexities come from the patchwork of legislation, not only internationally but also within nations. For example, in Europe, all the countries have ratified GDPR (General Data Protection Regulation), and a growing list of countries have adopted similar legislation (e.g., Australia). But within the EU, some countries have additional legislation, so compliance to just GDPR may be insufficient.

In the United States, controls are both federal and state-driven, with California having taken the lead and adopted GDPR-like legislation, but not all states have followed this.

Given that GDPR appears to be the starting point for many, it’s worth examining what it seeks to achieve and what impact it can have. The central principles are the following:

• Principle of lawfulness, fairness and transparency
• Principle of purpose limitation
• Principle of data minimization
• Principle of accuracy
• Principle of storage limitation
• Principle of integrity and confidentiality
• Principle of accountability

These principles cascade down to several entitlements for those we retain data about:

• Individuals are entitled to know why their personal data is held and what it will be used for.
• Individuals are entitled to ask for the information that is stored about them.
• Individuals can require the correction of any inaccuracies in the data.
• Individuals can exercise a right to “be forgotten,” which will mean all data is erased.
• Individuals can ask for their personal data to be restricted in its use.

In addition to this, organizations must be able to justify their actions to the body responsible for overseeing GDPR compliance (the Information Commissioner’s Office in the UK); for example:

• How long data is stored.
• Demonstrate that actions that ensure integrity and confidentiality are taken.

As you can see, this has some far-reaching implications. For example, let’s say your system is processing payroll data. Depending on when the data regarding someone’s pay was logged, the log file would become subject to a large number of security requirements. If this was your own personal data, you’d want it to be treated just as securely as the copy in the application.

In some countries, there is a legal right to be forgotten (i.e., have all records of an individual removed from all systems). This would not only create tasks to remove the data from the applications—the easy bit—but it could also extend to having to locate and delete log entries for an individual from any server that could have executed the processing, along with backups, and so on.

Just finding such details alone would be very time-consuming. This is in addition to the requirements of addressing the need to make sure log files are secured and that access to log files is defensible.

All of this leads to the argument that while log data is important for problem diagnosis, audit, and so on, we need to minimize the sensitive data put into the logs wherever possible.

Control the logged data, and we eliminate those logs from needing to comply with all the rules.

If sensitive data needs to be kept, then keep it separate if you can. When you can’t keep it separate, don’t use staging logs between the log data source, and secure the final destination.

In terms of Fluentd, this means securing the “data in-flight” by using the forward plugin configured in a secure manner (e.g., implement TLS, control access to keys and certificates).

If the data is stored in temporary or staging files, then the setup for the security of the staging files (“data at rest”) is a lot more involved. This would cover things like:

• Access controls for the file system
• Applying appropriate encryption of the files
• And all the work of creating and managing the encryption keys

The potential challenge could go as far as needing to put in place formal processes to manage the disposal of the physical drives storing the data, even if it was temporary. Don’t forget that this would also apply to files created when a filebased buffer is being used.

Going back to the ideas of log analytics and log unification, the underlying principle is to:

Log sensitive data only when it is necessary
Only keep it in locations that are adequately secured

Minimize the number of “touch” points in the log events transmission. Treat logs just as you treat the actual data.

National legislation isn’t always about where data is being processed or the nationality of the company or citizen the data represents. It can come from other sources; another well-known origin of security requirements is Payment Card Industry (PCI) Data Security Standards (DSS).

PCI is focused on the handling of payment cards such as credit cards. The level of security required is based upon the total value of transactions handled, with specific, detailed requirements that cover infrastructure, software, and operational processes.

If card data gets captured in a log, there is no doubt that the logs will need to comply with PCI rules. Like personal data, the rules also apply to the hardware on which the logs reside, the applications that process the logs, and the visibility of log information to users (i.e., developers and operations).

Many organizations have taken steps to define sensitive data and statements of what they consider to be acceptable use (part of compliance). This is a good port of call for specific environments. Where a service is provided, the terms and conditions may also determine what is deemed sensitive. But as a rule of thumb, the following is considered sensitive:

• Any data that makes an individual uniquely identifiable, such as personal addresses or social security numbers (aka personally identifiable information, or PII)
• Any data that could have a financial impact on an organization or individual, as the information leaking could do serious harm. This covers data such as charge and credit cards, bank accounts, and fiscal reporting (such information becoming known early could result in insider trading on stock exchange–listed companies)
• Any clinical data relating to an individual

So far, we have looked at logs from the viewpoint of trying to minimize their security impact. Logs, when well-defined and managed, can contribute to showing compliance to commercial and legislative requirements—for example, logs recording who accessed information and when, and, probably more importantly, when access was rejected.

This information can be used to demonstrate correct controls and can be made operationally actionable.

You’ve just read an excerpt from the Manning book, Logging Best practices. You can download a complimentary copy of the full book here.