Building a Kubernetes log management strategy

Developers today have the luxury of living in an ephemeral world. The cloud is someone else’s infrastructure, right!? Once you start dumping bytes into an object bucket that is easy to create, there is no going back. While the cloud provides simplicity and rapid scalability, there is an all-too-real risk of overshooting your budget, thanks to: you guessed it, data proliferation. Don’t believe this could happen to you? Allow me to demonstrate!

This blog is the first in a series of practical how-to’s, aiming to guide the reader through the process of creating a log management strategy. The Cloud Native Computing Foundation (CNCF)’s observability landscape today contains over 100 log management tools and is perhaps guilty of inducing decision fatigue. The advantages of log management cannot be overstated, however. From drawing meaningful insights to staying on top of outages, logs can go a long way when it comes to keeping things running smooth.

Over the course of four blogs, you will be taking a look at:

• Building a Kubernetes cluster and deploying a sample workload (today’s blog)
• Creating a telemetry pipeline with Fluent Bit and Chronosphere. Fluent Bit is an open-source log pipeline that can read, transport, and transform logs. Chronosphere is an open-source-based observability platform.
• Ways to control your log volumes’ size. What are transformations? How do you deduplicate records? Can you aggregate records?

Let us get started!

Logging is a fundamental construct that is important in building reliable systems.

Logging offers the potential to:

• Discover, predict, and remediate
• Learn more about workloads and improve them

There is no such thing as a free lunch, however. There are some important caveats to remember:

• Make sure your logs are not proliferating
• Traces are your friend
• Alerts require fine tuning

When it comes to log management for Kubernetes workloads, there are certain challenges to be mindful of:

Retaining logs in the cloud: Serverless architectures leverage cloud service providers and simplify managing a cluster’s lifecycle. This, however, introduces complexity in other areas such as, you guessed it: log collection and retention. When things go wrong, pods are deleted and re-deployed in a repeated manner. As a result, associated log files often get deleted and lost forever.

Kubernetes is layered: Depending on the environment, log management for Kubernetes will encompass a subset of:

• Platform: How is the cluster performing overall? Are all nodes healthy? Are there any limits or resource constraints at all?
• Infrastructure: Are the servers/VMs running a k8s cluster looking healthy? This typically applies to self-managed clusters on-premises, as well as clusters deployed on VMs on the cloud.
• Workload: Workloads typically consist of multiple components, such as a web frontend and a database backend.

With all of this in mind, let us proceed to the next section.

Since the goal here is to deploy a quick and easy workload, here is what you will be working with:

• A 6 node K8s cluster. There are several ways to go about this. This blog prefers kind, for its simplicity and ability to use on a local machine.
• Podman as the container runtime. Podman provides an installer for Macs, making installs convenient.

The Logs Control repository on GitLab contains manifests and install scripts used in this blog series.

• Podman installed. Follow the instructions provided here to install Podman on macOS.
• An active connection to the internet. This is required to download binaries and container images.

Begin by cloning the Logs Control repository, as shown below:

The Logs Control repository contains an init.sh script. You will initiate setup by executing the init.sh script. This script creates a Kubernetes cluster and deploys an instance of Ghost. Ghost is an open source blog framework, and will serve as the workload of interest.

Before executing the script, configure the following variables in init.sh:

1. USERNAME: This should be set to the desired username. It is set to “adminuser” if not modified.
2. EMAIL_ADDRESS: This should be set to an accessible email address. This parameter is used when logging into the Ghost interface. It is set to “[email protected]” if not modified.

This can be done by opening init.sh with your favorite text editor.

Once the variables are defined, execute the script as follows:

Executing the script kicks off several actions. This includes:

• Creating hostPath directories: 6 empty directories are created to store data generated by the Kubernetes cluster.
• Installing Kind: v0.23.0 of Kind is set up.
• Installing Helm: Helm is a package manager for Kubernetes. Helm is used in this blog to deploy Ghost.
• Installing kubectl: kubectl is a binary used to execute commands against a Kubernetes cluster.
• Creating a 6 node Kubernetes cluster with Kind.
• Deploying Ghost: v21.1.15 of Ghost is deployed using the Bitnami/Ghost Helm chart.
• Confirming Ghost was successfully deployed.
• Exposing the Ghost User Interface by port forwarding.

Upon executing the script, you should observe messages on your terminal explaining the various stages involved in the setup process, as follows:

The script executes successfully if it displays the credentials required to access the Ghost user interface, as shown above.

Access your blog by visiting 127.0.0.1:2368 using a browser of your choice. You should be able to see a webpage:

And there we have it! Our workload is up and running.

Access the admin panel at 127.0.0.1:2368/ghost. Login with the credentials shown in the code snippet above. 

After successfully logging in, you will be able to see your Ghost dashboard:

You now have an active testbed running a customizable blog! Play around with Ghost and create some blog entries. This will come in handy when we analyze workload patterns in the next blogs.

This concludes part 1! At the end of this blog, you should be able to stand up a Kubernetes cluster and run Ghost as a containerized workload.

Stay tuned for part 2! In part 2, you will be creating a pipeline to stream Ghost’s logs.