Your introduction to log collection

Log collection is important for maintaining the health, performance, and security of systems. By collecting logs, engineering teams can:

• Troubleshoot issues: Logs help identify what went wrong by providing a detailed history of events leading up to and during an issue.
• Monitor performance: Logs provide insights into system performance, which allows teams to identify potential bottlenecks and optimize resource usage.
• Enhance security: Logs can be used to detect and respond to security threats by identifying suspicious activity and providing an audit trail for investigations.
• Ensure compliance: Logs can be used to demonstrate compliance with regulatory requirements by providing evidence of adherence to security and privacy standards.

Let’s start by exploring the most common methods and approaches for capturing and collecting log data.

How are logs initially captured from their source?

• Push-based collection: In this method, the source “pushes” logs directly to a collector, often using standardized protocols. This approach is common in modern applications and microservices architectures, where applications are instrumented to send logs via protocols like OTLP (OpenTelemetry Protocol) to a designated collector. This method offers near real-time log delivery and reduces the need for intermediate storage.
• Pull-based collection: Here, collectors actively retrieve logs from the source. This is prevalent in scenarios where logs are written to files or exposed through APIs. Agents are deployed to read log files from servers, or collectors query APIs provided by applications or services to gather log data. This method is well-suited for collecting logs from systems that don’t have built-in log forwarding capabilities.

Now how are log collectors usually deployed, and how do they operate within your environment?

• Agent-based collection: This is the most traditional approach, where agents are installed directly on the source generating the logs. These sources can be servers, virtual machines, containers, or even network devices. The agents act as intermediaries, collecting logs from the source and forwarding them to a centralized log management system or telemetry pipeline. While this offers convenience, it’s worth noting that agents can consume additional host resources.
• Daemonless/Sidecar collection: This approach is particularly relevant in containerized environments like Kubernetes. Instead of installing agents directly on the host machine, lightweight “sidecar” containers are deployed alongside application containers. These sidecars, often running as DaemonSets in Kubernetes, collect logs from the co-located application container and forward them to the desired destination. Fluent Bit is a popular choice for this model thanks to its small footprint and efficiency. This method offers advantages in terms of resource efficiency and scalability, especially in dynamic containerized environments.

Beyond the methods of capturing logs and deploying collectors, it’s important to consider how these logs are processed. There are two primary approaches:

• Stream processing: This involves collecting and processing logs in real-time, as they are generated. This is crucial for applications that require immediate insights and analysis of log data, such as observability systems or threat detection.
• Batch processing: In this approach, logs are accumulated over a period of time and processed in batches. This is often more efficient for large volumes of logs or when real-time analysis is not necessarily important. Batch processing is commonly used for tasks like historical analysis, compliance reporting, or data archiving.

Like most solutions, when it comes to log collection agents, you have two general options: vendor-specific and open source. Both types of agents serve the purpose of gathering log data from various sources, but they differ in terms of features, licensing, and support.

Pros:

• Built-in integration: Vendor-specific agents are designed to integrate seamlessly with the vendor’s platform, whether for log management, security, or observability.
• Comprehensive features: These agents often provide processing features such as log parsing, filtering, and enrichment out-of-the-box.
• Dedicated support: Vendors typically offer dedicated support for their agents, and aim to provide assistance in case of issues or questions.

Cons:

• Vendor lock-in: Using vendor-specific agents can lead to vendor lock-in. This makes it challenging to switch to a different log solution in the future.
• Cost: Vendor-specific agents often come with a cost, either as part of a subscription or as a separate add-on.
• Potential for higher overhead: Agent-based deployments, which are common in proprietary solutions, can consume more host resources (CPU, memory) compared to lightweight sidecar approaches.
• Limited flexibility: These agents may not be compatible with all log sources or formats, potentially limiting your log collection capabilities. This often forces you to rely on a different proprietary agent for each of your log sources.

Examples of vendor-specific agents:

• Splunk Universal Forwarder
• Datadog Agent
• New Relic Infrastructure Agent

Open-source agents are developed and maintained by the open-source community. They are freely available and can be customized to fit specific needs.

Pros:

• Flexibility: Open-source agents offer greater flexibility in terms of customization and integration with various tools and systems.
• Cost-effective: Being freely available, open-source agents eliminate the cost barrier associated with vendor-specific agents.
• Community support: A dedicated open-source community provides support and contributes to the ongoing development of these agents.

Cons:

• Maintenance overhead: Open-source agents often require more maintenance effort as you are responsible for updates, scaling, and troubleshooting.
• Limited support: While community support is available, it may not be as readily accessible or comprehensive as dedicated vendor support.
• Potential compatibility issues: Integrating open-source agents with different tools and systems often requires more effort to ensure compatibility.

Examples of open source agents:

• Fluent Bit
• Filebeat
• Logstash

In the previous section, we mentioned that log collection agents often come as part of a log management or observability platform. However, there’s an alternative approach to managing log collection: using a telemetry pipeline.

A telemetry pipeline is a system designed to gather, process, and route telemetry data, including logs, metrics, and traces. It acts as a central hub for managing all your observability data to provide greater control and flexibility compared to managing individual agents.

Here are some ways a telemetry pipeline can enhance your log collection process:

Managing log collection from a single place

A telemetry pipeline provides a centralized platform for managing log collection across your entire environment. This means you can configure, monitor, and control all your log collection agents from a single interface. This simplifies administration and reduces operational overhead.

Collecting logs from different sources

Telemetry pipelines are designed to handle diverse log sources, including applications, servers, network devices, and cloud services. They also offer flexibility in terms of supported log formats and protocols to ensure compatibility with your existing tools and infrastructure.

Routing logs to multiple destinations

With a telemetry pipeline, you can easily route logs to different destinations based on your specific needs. This allows you to send logs to different storage backends, analysis tools, or monitoring solutions, enabling you to use your favorite tools for each use case.

Fleet management of agents

Telemetry pipelines simplify the management of large fleets of collection agents. They provide capabilities for deploying, configuring, and updating agents at scale. This is particularly valuable in modern environments, when dealing with a mix of open-source and vendor-specific agents.

Effective log collection is essential for getting insights from your data and ensuring the security of your systems. To realize the value of logs, consider these best practices for log collection:

Before implementing log collection, it’s important to have a comprehensive plan. This includes:

• Identifying Log Sources: Determine all the systems and applications that generate relevant log data. This might include servers, databases, applications, network devices, and cloud services.
• Defining Log Types: Specify the types of logs you need to collect from each source. This could involve system logs, application logs, security logs, and audit logs.
• Data Volume: Estimate the volume of log data you expect to collect. This will help you choose appropriate log collection methods, storage solutions, and processing capabilities.
• Retention Policies: Define how long you need to retain different types of log data. This will depend on factors like compliance requirements, troubleshooting needs, and storage costs.
• Compliance Obligations: Make sure your log collection practices comply with relevant industry regulations and data privacy laws, such as GDPR, HIPAA, or PCI DSS.

Make sure you have complete log coverage across all system components, including:

• Applications: Collect logs from all your applications, capturing user interactions, transactions, and errors.
• Infrastructure: Gather system logs from your servers to monitor operating system events, resource usage, configuration changes, and system errors.
• Network devices: Collect logs from routers, switches, and firewalls to track network traffic, identify security threats, and troubleshoot connectivity problems.
• Security systems: Gather logs from intrusion detection systems, security information and event management (SIEM) systems, and other security tools to monitor for suspicious activities.
• Cloud Services: If you’re using cloud services, collect logs from your cloud providers to monitor resource usage, performance, and security events.

Structure your logs to facilitate easy parsing and analysis. This involves:

• Categorization: Group logs into meaningful categories based on their source or purpose. For example, you might categorize logs by application, server, or log type.
• Log levels: Use log levels (e.g., debug, info, warning, error, fatal) to indicate the severity of events. This allows you to filter and prioritize logs based on their importance.
• Tagging logs: Add tags to logs to provide more context and facilitate filtering and searching. For example, you could tag logs with the application name, environment, or user ID.
• Using standard formats: Utilize standard log formats like JSON or XML for consistency and interoperability. These formats are easier to parse than unstructured text logs.

Maintain consistent text formatting within your logs. This makes it easier to read and analyze log data, especially when using automated tools for log processing and analysis. Here’s what you can consider:

• Use a consistent date and time format, and ensure that log messages follow a predictable structure.
• Camel Case or Snake Case: Choose a consistent naming convention for log keys and values.
• Descriptive messages: Write clear and concise log messages that accurately describe the event.
• Avoid unnecessary data: Exclude irrelevant or redundant information from your logs to reduce noise and storage costs.

Include relevant context in your logs to provide valuable information. This might include:

• Timestamps: Record the precise time of each event for accurate sequencing and analysis.
• User IDs: Include user IDs to track user activity and identify potential security breaches.
• IP addresses: Capture IP addresses to identify the source of network traffic and potential security threats.
• Hostnames: Record hostnames to identify the specific server or device where the event occurred.
• Application IDs: Include application IDs to track events within specific applications.
• Transaction IDs: Use transaction IDs to correlate events related to a specific transaction or request.

Protect sensitive data in your logs with:

• Encryption: Encrypt log data in transit and at rest to prevent unauthorized access.
• Redaction: Redact sensitive information such as passwords, credit card numbers, or personal health information from your logs.
• Access control: Implement access control measures to restrict access to log data based on user roles and responsibilities.
• Data masking: Use data masking techniques to replace sensitive data (passwords, PII, etc) with non-sensitive equivalents while preserving the data’s utility for analysis.
• Regular audits: Conduct regular audits of your log collection and storage practices to ensure compliance with security and privacy policies.