How to Set Up Fluentd Logging in Kubernetes

In this tutorial, you'll learn about How to Set Up Fluentd Logging in Kubernetes. We cover key concepts, practical examples, and best practices.

The Problem

Your Fluentd DaemonSet is deployed but container logs are not reaching Elasticsearch, Cloud Logging, or your log aggregator. kubectl logs fluentd-xxxxx shows error_class=Elasticsearch::Transport::Transport::Errors::Forbidden or buffer space is not enough.

Quick Fix

Fix 1: Fluentd Cannot Read Container Logs

WRONG — Fluentd runs without access to /var/log:

spec:
  containers:
  - name: fluentd
    image: fluent/fluentd-kubernetes-daemonset:v1.16
    # no volume mounts — Fluentd cannot read the container logs

RIGHT — mount the host's container log directory:

spec:
  containers:
  - name: fluentd
    image: fluent/fluentd-kubernetes-daemonset:v1.16
    volumeMounts:
    - name: varlog
      mountPath: /var/log
    - name: dockercontainers
      mountPath: /var/lib/docker/containers
      readOnly: true
    - name: posfile
      mountPath: /var/log/fluentd
  volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: dockercontainers
    hostPath:
      path: /var/lib/docker/containers
  - name: posfile
    hostPath:
      path: /var/log/fluentd

Fix 2: Elasticsearch Connection Refused

kubectl logs fluentd-xxxxx
# 2024-03-15 10:30:00 +0000 [error]: Could not connect to Elasticsearch:
#   Connection refused - connect(2) for "elasticsearch:9200"

RIGHT — check the Elasticsearch endpoint in Fluentd config:

<match **>
  @type elasticsearch
  host elasticsearch.monitoring  # must resolve to the Elasticsearch service
  port 9200
  logstash_format true
</match>

Fix 3: Buffer Overflow

# [error]: buffer space is not enough
# (Fluentd cannot write logs fast enough to Elasticsearch)

RIGHT — increase buffer size:

<buffer>
  @type file
  path /var/log/fluentd/buffer
  flush_interval 5s
  chunk_limit_size 8MB
  total_limit_size 1GB
</buffer>

Fix 4: Log Parsing Errors

# [warn]: pattern not matched: "2024-03-15 10:30:00 some log line"
# (the log format does not match the parser)

RIGHT — use the Kubernetes CRI parser for containerd:

<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd/containers.log.pos
  tag kubernetes.*
  <parse>
    @type cri  # for containerd runtime
    # or @type json for Docker JSON log format
  </parse>
</source>

Fix 5: Fluentd DaemonSet Not Scheduled on All Nodes

kubectl get pods -n logging -o wide
# fluentd-xxxxx    Running   worker-1
# (not running on worker-2 or worker-3)

RIGHT — ensure the DaemonSet tolerates node conditions:

spec:
  tolerations:
  - operator: Exists  # tolerate all taints

Fix 6: Too Many Open Files

# [error]: too many open files

RIGHT — increase the file limit in the Fluentd container:

spec:
  containers:
  - name: fluentd
    resources:
      limits:
        memory: 512Mi
    env:
    - name: FLUENTD_SYSTEMD_CONF
      value: disable

Use DodaTech's Log Pipeline Analyzer to trace log flow from container to storage, identify bottlenecks, and optimize Fluentd buffer configurations.

Prevention

• Mount /var/log and /var/lib/docker/containers for log access.
• Set appropriate buffer sizes for your log volume.
• Monitor Fluentd metrics with Prometheus.
• Use the correct parser for your container runtime (CRI or JSON).
• Test log forwarding in a staging environment first.

Common Mistakes with fluentd logging

1. Misunderstanding that String is [Char] with poor performance for large text operations
2. Using foldl instead of foldl' causing stack overflow on large lists
3. Forgetting deriving (Show, Eq) on custom data types needed for debugging

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ