Centralized Logging — Loki, ELK Stack, and Fluentd for Log Aggregation and Analysis

In this tutorial, you'll learn about Centralized Logging. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Centralized logging aggregates logs from all services and infrastructure into a single platform, enabling search, correlation, alerting, and historical analysis that is impossible when logs are scattered across individual servers or containers.

What You'll Learn

Why It Matters

When an incident occurs, the first question is always "what happened?" Without centralized logging, operators SSH into servers, run journalctl or docker logs, and manually piece together a timeline from scattered sources. This takes 30 minutes during an outage when every second counts. Centralized logging collects all logs -- application, system, container, cloud -- into one searchable store with structured metadata, enabling 10-second investigations by filtering on service name, error code, trace ID, or time range.

Real-World Use

DodaTech ingests 500GB of logs daily from Durga Antivirus Pro's 200+ services into a Loki cluster. Logs are structured JSON with trace IDs, correlated with Prometheus metrics, and queried through Grafana. Incident responders search across all services in seconds using LogQL.

flowchart TD
    A["Application Pods"] --> B["Promtail DaemonSet"]
    C["Kubernetes API Server"] --> B
    D["Node System Logs"] --> B
    B --> E["Loki: Log Storage"]
    E --> F["Grafana: Log Explorer"]
    B --> G["Fluentd Aggregator"]
    G --> H["Elasticsearch"]
    H --> I["Kibana"]
    E --> J["LogQL Queries"]
    J --> K["Alerting Rules"]
    K --> L["Alertmanager"]
    style B fill:#269539,color:#fff
    style E fill:#326CE5,color:#fff
    style G fill:#F46800,color:#fff

Structured Logging

Structured logs in JSON format enable powerful querying and filtering. Unstructured text logs are nearly impossible to query effectively at scale.

# Bad: unstructured log
2024-06-22 10:00:00 ERROR User login failed for user@example.com

# Good: structured JSON log
{"timestamp":"2024-06-22T10:00:00Z","level":"error","service":"auth-api","trace_id":"abc123","user_id":"user@example.com","message":"User login failed","error":"invalid_password","duration_ms":45,"status_code":401}

// Node.js structured logging with pino
const pino = require("pino");
const logger = pino({
  level: process.env.LOG_LEVEL || "info",
  formatters: {
    level(label) {
      return { level: label };
    },
  },
  serializers: {
    err: pino.stdSerializers.err,
    req: pino.stdSerializers.req,
    res: pino.stdSerializers.res,
  },
  timestamp: pino.stdTimeFunctions.isoTime,
});

app.get("/api/users/:id", async (req, res) => {
  const start = Date.now();
  try {
    const user = await db.findUser(req.params.id);
    logger.info({ userId: req.params.id, duration: Date.now() - start }, "User fetched");
    res.json(user);
  } catch (err) {
    logger.error({ err, userId: req.params.id }, "Failed to fetch user");
    res.status(500).json({ error: "Internal server error" });
  }
});

Expected behavior: Every log line is a JSON object with structured fields: level, service, trace_id, request context, and timing. LogQL queries can filter on any field: {service="auth-api"} |= "error" | json | user_id = "user@example.com".

Loki with Promtail

Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It indexes metadata (labels) rather than full text, making it more cost-effective than Elasticsearch for high-volume logging.

# promtail-config.yaml
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - cri: {}
      - regex:
          expression: "^(?s)(?P<content>.*)$"
      - json:
          expressions:
            level: level
            service: service
            trace_id: trace_id
      - labels:
          level:
          service:
      - drop:
          source: "level"
          value: "debug"
    relabel_configs:
      - source_labels: ["__meta_kubernetes_pod_label_app"]
        target_label: app
      - source_labels: ["__meta_kubernetes_pod_label_component"]
        target_label: component
      - source_labels: ["__meta_kubernetes_namespace"]
        target_label: namespace
      - source_labels: ["__meta_kubernetes_pod_node_name"]
        target_label: node

Expected behavior: Promtail reads container logs from /var/log/pods/*, parses structured JSON fields, creates Loki labels from level and service, drops debug-level logs to save storage, and attaches Kubernetes metadata (namespace, app, node) as labels.

# LogQL queries for log analysis

# Find all errors in the auth service in the last hour
{service="auth-api", level="error"} |= "login" |= "failed"

# Show error rate per service over time
sum by (service) (
  rate({level="error"}[5m])
)

# Find a specific trace across all services
{trace_id="abc123def456"}

# Correlate with metrics: count of errors with status 500
sum by (service) (
  count_over_time({level="error"} |= "500" [1h])
)

ELK Stack with Filebeat

The ELK stack (Elasticsearch, Logstash, Kibana) provides full-text search, powerful aggregations, and rich visualizations for log analysis.

# filebeat.yml
filebeat.inputs:
  - type: container
    paths:
      - "/var/log/containers/*.log"
    processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
      - decode_json_fields:
          fields: ["message"]
          target: ""
          overwrite_keys: true
      - drop_event:
          when:
            equals:
              level: "debug"

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: ${ELASTICSEARCH_USER}
  password: ${ELASTICSEARCH_PASSWORD}
  ssl.verification_mode: certificate
  index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"

setup.kibana:
  host: "https://kibana:5601"
  username: ${ELASTICSEARCH_USER}
  password: ${ELASTICSEARCH_PASSWORD}

Expected behavior: Filebeat reads container logs, enriches them with Kubernetes metadata (pod name, namespace, node), decodes JSON log messages into structured Elasticsearch fields, filters out debug-level logs, and sends them to Elasticsearch daily indices.

// Kibana search query (DSL)
{
  "query": {
    "bool": {
      "must": [
        {"term": {"level.keyword": "error"}},
        {"term": {"service.keyword": "auth-api"}},
        {"range": {"@timestamp": {"gte": "now-1h"}}
      ]
    }
  },
  "aggs": {
    "errors_by_user": {
      "terms": {"field": "user_id.keyword", "size": 10}
    }
  }
}

Logging Architecture Comparison

Fluentd as a Unified Logging Layer

Fluentd acts as a unified logging layer that collects from multiple sources, transforms data, and routes to multiple outputs.

# fluentd.conf
<source>
  @type tail
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  format json
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  read_from_head true
</source>

<filter kubernetes.**>
  @type kubernetes_metadata
</filter>

<filter kubernetes.**>
  @type stdout
  @id stdout_output
</filter>

<match kubernetes.**>
  @type copy
  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix kubernetes-logs
    user "#{ENV['ES_USER']}"
    password "#{ENV['ES_PASSWORD']}"
  </store>
  <store>
    @type s3
    s3_bucket dodatech-logs-archive
    s3_region us-east-1
    path logs/${year}/${month}/${day}/
    <format>
      @type json
    </format>
    <buffer>
      @type file
      path /var/log/fluentd-s3-buffer
      flush_interval 300s
    </buffer>
  </store>
</match>

Expected behavior: Fluentd tails all container logs, enriches with Kubernetes metadata, sends to Elasticsearch for real-time search and Kibana visualization, and simultaneously archives raw JSON to S3 for long-term Compliance storage.

Common Errors

1. Logging to stdout without structured format: Applications that log plain text strings ("User 12345 logged in") cannot be parsed or filtered programmatically. Always log in structured JSON format with consistent field names across all services.

2. Not handling log rotation: Container logs grow indefinitely without rotation, filling the disk. Docker has built-in log rotation but defaults are too generous (unlimited). Configure max-size: 10m and max-file: 3 in the Docker daemon or container logging driver.

3. Sending too many labels to Loki: Loki indexes every label. Adding a label with high cardinality (like user_id, trace_id, or request_id) creates millions of index entries and degrades performance. Use structured logging with JSON parsing instead of labels for high-cardinality fields.

4. No log retention or archival policy: Without a retention policy, logs accumulate until disk runs out. Set retention: Loki can retain 7-30 days of hot logs with longer-term archival to S3/GCS. Define retention per log type (debug=3 days, info=7 days, error=30 days, audit=1 year).

5. Not correlating logs with traces and metrics: Logs without trace IDs cannot be correlated with requests. A single user request generates logs across 5 services. Without a shared trace_id field, investigating that request means manually searching each service's logs with time-based guessing.

Practice Questions

1. What is the main difference between Loki and Elasticsearch for log storage? Answer: Loki indexes only labels (metadata) and stores log content as compressed blocks without full-text indexing. Elasticsearch indexes the full content of every log line for fast text search. Loki is cheaper for high-volume logging but slower for arbitrary text search. Elasticsearch excels at full-text search but costs more to store.

2. Why should application logs be structured JSON rather than plain text? Answer: Structured JSON enables programmatic parsing, filtering, and querying. A plain text log "ERROR login failed" cannot be filtered by user_id, trace_id, or status_code. Structured JSON logs allow LogQL queries like {service="auth"} | json | user_id = "alice", which is impossible with plain text.

3. What is the role of Promtail in the Loki stack? Answer: Promtail is the log collector agent. It reads log files from the host or container filesystem, attaches Kubernetes metadata as labels, performs log transformations (parsing JSON, dropping debug logs), and pushes the logs to Loki. It is the equivalent of Filebeat in the ELK stack.

4. How does log correlation with trace IDs improve Incident Response? Answer: A trace ID is passed through all services handling a single request. When an error occurs, searching for that trace ID returns logs from all services involved in the request, creating a complete timeline of what happened. Without trace IDs, each service's logs must be searched independently.

Challenge

Design a centralized logging architecture for a 50-microservice platform: choose between Loki and ELK (or both) based on requirements (10TB/month log volume, 30-day hot retention, 1-year archive, full-text search needed for audit Compliance, Grafana for visualization, correlation with Prometheus metrics), configure structured JSON logging in a sample Node.js application using pino, set up Promtail or Fluentd to collect logs from Kubernetes with proper label strategy (low cardinality only), implement a log archival pipeline to S3 with 1-year retention, create a LogQL alert that fires when the error rate in the auth service exceeds 5 percent over 5 minutes, and design a Grafana dashboard that shows log volume, error rate by service, and top error messages.

Mini Project

Build a complete centralized logging platform from scratch: deploy Loki (with single binary or Microservices mode) on a Kubernetes cluster, deploy Promtail as a DaemonSet with proper configuration for structured JSON log parsing and label extraction, create structured logging in three sample applications (Node.js with pino, Python with structlog, Go with zap), configure LogQL alerts for error rate spikes and absent logs, integrate Loki as a Grafana data source and build a log exploration dashboard with template variables for service, namespace, and level, set up log-to-metrics rules that generate Prometheus metrics from log streams, configure Elasticsearch and Kibana as a secondary log store for full-text search capability, implement log archival to S3 with 30-day hot retention and 1-year cold retention, and document the architecture with a runbook for adding a new service to the logging pipeline.

Related Resources

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.