Logging Best Practices: Structured Logging with ELK and Loki

In this tutorial, you'll learn about Logging Best Practices: Structured Logging with ELK and Loki. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

What You Will Learn

This tutorial teaches you structured logging principles, how to configure the ELK Stack (Elasticsearch, Logstash, Kibana) and Grafana Loki for centralized log aggregation, and how to write searchable, actionable log entries.

Why It Matters

Unstructured logs are noise. When an incident happens, you need to search, filter, and correlate log entries across dozens of services in seconds. Centralized logging turns firefighting into data-driven debugging.

Real-World Use

When a DodaTech user reported that file sync stopped working, the team searched Loki for "ERROR sync" across all services, filtered by the affected user ID, and found a database connection timeout in the indexer service within 30 seconds. Without centralized logging, that investigation would have taken hours.

Structured logging means writing log entries in a machine-parseable format (usually JSON) instead of plain text. Each log entry has consistent fields like @timestamp, level, service, message, and trace_id. This structure lets tools like ELK Stack and Loki index and query logs efficiently.

Prerequisites

• A server or local machine with Docker installed
• Basic understanding of JSON format
• Familiarity with Prometheus Introduction is helpful but not required
• Ports 5601, 9200, 3100 accessible

Step-by-Step Tutorial

Step 1: Deploy the ELK Stack with Docker Compose

Create docker-compose.yml:

version: "3.8"
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - "9200:9200"

  logstash:
    image: docker.elastic.co/logstash/logstash:8.13.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - "5000:5000"

  kibana:
    image: docker.elastic.co/kibana/kibana:8.13.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200

Step 2: Configure Logstash

Create logstash.conf:

input {
  tcp {
    port => 5000
    codec => json
  }
}

filter {
  if [level] {
    mutate {
      add_tag => ["structured"]
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
}

Step 3: Write Structured Logs in Python

import json
import logging
import socket
import datetime

class StructuredLogger:
    def __init__(self, service_name, host="localhost", port=5000):
        self.service_name = service_name
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.sock.connect((host, port))

    def log(self, level, message, **extra):
        record = {
            "@timestamp": datetime.datetime.utcnow().isoformat() + "Z",
            "level": level.upper(),
            "service": self.service_name,
            "message": message,
            **extra
        }
        self.sock.sendall((json.dumps(record) + "\n").encode())

logger = StructuredLogger("my-app")
logger.log("INFO", "User registered", user_id=42, email="test@example.com")
logger.log("ERROR", "Database timeout", query="SELECT * FROM users", duration_ms=12000)

Expected output: Log entries appear in Kibana with all fields indexed and searchable.

Step 4: Deploy Grafana Loki as an Alternative

services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:3.0.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yaml

Create promtail-config.yaml:

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*.log

Step 5: Query Logs in Kibana

1. Open http://localhost:5601
2. Go to Analytics > Discover
3. Create an index pattern app-logs-*
4. Enter a query: level: ERROR
5. Filter by service: service: "my-app"

Step 6: Query Logs in Loki with LogQL

{service="my-app"} |= "ERROR"

{job="varlogs"} | json | level = "ERROR"

rate({service="my-app"} |= "ERROR" [5m])

Step 7: Use Correlation IDs Across Services

import uuid

def generate_correlation_id():
    return str(uuid.uuid4())

# In the entrypoint service
correlation_id = generate_correlation_id()
logger.log("INFO", "Request started", correlation_id=correlation_id)

# Pass correlation_id via HTTP headers to downstream services

Step 8: Set Up Log Retention Policies

For Elasticsearch:

curl -X PUT "localhost:9200/_ilm/policy/log_retention" -H "Content-Type: application/json" -d '{
  "policy": {
    "phases": {
      "hot": {"min_age": "0d", "actions": {"rollover": {"max_size": "50GB"}},
      "delete": {"min_age": "30d", "actions": {"delete": {}}
    }
  }
}'

For Loki, configure retention_period in loki.yaml:

table_manager:
  retention_deletes_enabled: true
  retention_period: 720h

Learning Path

flowchart LR
    A[Structured Logging] --> B[Centralized Collection]
    B --> C{Storage Backend}
    C --> D[ELK Stack]
    C --> E[Grafana Loki]
    D --> F[Kibana Dashboards]
    E --> G[Grafana Explore]
    F --> H[Log Analysis]
    G --> H
    style A fill:#4a90d9,color:#fff
    style H fill:#e67e22,color:#fff

Common Errors

1. No logs appear in Kibana -- The index pattern is wrong or Logstash is not receiving logs. Verify Logstash logs for connection errors.

2. JSON parsing errors in Logstash -- The log entry is not valid JSON. Use a JSON linter on your log output before debugging.

3. Promtail cannot read log files -- The container does not have permission to read the log directory. Mount volumes with correct permissions.

4. Loki returns 429 Too Many Requests -- The ingestion rate exceeds the configured limits. Increase ingestion_rate_mb in the Loki config.

5. Correlation IDs are missing from some logs -- The correlation ID is not propagated through the entire request chain. Verify header forwarding in API gateways.

6. High disk usage from Elasticsearch -- No index lifecycle policy is applied. Configure ILM to roll over and delete old indices.

7. Kibana dashboards show no data after restart -- Elasticsearch data persists in volumes. Ensure Docker volumes are mounted correctly.

Practice Questions

1. What is structured logging? Answer: Writing log entries in a consistent, machine-parseable format (usually JSON) with fixed fields instead of plain text.

2. What are the main components of the ELK Stack? Answer: Elasticsearch (storage and search), Logstash (ingestion and transformation), Kibana (visualization).

3. How does Grafana Loki differ from Elasticsearch? Answer: Loki indexes only metadata labels and stores log content in compressed chunks, making it cheaper and faster for log-specific workloads.

4. Why should you include a correlation ID in every log entry? Answer: To trace a single request across all services and correlate log entries from different components.

5. What is the purpose of an index lifecycle policy in Elasticsearch? Answer: To automatically manage indices over time -- roll over at a certain size and delete after a retention period.

Challenge

Set up a complete structured logging pipeline for a three-tier application. Write a Python script that generates structured logs with levels (INFO, WARN, ERROR), service names, correlation IDs, and request durations. Send these logs to both ELK and Loki simultaneously. Create a Kibana dashboard showing error rates by service and a Grafana dashboard using Loki as the data source. Configure a 7-day retention policy on both backends. Write a LogQL query that finds the top 5 slowest requests in the last hour.

FAQ