Centralized Logging: Aggregating Logs with ELK Stack and Loki

In this tutorial, you will learn about Centralized Logging: Aggregating Logs with ELK Stack and Loki. We cover key concepts, practical examples, and best practices to help you master this topic.

Centralized log aggregation collects logs from multiple services and instances into a single searchable platform. The ELK stack (Elasticsearch, Logstash, Kibana) and Grafana Loki are the most popular solutions, enabling full-text search, filtering, dashboards, and alerting across all logs.

flowchart TB
    Service1[Service 1] -->|Filebeat| Logstash[Logstash]
    Service2[Service 2] -->|Filebeat| Logstash
    Service3[Service 3] -->|Fluentd| Logstash
    Logstash -->|Parse & Transform| Elasticsearch[(Elasticsearch)]
    Elasticsearch --> Kibana[Kibana Dashboard]
    Elasticsearch --> Alert[Alerting]
    Elasticsearch --> Search[Log Search]
    
    Service1 -.->|Promtail| Loki[(Grafana Loki)]
    Service2 -.->|Promtail| Loki
    Loki --> Grafana[Grafana Dashboard]

What You'll Learn

• ELK stack architecture: Filebeat, Logstash, Elasticsearch, Kibana
• Grafana Loki for cost-effective log aggregation
• Log shipping configuration and index management
• Log search optimization and retention policies

Why It Matters

Without centralized logging, you must SSH into each server and grep log files. This does not scale beyond a few instances. Centralized logging enables searching across all services, correlating events by time, and setting up automated alerting.

Real-World Use

A 50-microservice platform uses Loki for log aggregation. Each service emits structured JSON logs to stdout. Promtail (log shipper) collects and sends them to Loki. The operations team uses Grafana Explore to search across all services by correlationId or userId.

Centralized Logging Implementation

Log Shipping with Filebeat Configuration

# filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    json.keys_under_root: true
    json.overwrite_keys: true
    json.add_error_key: true
    multiline:
      pattern: '^\{"timestamp'
      negate: true
      match: after

output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: "filebeat"
  password: "${ES_PASSWORD}"
  index: "app-logs-%{+yyyy.MM.dd}"
  ssl.verification_mode: none

setup.kibana:
  host: "https://kibana:5601"

Expected output:

Filebeat ships logs from /var/log/app/*.log to Elasticsearch. Multi-line JSON logs (error stacks) are combined. Index is per day.

Logstash Pipeline for Parsing

# logstash.conf
input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/logstash/certs/logstash.crt"
    ssl_key => "/etc/logstash/certs/logstash.key"
  }
}

filter {
  # Parse JSON logs
  if [fields][format] == "json" {
    json {
      source => "message"
      target => "parsed"
    }
    date {
      match => ["[parsed][timestamp]", "ISO8601"]
      target => "@timestamp"
    }
    mutate {
      remove_field => ["message"]
    }
  }

  # Add geoip for IP addresses
  if [parsed][ip] {
    geoip {
      source => "[parsed][ip]"
      target => "geo"
    }
  }

  # Add environment tag
  mutate {
    add_field => { "environment" => "%{[fields][environment]}" }
  }
}

output {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    index => "app-logs-%{+yyyy.MM.dd}"
    user => "logstash"
    password => "${ES_PASSWORD}"
  }
}

Expected output:

Logstash parses JSON logs, extracts timestamp, adds geoip for IPs, tags with environment, and indexes into daily Elasticsearch indices.

Loki Configuration with Promtail

# promtail.yml
scrape_configs:
  - job_name: app-logs
    static_configs:
      - targets: [localhost]
        labels:
          job: app-logs
          service: api-gateway
          environment: production
          __path__: /var/log/app/*.log

  - job_name: container-logs
    pipeline_stages:
      - json:
          expressions:
            level: level
            service: service
            correlationId: correlationId
            userId: userId
      - labels:
          level:
          service:
    static_configs:
      - targets: [localhost]
        labels:
          job: containers
          __path__: /var/lib/docker/containers/*/*-json.log

Expected output:

Promtail ships container logs to Loki. JSON fields (level, service, correlationId) are extracted as labels for fast filtering. Log content is stored as compressed chunks.

Log Search Optimization

// Elasticsearch query for log search
const searchQuery = {
  query: {
    bool: {
      must: [
        { match: { level: 'error' } },
        { range: { '@timestamp': { gte: 'now-1h', lte: 'now' } } }
      ],
      filter: [
        { term: { 'service.keyword': 'payment-service' } },
        { term: { 'environment.keyword': 'production' } }
      ],
      should: [
        { match: { 'parsed.userId': '123' } },
        { match: { 'parsed.correlationId': 'abc-def' } }
      ],
      minimum_should_match: 1
    }
  },
  sort: [{ '@timestamp': 'desc' }],
  size: 100,
  _source: ['parsed.message', 'parsed.level', 'parsed.error', '@timestamp', 'parsed.correlationId']
};

// Log retention policy (ILM)
const ilmPolicy = {
  policy: {
    phases: {
      hot: { min_age: '0ms', actions: { rollover: { max_size: '50GB', max_age: '1d' } } },
      warm: { min_age: '1d', actions: { readonly: {} } },
      cold: { min_age: '30d', actions: {} },
      delete: { min_age: '90d', actions: { delete: {} } }
    }
  }
};

Expected output:

Search for errors in payment-service in the last hour with userId=123 returns 15 results.
Indices are rolled daily at 50GB, moved to warm after 1 day, cold after 30 days, deleted after 90 days.

Common Mistakes

• Storing logs indefinitely — log volume grows exponentially. Define retention policies (30-90 days typical).
• Not structuring log indices — daily indices are easier to manage than a single massive index.
• Indexing every field without mapping — dynamic field mapping can cause mapping explosions from high-cardinality fields.
• Not using log labels in Loki — labels enable efficient querying. Too many labels increase memory usage; too few make queries slow.
• Searching without time range filters — full-index searches are slow and expensive. Always include a time range.

Practice Questions

1. What is the difference between ELK and Loki?
2. How does log shipping work with Filebeat?
3. Why is index lifecycle management important?
4. What are Loki labels and how do they affect query performance?
5. How do you set up alerting based on log patterns?

Challenge

Set up a centralized logging stack with Docker Compose. Include: (1) Elasticsearch with ILM (30-day retention), (2) Logstash for parsing JSON logs, (3) Kibana for search and dashboards, (4) Filebeat for log shipping from a sample Node.js app, (5) create a dashboard showing error rate over time.

FAQ

{{< faq "How do I search logs by correlation ID?" "Include correlationId as a field in your structured log. In Kibana: parsed.correlationId: "abc-123". In Loki: use correlationId as a label or use |= "abc-123" filter." >}}

Mini Project

Deploy a local ELK stack with Docker Compose. Configure: (1) Elasticsearch with ILM policy, (2) Logstash to parse JSON logs, (3) Kibana dashboard showing request volume, error rate, and p99 latency, (4) Filebeat to ship logs from a sample API. Generate 1000 requests and verify logs appear in Kibana.

What's Next

Continue to Contextual Logging for logging with request/session context.