Grafana Loki: Log Aggregation for Cloud-Native Environments

In this tutorial, you'll learn about Grafana Loki: Log Aggregation for Cloud. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

What You Will Learn

This tutorial teaches you how to deploy Grafana Loki for log aggregation, configure Promtail as a log agent, write LogQL queries, and create log-based alerts and dashboards.

Why It Matters

Traditional log aggregation tools like Elasticsearch are expensive at scale. Loki was designed for cloud-native environments -- it indexes only metadata labels and stores compressed log content, reducing storage costs by up to 10x compared to full-text indexing solutions.

Real-World Use

The DodaTech platform ingests over 50 TB of logs per month from thousands of Microservices. Loki stores all of them with a 30-day retention at one-fifth the cost of their previous Elasticsearch cluster. Queries that used to take 30 seconds now complete in under 2 seconds.

Grafana Loki is a horizontally-scalable, highly-available log aggregation system. It is designed to be cost-effective and easy to operate. Loki does not index the content of logs -- it indexes only labels (similar to Prometheus) and stores log data in compressed chunks in object storage.

Prerequisites

• Basic knowledge of Grafana Dashboards
• A server with Docker and Docker Compose installed
• Understanding of Prometheus Introduction helps but is not required

Step-by-Step Tutorial

Step 1: Deploy Loki with Docker Compose

Create docker-compose.yml:

version: "3.8"
services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:3.0.0
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yaml:/etc/promtail/config.yaml

  grafana:
    image: grafana/grafana:11.1.0
    ports:
      - "3000:3000"

Step 2: Configure Loki

Create loki-config.yaml:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /tmp/loki/index
    cache_location: /tmp/loki/cache
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  retention_period: 744h

Step 3: Configure Promtail

Create promtail-config.yaml:

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: my-server
          __path__: /var/log/syslog

  - job_name: applications
    pipeline_stages:
      - json:
          expressions:
            level: level
            service: service
    static_configs:
      - targets:
          - localhost
        labels:
          job: app-logs
          __path__: /var/log/app/*.log

Step 4: Start the Stack

docker-compose up -d

Expected output: Three containers start. Loki on port 3100, Grafana on 3000.

Step 5: Add Loki as a Data Source in Grafana

1. Log in to Grafana at http://localhost:3000
2. Go to Configuration > Data Sources
3. Click Add data source and select Loki
4. Set URL to http://loki:3100
5. Click Save & Test

Step 6: Write Your First LogQL Query

In Grafana, go to Explore and select the Loki data source:

{job="varlogs"}

This returns all log lines from the system log.

{job="varlogs"} |= "error"

This filters for lines containing "error".

{job="app-logs"} | json | level = "ERROR"

This parses JSON logs and filters by level.

Step 7: Create Metric Queries from Logs

rate({job="app-logs"} | json | level = "ERROR" [5m])

Expected output: A time-series graph showing the per-second rate of error log lines.

sum by (service) (rate({job="app-logs"} | json | level = "ERROR" [5m]))

This breaks down error rates by service name.

Step 8: Set Up Log-Based Alerts

1. In Grafana, go to Alerting > Alert rules > New alert rule
2. Select Loki as the data source
3. Enter the query:

sum(rate({job="app-logs"} | json | level = "ERROR" [5m])) > 10

4. Set evaluation interval to 1m
5. Set condition: When A is above 10
6. Configure notification channel (Slack, email, etc.)

Step 9: Create a Log Dashboard

1. Create a new dashboard
2. Add a Logs panel with {job="app-logs"} |= "ERROR"
3. Add a Time Series panel with rate({job="app-logs"} | json | level = "ERROR" [5m])
4. Add a Stat panel with topk(5, sum by (service) (count_over_time({job="app-logs"} | json | level = "ERROR" [5m])))

Learning Path

flowchart LR
    A[Deploy Loki] --> B[Configure Promtail]
    B --> C[Ingest Logs]
    C --> D[LogQL Queries]
    D --> E[Metric Queries]
    D --> F[Log Dashboards]
    D --> G[Log Alerts]
    E --> H[Grafana Explore]
    style A fill:#4a90d9,color:#fff
    style D fill:#e67e22,color:#fff

Common Errors

1. Promtail cannot connect to Loki -- The clients.url is wrong or the Loki container is unreachable. Check Docker network and container names.

2. Logs appear in Promtail output but not in Loki -- Log parsing fails silently. Add --inspect to Promtail startup to see pipeline stage logs.

3. Loki returns 429 rate limit exceeded -- The ingestion_rate_mb in limits_config is too low for your log volume. Increase the limit.

4. LogQL parser stage returns no output -- The log lines are not valid JSON. Use \| pattern instead of \| json for unstructured logs.

5. Retention setting has no effect -- Loki requires table_manager.retention_deletes_enabled: true and the retention period must be set in the limits_config.

6. High memory usage from Loki -- The ingester flushes chunks too slowly. Reduce chunk_target_size in the ingester config.

7. Grafana shows "Data source connected, but no labels received" -- No logs have been pushed yet. Generate some test logs with logger "test".

Practice Questions

1. How does Loki differ from Elasticsearch in terms of indexing? Answer: Loki indexes only metadata labels and stores raw log content as compressed chunks. Elasticsearch full-text indexes every log field.

2. What is Promtail and what is its role? Answer: Promtail is the log collection agent that discovers log files, attaches labels, and pushes log entries to Loki.

3. What is the difference between {job="varlogs"} and {job="varlogs"} |= "error"? Answer: The first returns all log lines from the varlogs stream. The second filters to only lines containing "error".

4. How do you parse JSON log fields in LogQL? Answer: Use the | json pipeline stage, then filter by field: | json | level = "ERROR".

5. What storage backends does Loki support? Answer: Local filesystem, S3, GCS, Azure Blob Storage, and MinIO.

Challenge

Set up a complete Loki stack that collects logs from three sources: the system syslog, an Nginx access log, and a JSON-structured application log. Configure Promtail to add appropriate labels (job, host, service) to each log stream. Write a LogQL query that shows the top 5 IP addresses making the most requests from the Nginx log. Create a Grafana dashboard with one Logs panel showing real-time application errors and one Time Series panel showing the error rate per service. Configure an alert that fires when error rate exceeds 5 per second for 5 minutes.

FAQ