Grafana Loki Label Model Cardinality Explosion Fix

In this tutorial, you'll learn about Grafana Loki Label Model Cardinality Explosion Fix. We cover key concepts, practical examples, and best practices.

Your Grafana Loki cluster slows down and struggles with queries — label_names() returns thousands of unique values for a single label. High-cardinality labels like user_id, request_id, or ip_address create too many streams, degrading query performance and increasing storage.

The Problem

# Promtail config with high-cardinality labels
scrape_configs:
  - job_name: myapp
    static_configs:
      - targets: [localhost]
        labels:
          job: myapp
          user_id: __user_id__  # WRONG: user_id is high-cardinality
          request_id: __request_id__  # WRONG: each request is unique

Each unique user_id creates a new stream in Loki. With 10,000 users, this creates 10,000 streams — overwhelming the ingester.

Step-by-Step Fix

1. Identify high-cardinality labels

// Check label cardinality
max_over_time(
  count_over_time({job="myapp"} |= "error" [1h])
) by (user_id)

// Use Grafana's label explorer to see value counts
// Labels with 1000+ unique values are high-cardinality

2. Restructure labels to low cardinality

# Promtail config with fixed labels
scrape_configs:
  - job_name: myapp
    static_configs:
      - targets: [localhost]
        labels:
          job: myapp                # Low cardinality (1-10 values)
          env: production            # Low cardinality (3-5 values)
          service: web               # Low cardinality (5-20 values)
          level: info                # Low cardinality (5 values)

3. Move high-cardinality values to structured metadata

# Promtail config with structured metadata
scrape_configs:
  - job_name: myapp
    pipeline_stages:
      - json:
          expressions:
            user_id: user_id
            request_id: request_id
      - structured_metadata:
          user_id:     # Structured metadata, not labels
          request_id:  # Structured metadata, not labels

4. Query with structured metadata

{job="myapp", env="production"} |= "error" | user_id = "12345"

// Structured metadata is searchable but not indexed as labels
// More efficient than creating a stream per user_id

5. Monitor label cardinality

// Count unique label values
count by (__label_name__) (
  label_replace(
    {job="myapp"},
    "__label_name__",
    "$1",
    "",
    "(.+)"
  )
)

// Set alerts in Grafana for cardinality spikes
// Action: Investigate new labels or values

Expected output after fix:

Before: 10,000 streams (user_id label)
After: 5 streams (job, env, service labels)
Query performance: improved 20x

Prevention Tips

• Keep labels to low-cardinality values (<100 unique values per label)
• Use structured metadata for dynamic fields (user_id, request_id, IP)
• Monitor label cardinality in Grafana with Loki label explorer
• Set up alerting when new labels appear unexpectedly
• Restrict label creation with Loki's per-stream limit configuration

Common Mistakes with loki label

1. Overlapping type class instances that cause GHC to reject the program with ambiguous dispatch errors
2. Non-exhaustive pattern matches that compile with warnings then crash at runtime
3. Misunderstanding that String is [Char] with poor performance for large text operations

These mistakes appear frequently in real-world GRAFANA code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ