Grafana Tempo Search Returns No Results Fix

In this tutorial, you'll learn about Grafana Tempo Search Returns No Results Fix. We cover key concepts, practical examples, and best practices.

Your Grafana Tempo search field shows "no traces found" — the search tags are not configured, the time range has no data, or the ingester search is disabled. Tempo requires specific configuration for its search functionality beyond trace ID lookup.

The Problem

# tempo.yaml — minimal config without search
distributor: ...
ingester: ...
compactor: ...

The trace ID lookup works (/api/traces/{traceID}) but the search API returns empty. Tempo's search feature requires the ingester to index span attributes for search.

Step-by-Step Fix

1. Enable ingester search

ingester:
  lifecycler: ...
  trace_idle_period: 10s
  max_block_duration: 5m
  # Enable search on ingester data
  search:
    enabled: true

2. Configure search tags

# tempo.yaml
search_tags:
  - service.name
  - http.method
  - http.status_code
  - http.url
  - db.system
  - span.kind

3. Configure search limits

# tempo.yaml
overrides:
  defaults:
    search:
      max_duration: 24h        # Max time range for search
      max_spans_per_span_set: 100
      max_results: 10000

4. Query via Grafana Explore

# Use Grafana Explore with Tempo data source:
# 1. Select "Tempo" as data source
# 2. Select "Search" query type
# 3. Add filters:
#    - resource.service.name = "payment-service"
#    - span.http.status_code >= 400
# 4. Set time range to last 1h
# 5. Click "Run query"

5. Use TraceQL instead of search

// TraceQL is more powerful than the search API
{ resource.service.name = "payment-service" && span.http.status_code >= 400 }

Expected output:

Search results: 23 traces found in last 1h
  Trace 1: payment-service → POST /api/orders (500) 234ms
  Trace 2: payment-service → GET /api/users (401) 12ms
  ...

Prevention Tips

• Enable ingester.search.enabled for search on recent traces
• Configure search_tags with relevant span and resource attributes
• Use TraceQL for complex queries beyond simple tag matching
• Set appropriate max_duration limits for search performance
• Use trace ID lookup for known problematic traces

Common Mistakes with tempo search

1. Using head and tail instead of pattern matching, causing runtime errors on empty lists
2. Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks
3. Using return to exit a function early instead of wrapping a pure value in the monad

These mistakes appear frequently in real-world GRAFANA code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ