Grafana Tempo TraceQL Query Not Returning Spans Fix

In this tutorial, you'll learn about Grafana Tempo TraceQL Query Not Returning Spans Fix. We cover key concepts, practical examples, and best practices.

Your Grafana Tempo TraceQL query returns no spans — the query syntax is invalid, the attribute key does not exist, or the time range contains no matching traces. TraceQL has a specific syntax for selecting spans by attributes, resources, and duration.

The Problem

{ .http.status_code = 200 }

(no spans returned)

The attribute .http.status_code does not exist in the spans, or the name is wrong. Tempo uses span.http.status_code or resource.http.status_code depending on where the attribute was set.

Step-by-Step Fix

1. Use correct attribute scope

// Span attributes (set by application code)
{ span.http.status_code = 200 }

// Resource attributes (set by instrumentation)
{ resource.service.name = "payment-service" }

// Any scope (matches span or resource)
{ .http.status_code = 200 }
// Prefer explicit scope for clarity

2. Discover available attributes

// Find all span attribute keys
{ } | by(.http.status_code)

// Find all resource attribute keys
{ } | by(resource.service.name)

3. Use comparison operators correctly

// Numeric comparison
{ span.http.status_code >= 400 }

// Duration comparison
{ duration > 100ms }

// String comparison
{ resource.service.name = "payment-service" }

// Regex match
{ resource.service.name =~ "payment-.*" }

4. Combine multiple conditions

{ span.http.status_code >= 400 && duration > 500ms }

// Nested conditions
{ resource.service.name = "api" && { span.http.method = "POST" || span.http.method = "PUT" } }

5. Use rate for metric-like queries

// Rate of error spans over time
rate({ span.http.status_code >= 500 } | by(resource.service.name))

// Count spans by service
{ } | by(resource.service.name) | count()

Expected output:

{ span.http.status_code = 200 && duration > 100ms }
→ 15 spans found
  Trace ID: 1234abcd
  Service: payment-service
  Duration: 234ms
  Status: 200

Prevention Tips

• Use explicit scope (span. or resource.) when querying attributes
• Discover available attributes with by() aggregation
• Use duration > 0 to exclude empty spans
• Set reasonable time ranges in Grafana Explore
• Test simple queries first, then add conditions

Common Mistakes with tempo traceql

1. Forgetting deriving (Show, Eq) on custom data types needed for debugging
2. Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable
3. Using head and tail instead of pattern matching, causing runtime errors on empty lists

These mistakes appear frequently in real-world GRAFANA code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ