Grafana Tempo Service Graph Not Showing Edges Fix

In this tutorial, you'll learn about Grafana Tempo Service Graph Not Showing Edges Fix. We cover key concepts, practical examples, and best practices.

Your Grafana Tempo service graph shows nodes (services) but no edges between them — the span metrics are not generated, the service graph processor is not configured, or spans lack the required span.kind attribute for client-server relationships.

The Problem

# Tempo config missing service graph processor
distributor:
  receivers:
    otlp: ...

ingester: ...
compactor: ...

# No metrics_generator section

The service graph requires the metrics generator to process span pairs and compute edges. Without it, Tempo stores traces but never generates the relationship data.

Step-by-Step Fix

1. Enable the metrics generator

# tempo.yaml
metrics_generator:
  processor:
    service_graphs:
      enabled: true
    span_metrics:
      enabled: true
  registry:
    collection_interval: 15s
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write

2. Ensure spans have span.kind attribute

# Application code — set span.kind correctly
from opentelemetry import trace
from opentelemetry.trace import SpanKind

tracer = trace.get_tracer(__name__)

def make_request():
    # Client span — creates outgoing edge
    with tracer.start_as_current_span("http-call", kind=SpanKind.CLIENT):
        http.get("http://service-b/api")

# In Service B:
def handle_request():
    # Server span — creates incoming edge
    with tracer.start_as_current_span("handle-request", kind=SpanKind.SERVER):
        process()

3. Configure remote write target

metrics_generator:
  processor:
    service_graphs:
      enabled: true
    span_metrics:
      enabled: true
  registry:
    collection_interval: 15s
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_native_histograms: true
      - url: http://mimir:9009/api/v1/push

4. Query service graph metrics in Prometheus

// Request rate between services
sum(rate(traces_service_graph_request_total{job="tempo"}[5m])) by (client, server)

// Error rate between services
sum(rate(traces_service_graph_request_failed_total{job="tempo"}[5m])) by (client, server)

// Request duration between services
histogram_quantile(0.99,
  sum(rate(traces_service_graph_request_duration_seconds_bucket{job="tempo"}[5m])) by (le, client, server)
)

5. View in Grafana

# Install the service graph dashboard
# Grafana → Dashboards → Import → 13402 (Tempo Service Graph)

# Or build a custom panel:
# Data source: Prometheus
# Query: traces_service_graph_request_total
# Panel type: Node Graph

Expected output:

Service Graph:
  payment-service ──HTTP POST──→ order-service (120 req/s, 0.5% err)
       │                              │
       │                         SQL  │
       ▼                              ▼
  user-service ──gRPC──→ inventory-service

Prevention Tips

• Enable metrics_generator with service_graphs and span_metrics processors
• Set span.kind to CLIENT and SERVER on all inter-service calls
• Configure remote_write to a Prometheus-compatible store
• Use auto-instrumentation which sets span.kind automatically
• Verify span kind with TraceQL: { span.span.kind = 2 } (2=CLIENT, 3=SERVER)

Common Mistakes with tempo service graph

1. Forgetting deriving (Show, Eq) on custom data types needed for debugging
2. Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable
3. Using head and tail instead of pattern matching, causing runtime errors on empty lists

These mistakes appear frequently in real-world GRAFANA code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ