Distributed Tracing — Tracking Requests Across Microservices
In this tutorial, you will learn about Distributed Tracing. We cover key concepts, practical examples, and best practices to help you master this topic.
Distributed tracing tracks a single request as it travels through multiple Microservices by propagating trace context and recording spans, enabling end-to-end latency analysis and debugging.
What You'll Learn
By the end of this lesson you will understand trace context propagation, create and manage spans, implement distributed tracing with OpenTelemetry, correlate logs across services, and visualize traces to identify performance bottlenecks.
Why It Matters
In a monolithic application, a single log file shows the entire request flow. In microservices, a single request may touch 10+ services across different hosts. Without distributed tracing, diagnosing a slow request requires manually correlating logs from multiple services, which is nearly impossible.
Real-World Use
DodaZIP uses OpenTelemetry for distributed tracing across all services. When a user uploads a file, a single trace spans the API Gateway, upload service, virus scanner, compression worker, and storage service. Engineers can see exactly which step is slow for any given request.
flowchart LR
A[Client] -->|Trace ID: abc123| B[Gateway]
B --> C[User Service]
B --> D[File Service]
D --> E[Compression]
D --> F[Storage]
subgraph Trace
G[Span 1: Gateway]
H[Span 2: File Upload]
I[Span 3: Compression]
J[Span 4: Storage]
end
style G fill:#2d3748,color:#fff
style H fill:#2d3748,color:#fff
style I fill:#2d3748,color:#fff
style J fill:#2d3748,color:#fff
Trace Context Propagation
How trace information travels between services.
# trace_context.py
# Trace context propagation
def trace_context():
print("Trace Context Propagation")
print("=" * 40)
print()
propagation_code = """
import uuid
import time
class TraceContext:
"""Trace context that propagates across service boundaries."""
def __init__(self, trace_id=None, parent_span_id=None):
self.trace_id = trace_id or self._generate_trace_id()
self.span_id = self._generate_span_id()
self.parent_span_id = parent_span_id
self.start_time = time.time()
def _generate_trace_id(self):
return uuid.uuid4().hex # 32 hex chars
def _generate_span_id(self):
return uuid.uuid4().hex[:16] # 16 hex chars
def to_headers(self):
"""Convert to HTTP headers for propagation."""
return {
"X-Trace-ID": self.trace_id,
"X-Span-ID": self.span_id,
"X-Parent-Span-ID": self.parent_span_id or "",
}
@classmethod
def from_headers(cls, headers):
"""Extract trace context from incoming headers."""
trace_id = headers.get("X-Trace-ID")
parent_span_id = headers.get("X-Span-ID")
if trace_id:
return cls(trace_id=trace_id, parent_span_id=parent_span_id)
return cls() # Start new trace
# Propagation via HTTP headers
def incoming_request_middleware(request):
# Extract context from incoming headers
ctx = TraceContext.from_headers(request.headers)
request.trace_context = ctx
print(f"Trace: {ctx.trace_id[:12]}..., Span: {ctx.span_id[:8]}...")
return ctx
def outgoing_request_middleware(request, trace_context):
# Inject context into outgoing request
headers = trace_context.to_headers()
for key, value in headers.items():
request.headers[key] = value
"""
print(propagation_code)
trace_context()
Span Creation and Management
Creating and recording spans.
# span_management.py
# Span creation and management
def span_management():
print("Span Creation and Management")
print("=" * 40)
print()
code = """
import time
import json
class Span:
"""Represents a single unit of work within a trace."""
def __init__(self, name, trace_context, service_name,
kind="internal"):
self.name = name
self.trace_id = trace_context.trace_id
self.span_id = trace_context.span_id
self.parent_span_id = trace_context.parent_span_id
self.service_name = service_name
self.kind = kind
self.start_time = None
self.end_time = None
self.status = "ok"
self.attributes = {}
self.events = []
def start(self):
self.start_time = time.time()
print(f" Span START: [{self.service_name}] {self.name}")
return self
def end(self, status="ok"):
self.end_time = time.time()
self.status = status
duration = (self.end_time - self.start_time) * 1000
print(f" Span END: [{self.service_name}] {self.name} "
f"({duration:.1f}ms) - {status}")
def set_attribute(self, key, value):
self.attributes[key] = value
def add_event(self, name, attributes=None):
self.events.append({
"name": name,
"timestamp": time.time(),
"attributes": attributes or {}
})
def to_dict(self):
return {
"name": self.name,
"trace_id": self.trace_id,
"span_id": self.span_id,
"parent_span_id": self.parent_span_id,
"service": self.service_name,
"kind": self.kind,
"start_time": self.start_time,
"end_time": self.end_time,
"duration_ms": (
(self.end_time - self.start_time) * 1000
if self.end_time else None
),
"status": self.status,
"attributes": self.attributes,
"events": self.events
}
class Tracer:
"""Manages span lifecycle."""
def __init__(self, service_name, span_exporter):
self.service_name = service_name
self.exporter = span_exporter
self.active_spans = {} # trace_id -> [spans]
def start_span(self, name, trace_context=None, kind="internal"):
ctx = trace_context or TraceContext()
span = Span(name, ctx, self.service_name, kind)
span.start()
if ctx.trace_id not in self.active_spans:
self.active_spans[ctx.trace_id] = []
self.active_spans[ctx.trace_id].append(span)
return span
def end_span(self, span, status="ok"):
span.end(status)
self.exporter.export(span)
"""
print(code)
span_management()
OpenTelemetry Implementation
Real-world tracing with OpenTelemetry.
# opentelemetry_impl.py
# OpenTelemetry implementation
def opentelemetry_impl():
print("OpenTelemetry Distributed Tracing")
print("=" * 45)
print()
code = """
# OpenTelemetry setup with Python
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.propagators.jaeger import JaegerPropagator
# Configure tracer provider
provider = TracerProvider(
resource=Resource.create({
"service.name": "order-service",
"service.version": "1.2.3",
"deployment.environment": "production"
})
)
# Add OTLP exporter (sends to Jaeger, Zipkin, or Grafana Tempo)
otlp_exporter = OTLPSpanExporter(
endpoint="http://otel-collector:4317",
insecure=True
)
provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
# Set global tracer
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
# Auto-instrument Flask
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
# Auto-instrument HTTP client
RequestsInstrumentor().instrument()
# Manual span creation
def process_order(order_id):
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
span.add_event("order.validation.started")
# Child span for payment
with tracer.start_as_current_span("process_payment") as child:
child.set_attribute("payment.amount", 59.99)
# ... payment logic ...
# Child span for inventory update
with tracer.start_as_current_span("update_inventory") as child:
# ... inventory logic ...
pass
span.add_event("order.processed")
"""
print(code)
opentelemetry_impl()
Log Correlation
Correlating logs with trace context.
# log_correlation.py
# Log correlation with traces
def log_correlation():
print("Log Correlation with Distributed Tracing")
print("=" * 45)
print()
code = """
import logging
import json
class TraceAwareLogger:
"""Logger that includes trace context in all log entries."""
def __init__(self, name, tracer):
self.logger = logging.getLogger(name)
self.tracer = tracer
def _get_trace_context(self):
span = trace.get_current_span()
if span:
ctx = span.get_span_context()
return {
"trace_id": format(ctx.trace_id, "032x"),
"span_id": format(ctx.span_id, "016x"),
}
return {}
def info(self, message, **extra):
ctx = self._get_trace_context()
self.logger.info(
message,
extra={"trace": json.dumps(ctx), **ctx, **extra}
)
def error(self, message, **extra):
ctx = self._get_trace_context()
self.logger.error(
message,
extra={"trace": json.dumps(ctx), **ctx, **extra}
)
# Structured logging output
# {
# "timestamp": "2026-06-28T10:30:00Z",
# "level": "INFO",
# "service": "order-service",
# "message": "Payment processed successfully",
# "trace_id": "a1b2c3d4e5f6...",
# "span_id": "a1b2c3d4e5f6...",
# "order_id": "ORD-12345",
# "amount": 59.99
# }
# Query example (in Loki / Elasticsearch):
# {service="order-service"} | json | trace_id="a1b2c3d4e5f6..."
# This returns ALL log entries from ALL services for this trace
"""
print(code)
log_correlation()
Common Mistakes
Not propagating trace context: If a service does not forward trace headers to downstream services, the trace breaks into disconnected segments. Every HTTP/gRPC client must propagate trace context.
Sampling rate too high: Tracing every request generates massive data volume. Sample 1-10% of requests in production. Increase sampling for specific endpoints or error traces.
Missing span attributes: A span without attributes (order ID, customer ID, error details) is hard to debug. Always add relevant business context to spans.
Ignoring async boundaries: Trace context is lost when messages are published to queues and consumed later. Include trace context in message headers and create a new span on the consumer side.
No exporter batching: Exporting spans synchronously on every request adds latency. Use batch span processors that send spans in the background.
Practice Questions
What is a trace in distributed tracing? A trace represents the entire journey of a single request as it travels through multiple services, composed of multiple spans.
What is a span? A named, timed operation representing a Unit of Work within a trace (e.g., a database query, an HTTP call to another service).
How is trace context propagated across HTTP calls? Via HTTP headers (X-Trace-ID, X-Span-ID, X-Parent-Span-ID) or W3C Trace Context headers (traceparent, tracestate).
Why is sampling important in distributed tracing? Tracing every request generates too much data. Sampling collects enough traces for analysis while keeping storage and cost manageable.
Challenge: Implement distributed tracing for a three-service application (api-gateway, user-service, order-service). Use OpenTelemetry with header propagation, create meaningful spans with attributes, correlate logs with trace IDs, and export to a tracing backend.
FAQ
Mini Project
Implement distributed tracing for a ride-sharing application with three services: API gateway, ride service, and payment service. Propagate trace context via HTTP headers, create spans for each operation (request handling, database queries, external API calls), correlate all logs with trace IDs, and export traces to Jaeger.
def tracing_design():
print("Ride-Sharing Distributed Tracing Design")
print("=" * 45)
print()
print("Services and Spans:")
print()
print("API Gateway:")
print(" Span: gateway.request - Incoming request handling")
print(" Span: gateway.proxy - Forward to ride service")
print()
print("Ride Service:")
print(" Span: ride.create - Create ride record")
print(" Span: ride.db.query - Database insert")
print(" Span: ride.dispatch - Find nearby driver")
print(" Span: ride.payment - Call payment service")
print()
print("Payment Service:")
print(" Span: payment.authorize - Authorize payment")
print(" Span: payment.process - Process charge")
print()
print("Context Propagation:")
print(" Gateway -> Ride: HTTP headers")
print(" Ride -> Payment: gRPC metadata")
print()
print("Sampling: 10% of all rides, 100% of error rides")
tracing_design()
What's Next
Next: Async Communication SQS/Kafka for async messaging with AWS SQS and Kafka.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro