Distributed Tracing: Correlating Logs Across Microservices

In this tutorial, you will learn about Distributed Tracing: Correlating Logs Across Microservices. We cover key concepts, practical examples, and best practices to help you master this topic.

Distributed tracing tracks a single request as it flows through multiple microservices. Each service adds a span (unit of work) with timing and metadata. By correlating logs with trace IDs, you can understand the complete lifecycle of a request and identify bottlenecks.

flowchart LR
    Client -->|traceId: abc| ServiceA[Service A]
    ServiceA -->|spanId: 1| ServiceB[Service B]
    ServiceA -->|spanId: 2| ServiceC[Service C]
    ServiceB -->|spanId: 3| ServiceD[Service D]
    ServiceB -->|spanId: 4| ServiceE[Service E]
    
    subgraph Trace
        Span1["Span: Gateway (45ms)"]
        Span2["Span: Auth (12ms)"]
        Span3["Span: Orders (120ms)"]
        Span4["Span: Payments (85ms)"]
        Span5["Span: Notifications (30ms)"]
    end

What You'll Learn

• Trace context propagation (traceId, spanId, parentSpanId)
• Creating spans with OpenTelemetry
• Correlating logs with trace IDs
• Trace visualization with Jaeger

Why It Matters

In a monolith, a single log file traces a request. In microservices, a request touches 5-20 services. Without distributed tracing, you cannot see the complete picture. Tracing reduces MTTR for complex issues from hours to minutes.

Real-World Use

A payment processing request flows through: API Gateway → Auth Service → Order Service → Payment Service → Notification Service. When a payment fails, the trace shows it took 2 seconds in Payment Service (calling a slow third-party API). The team optimizes the third-party call.

Distributed Tracing Implementation

OpenTelemetry Trace Setup

const { NodeTracerProvider } = require('@opentelemetry/node');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { trace, context } = require('@opentelemetry/api');

const provider = new NodeTracerProvider();

const exporter = new JaegerExporter({
  endpoint: 'http://jaeger:14250',
  serviceName: 'order-service'
});

provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

const tracer = trace.getTracer('order-service');

Expected output:

Traces are exported to Jaeger at http://jaeger:14250. Service name appears in Jaeger UI.

Creating and Propagating Spans

const { SpanStatusCode } = require('@opentelemetry/api');

async function createOrder(orderData) {
  // Create a span for this operation
  return tracer.startActiveSpan('createOrder', async (span) => {
    try {
      span.setAttribute('orderId', orderData.id);
      span.setAttribute('userId', orderData.userId);
      span.setAttribute('amount', orderData.amount);

      // Call payment service with trace context
      const paymentResult = await callPaymentService(orderData.payment, span);

      span.setAttribute('paymentStatus', paymentResult.status);

      // Call notification service
      await callNotificationService(orderData.userId, span);

      span.setStatus({ code: SpanStatusCode.OK });
      return { success: true, orderId: orderData.id };
    } catch (err) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: err.message
      });
      span.recordException(err);
      throw err;
    } finally {
      span.end();
    }
  });
}

async function callPaymentService(paymentData, parentSpan) {
  return tracer.startActiveSpan('callPaymentService', {
    links: [{ context: parentSpan.spanContext() }]
  }, async (span) => {
    span.setAttribute('paymentMethod', paymentData.method);
    // ... call payment service
    span.end();
  });
}

Expected output:

Trace in Jaeger shows: createOrder (250ms) → callPaymentService (180ms) → callNotificationService (30ms).
Each span has attributes (orderId, userId, amount).

Log-Trace Correlation

const { trace } = require('@opentelemetry/api');

function createTraceAwareLogger(logger) {
  return {
    info: (msg, meta = {}) => {
      const span = trace.getSpan(context.active());
      const traceContext = span ? {
        traceId: span.spanContext().traceId,
        spanId: span.spanContext().spanId
      } : {};
      logger.info({ ...meta, ...traceContext }, msg);
    },
    error: (msg, meta = {}) => {
      const span = trace.getSpan(context.active());
      const traceContext = span ? {
        traceId: span.spanContext().traceId,
        spanId: span.spanContext().spanId
      } : {};
      logger.error({ ...meta, ...traceContext }, msg);
    },
    warn: (msg, meta = {}) => {
      const span = trace.getSpan(context.active());
      const traceContext = span ? {
        traceId: span.spanContext().traceId,
        spanId: span.spanContext().spanId
      } : {};
      logger.warn({ ...meta, ...traceContext }, msg);
    }
  };
}

const traceLogger = createTraceAwareLogger(logger);

// Usage - logs automatically include traceId and spanId
app.get('/api/orders/:id', async (req, res) => {
  traceLogger.info('Fetching order', { orderId: req.params.id });
  const order = await db.getOrder(req.params.id);
  traceLogger.info('Order fetched', { orderId: req.params.id, status: order.status });
});

Expected output:

{"level":"info","message":"Fetching order","traceId":"abc123...","spanId":"def456...","orderId":"123"}
{"level":"info","message":"Order fetched","traceId":"abc123...","spanId":"ghi789...","orderId":"123","status":"pending"}

Trace Context Propagation via HTTP Headers

// Trace context propagation middleware
const { propagation } = require('@opentelemetry/api');
const { W3CTraceContextPropagator } = require('@opentelemetry/core');

propagation.setGlobalPropagator(new W3CTraceContextPropagator());

app.use((req, res, next) => {
  // Extract trace context from incoming request
  const extractedContext = propagation.extract(context.active(), req.headers);
  context.with(extractedContext, () => {
    next();
  });
});

// Outgoing HTTP request propagation
const http = require('http');

function tracedHttpRequest(options) {
  return new Promise((resolve, reject) => {
    const span = tracer.startSpan(`HTTP ${options.method} ${options.hostname}`);
    const ctx = trace.setSpan(context.active(), span);

    context.with(ctx, () => {
      const req = http.request(options, (res) => {
        span.setAttribute('http.status_code', res.statusCode);
        span.end();
        resolve(res);
      });

      // Inject trace context into outgoing headers
      propagation.inject(context.active(), req.headers);

      req.on('error', (err) => {
        span.recordException(err);
        span.end();
        reject(err);
      });

      req.end();
    });
  });
}

Expected output:

Outgoing HTTP requests include traceparent header (W3C Trace Context format).
Receiving service extracts trace context and continues the trace.

Common Mistakes

• Not propagating trace context across service boundaries — without propagation, each service creates a new trace.
• Not including trace IDs in logs — log-to-trace correlation requires trace IDs in log entries.
• Creating too many spans — span overhead adds latency. Create spans for significant operations, not every function call.
• Not setting span status (OK/ERROR) — spans without status provide no indication of success or failure.
• Ignoring sampling — tracing every request at high volume generates significant data. Sample traces based on configuration.

Practice Questions

1. What is the difference between a trace and a span?
2. How does trace context propagate across services?
3. Why is log-trace correlation important?
4. What is W3C Trace Context and why is it used?
5. How does sampling affect distributed tracing?

Challenge

Set up distributed tracing for a three-service application. Use OpenTelemetry with Jaeger. Implement: (1) trace context propagation via HTTP headers, (2) spans for each service operation, (3) log-trace correlation (traceId in logs), (4) error span recording, (5) Jaeger dashboard showing request flow.

FAQ

Mini Project

Build a distributed tracing demo with three Node.js microservices. Use OpenTelemetry SDK. Implement: (1) trace context propagation via W3C headers, (2) spans for each service operation, (3) log-trace correlation, (4) Jaeger for visualization, (5) trace sampling at 50%.

What's Next

Continue to Logging Strategies for advanced logging Strategy patterns.