Observability Project: Building a Complete Logging and Monitoring System

In this tutorial, you will learn about Observability Project: Building a Complete Logging and Monitoring System. We cover key concepts, practical examples, and best practices to help you master this topic.

This project combines everything you have learned about logging and observability. You will build a complete observability stack for a two-service e-commerce application, including structured logging, centralized aggregation, distributed tracing, and monitoring dashboards.

flowchart TB
    subgraph Services
        API[API Gateway - Express]
        Orders[Order Service - Express]
    end
    
    subgraph Observability Stack
        Logger[Pino Logger]
        Tracer[OpenTelemetry]
        Collector[Log Collector]
    end
    
    subgraph Storage
        Loki[(Loki Logs)]
        Jaeger[(Jaeger Traces)]
        Prometheus[(Prometheus Metrics)]
    end
    
    subgraph Visualization
        Grafana[Grafana Dashboard]
        JaegerUI[Jaeger UI]
    end
    
    API --> Logger
    Orders --> Logger
    Logger --> Collector
    Collector --> Loki
    API --> Tracer
    Orders --> Tracer
    Tracer --> Jaeger
    Loki --> Grafana
    Jaeger --> JaegerUI
    Jaeger --> Grafana
    Prometheus --> Grafana

Project Requirements

1. Structured Logging with Pino

const pino = require('pino');

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
  formatters: {
    level: (label) => ({ level: label })
  },
  redact: {
    paths: ['req.headers.authorization', 'req.body.password', 'req.body.ssn'],
    censor: '[REDACTED]'
  },
  serializers: {
    req: pino.stdSerializers.req,
    res: pino.stdSerializers.res,
    err: pino.stdSerializers.err
  },
  base: {
    service: process.env.SERVICE_NAME,
    environment: process.env.NODE_ENV,
    version: process.env.APP_VERSION
  }
});

// Request logging middleware
function requestLogger(req, res, next) {
  const start = Date.now();
  res.on('finish', () => {
    logger.info({
      req: { method: req.method, url: req.originalUrl },
      res: { statusCode: res.statusCode },
      durationMs: Date.now() - start,
      correlationId: req.correlationId,
      userId: req.user?.id
    }, 'request completed');
  });
  next();
}

Expected output:

{"level":"info","time":1712345678901,"pid":1,"service":"order-service","environment":"production","req":{"method":"GET","url":"/api/orders/123"},"res":{"statusCode":200},"durationMs":45,"correlationId":"abc-123"}

2. OpenTelemetry Distributed Tracing

const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const provider = new NodeTracerProvider({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: process.env.SERVICE_NAME,
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.APP_VERSION
  })
});

provider.addSpanProcessor(new SimpleSpanProcessor(
  new OTLPTraceExporter({ url: 'http://jaeger:4318/v1/traces' })
));

provider.register();

const tracer = require('@opentelemetry/api').trace.getTracer('observability-project');

async function tracedOperation(operationName, fn) {
  return tracer.startActiveSpan(operationName, async (span) => {
    try {
      const result = await fn(span);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      span.recordException(err);
      throw err;
    } finally {
      span.end();
    }
  });
}

Expected output:

Traces exported to Jaeger via OTLP HTTP. Each request creates spans for API Gateway → Auth → Order Service.

3. Log Shipping with Promtail

# promtail-config.yml
scrape_configs:
  - job_name: app-logs
    pipeline_stages:
      - json:
          expressions:
            service: service
            level: level
            correlationId: correlationId
            userId: userId
      - labels:
          service:
          level:
    static_configs:
      - targets: [localhost]
        labels:
          job: app-logs
          __path__: /var/log/app/*.log
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_service]
        target_label: service

Expected output:

Promtail ships JSON logs to Loki. Labels: service, level, job. Log content is searchable.

4. Grafana Dashboard

// Dashboard JSON model (simplified)
const dashboard = {
  panels: [
    {
      title: 'Request Volume by Service',
      type: 'timeseries',
      datasource: 'Loki',
      targets: [{
        expr: 'sum by(service) (rate({job="app-logs"} |= "request completed" [1m]))',
        legendFormat: '{{service}}'
      }]
    },
    {
      title: 'Error Rate by Service',
      type: 'stat',
      datasource: 'Loki',
      targets: [{
        expr: 'sum by(service) (count_over_time({job="app-logs"} | level="error" [5m])) / sum by(service) (count_over_time({job="app-logs"} [5m])) * 100',
        legendFormat: '{{service}}'
      }]
    },
    {
      title: 'P90 Response Time (ms)',
      type: 'gauge',
      datasource: 'Loki',
      targets: [{
        expr: 'quantile_over_time(0.9, {job="app-logs"} | json | durationMs > 0 [5m])'
      }]
    },
    {
      title: 'Top 5 Slow Endpoints',
      type: 'table',
      datasource: 'Loki',
      targets: [{
        expr: 'topk(5, avg by(url) (avg_over_time({job="app-logs"} | json | durationMs > 0 [5m])))'
      }]
    }
  ]
};

Expected output:

Grafana dashboard shows: RPS by service, error rate %, P90 latency, top slow endpoints. All data from Loki logs.

5. Complete Docker Compose

version: '3.8'
services:
  api-gateway:
    build: ./api-gateway
    ports: ['3000:3000']
    environment:
      - SERVICE_NAME=api-gateway
      - LOG_LEVEL=info
    volumes:
      - ./logs:/var/log/app
    depends_on: [loki, jaeger]

  order-service:
    build: ./order-service
    ports: ['3001:3001']
    environment:
      - SERVICE_NAME=order-service
      - LOG_LEVEL=info
    volumes:
      - ./logs:/var/log/app
    depends_on: [loki, jaeger]

  promtail:
    image: grafana/promtail:latest
    volumes:
      - ./logs:/var/log/app
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

  loki:
    image: grafana/loki:latest
    ports: ['3100:3100']
    command: -config.file=/etc/loki/local-config.yaml

  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - '16686:16686'  # UI
      - '4318:4318'    # OTLP HTTP

  grafana:
    image: grafana/grafana:latest
    ports: ['3002:3000']
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
    depends_on: [loki]

Expected output:

All services start with Docker Compose. Logs flow: app → file → promtail → loki → grafana.
Traces flow: app → OTLP → jaeger → grafana.

Acceptance Criteria

1. Structured JSON logs with Pino (service, level, correlationId, durationMs).
2. Request logging middleware on all endpoints.
3. OpenTelemetry distributed tracing across both services.
4. Logs shipped to Loki via Promtail.
5. Jaeger UI shows trace spans with timing.
6. Grafana dashboard with 4 panels: RPS, error rate, P90 latency, slow endpoints.
7. Log ↔ Trace correlation (traceId in logs).
8. Error rate >5% triggers visible alert in dashboard.

Testing

#!/bin/bash
# Test observability stack

# Generate traffic
for i in {1..100}; do
  curl -s http://localhost:3000/api/orders > /dev/null
  curl -s http://localhost:3001/api/products > /dev/null
  curl -s http://localhost:3001/api/orders/error > /dev/null  # Trigger error
  sleep 0.1
done

# Check logs in Loki
curl -s "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={job="app-logs"} |= "error"' \
  --data-urlencode 'limit=5'

# Check traces in Jaeger
curl -s "http://localhost:16686/api/traces?service=order-service&limit=5"

# Check Grafana dashboard
echo "Open http://localhost:3002 in browser"

Expected output:

100 requests generate ~300 log entries in Loki.
Error endpoint generates error logs with trace IDs.
Jaeger shows traces for both services.
Grafana dashboard shows request volume, error rate, and latency.

Common Mistakes

• Not including traceId in log entries — log-to-trace correlation is impossible without it.
• Using different correlation IDs for logs and traces — use the same trace ID for both.
• Not configuring log retention — Loki defaults to 24h retention. Configure 30 days.
• Monitoring without alerting — a dashboard you never look at is not observability.
• Skipping the testing step — verify all components communicate before declaring done.

Submission Checklist

• Pino structured logging on both services
• Request logging middleware with duration
• OpenTelemetry tracing with OTLP exporter
• Promtail shipping logs to Loki
• Jaeger receiving and displaying traces
• Grafana dashboard with 4+ panels
• Log ↔ Trace correlation
• Docker Compose running all services
• 100 test requests executed and verified

What's Next

Congratulations on completing the backend logging patterns module. Continue to Middleware Patterns to learn about Express middleware Design Patterns.