Structured JSON Logging — Complete API Logging Guide

In this tutorial, you will learn about Structured JSON Logging. We cover key concepts, practical examples, and best practices to help you master this topic.

Structured logging outputs logs as JSON objects instead of plain text. Each log entry contains fields like timestamp, level, request_id, method, path, status, and duration.

What You'll Learn

You'll learn how to implement structured JSON logging, what fields to include, and how to integrate with log aggregation tools.

Why It Matters

Plain text logs are difficult to search, filter, and analyze programmatically. JSON logs enable powerful queries like "find all 500 errors in the last hour for /api/payments".

Real-World Use

A team uses ELK Stack with structured JSON logs. When a user reports an error, the team searches for the request_id, finds the exact log entry, and sees the full request context including headers, body, and response time.

Implementation

import logging
import json
import uuid
from datetime import datetime
from flask import Flask, request, g

app = Flask(__name__)

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage()
        }
        if hasattr(record, "request_id"):
            log_entry["request_id"] = record.request_id
        if hasattr(record, "extra"):
            log_entry.update(record.extra)
        return json.dumps(log_entry)

handler = logging.StreamHandler()
handler.setFormatter(JSONFormatter())
app.logger.addHandler(handler)
app.logger.setLevel(logging.INFO)

@app.before_request
def before_request():
    g.request_id = request.headers.get("X-Request-Id", str(uuid.uuid4()))
    g.start_time = datetime.utcnow()

@app.after_request
def after_request(response):
    duration = (datetime.utcnow() - g.start_time).total_seconds()
    app.logger.info("request_completed", extra={
        "request_id": g.request_id,
        "method": request.method,
        "path": request.path,
        "status": response.status_code,
        "duration": duration,
        "ip": request.remote_addr,
        "user_agent": request.headers.get("User-Agent")
    })
    return response

Common Fields

Common Mistakes

| Mistake | Fix | |---------|-----| | Logging sensitive data | PII and tokens in logs | Scrub sensitive fields | | No request_id | Cannot correlate logs | Generate or accept request_id header | | Inconsistent field names | Cannot query across services | Use standard field naming convention | | Logging too much (100k logs/sec) | Log storage costs explode | Sample high-volume logs (1:10) | | No log level consistency | Error vs warn vs info confusion | Define clear logging guidelines |

Practice Questions

1. Why use JSON for logs instead of plain text?
2. What fields should every log entry have?
3. How does request_id enable distributed tracing?
4. How do you prevent sensitive data in logs?
5. When should you sample logs?

Challenge

Implement structured JSON logging with request_id propagation. Add log scrubbing for sensitive fields (passwords, tokens, credit cards). Test with a sample API.

What's Next

Learn about distributed tracing for Microservices.