Logging Strategies: Patterns for Effective and Efficient Logging

In this tutorial, you will learn about Logging Strategies: Patterns for Effective and Efficient Logging. We cover key concepts, practical examples, and best practices to help you master this topic.

Logging Strategy patterns provide structured approaches to common logging challenges: filtering health check noise, sampling high-volume debug logs, circuit breaking when logging systems fail, and aggregating logs across service boundaries.

flowchart TB
    subgraph Strategies
        HealthFilter[Health Check Filter]
        Aggregation[Log Aggregation]
        Sampling[Log Sampling]
        CircuitBreaker[Log Circuit Breaker]
        Taxonomy[Log Taxonomy]
    end
    
    HealthFilter -->|Remove noise| CleanLogs[Clean Log Stream]
    Aggregation -->|Correlate across services| Correlated[Correlated Logs]
    Sampling -->|Reduce volume| Sampled[Sampled Logs]
    CircuitBreaker -->|Protect app| Resilient[Resilient Logging]
    Taxonomy -->|Categories| Structured[Structured Categories]

What You'll Learn

• Health check filtering to reduce log noise
• Log aggregation patterns for multi-service correlation
• Adaptive sampling for high-volume logs
• Log circuit breaker for resilience
• Log type taxonomy and categorization

Why It Matters

Without deliberate strategies, logging becomes either too noisy (expensive, hides real issues) or too sparse (useless for debugging). Strategy patterns provide the right balance of completeness and efficiency.

Real-World Use

A platform with 100 Microservices implemented a log taxonomy: OPERATIONAL (request logs), BUSINESS (transactions), SECURITY (auth events), ERROR (failures). Each type has different retention, sampling, and alerting rules. Operational logs are sampled at 10%; business logs at 100%; error logs at 100%.

Logging Strategy Implementations

Health Check Filter

class HealthCheckFilter {
  constructor() {
    this.healthPaths = ['/health', '/healthz', '/ready', '/live', '/metrics'];
  }

  shouldLog(req) {
    // Always log errors even on health checks
    if (req.status >= 500) return true;

    // Filter out health check requests
    return !this.healthPaths.includes(req.url);
  }

  createFilteredLogger(logger) {
    const self = this;
    return {
      info: (msg, meta) => {
        if (meta?.url && self.shouldLog({ url: meta.url, status: 200 })) {
          logger.info(meta, msg);
        }
      },
      warn: (msg, meta) => logger.warn(meta, msg),
      error: (msg, meta) => logger.error(meta, msg)
    };
  }
}

// Usage
const healthFilter = new HealthCheckFilter();
const filteredLogger = healthFilter.createFilteredLogger(logger);

app.use((req, res, next) => {
  res.on('finish', () => {
    filteredLogger.info('request', {
      method: req.method,
      url: req.originalUrl,
      status: res.statusCode
    });
  });
  next();
});

Expected output:

Health check endpoints (/health, /metrics) are NOT logged for successful responses.
Error responses on health checks ARE logged.
Other endpoints are logged normally.

Adaptive Log Sampling

class AdaptiveSampler {
  constructor(initialRate = 0.1, options = {}) {
    this.sampleRate = initialRate;
    this.minRate = options.minRate || 0.01;
    this.maxRate = options.maxRate || 1.0;
    this.errorRate = 0;
    this.totalCount = 0;
    this.errorCount = 0;
    this.adjustmentInterval = options.adjustmentInterval || 60000;
    this.lastAdjustment = Date.now();
  }

  shouldSample(level) {
    // Always sample errors
    if (level === 'error' || level === 'warn') return true;

    // Adjust rate periodically
    this.adjustRate();

    return Math.random() < this.sampleRate;
  }

  recordOutcome(level) {
    this.totalCount++;
    if (level === 'error') this.errorCount++;

    // Adjust rate on error spikes
    if (level === 'error') {
      this.sampleRate = Math.min(this.maxRate, this.sampleRate * 2);
    }
  }

  adjustRate() {
    const now = Date.now();
    if (now - this.lastAdjustment < this.adjustmentInterval) return;

    this.lastAdjustment = now;

    // Calculate error rate
    this.errorRate = this.totalCount > 0 ? this.errorCount / this.totalCount : 0;

    // Adjust sampling rate based on error rate
    if (this.errorRate > 0.05) {
      // High error rate: sample more to capture details
      this.sampleRate = Math.min(this.maxRate, this.sampleRate * 2);
    } else if (this.errorRate < 0.01 && this.sampleRate > this.minRate) {
      // Low error rate: reduce sampling
      this.sampleRate = Math.max(this.minRate, this.sampleRate * 0.9);
    }

    // Reset counters
    this.totalCount = 0;
    this.errorCount = 0;
  }
}

const sampler = new AdaptiveSampler(0.1);
// Usage: sampler.shouldSample('info') returns true/false based on adaptive rate

Expected output:

Normal conditions: 10% of info logs sampled.
Error rate >5%: sampling rate increases to capture more context.
Error rate <1%: sampling rate gradually decreases.
Errors and warnings are always sampled at 100%.

Log Circuit Breaker

class LogCircuitBreaker {
  constructor(options = {}) {
    this.failureThreshold = options.failureThreshold || 5;
    this.resetTimeout = options.resetTimeout || 30000;
    this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
    this.failureCount = 0;
    this.lastFailureTime = 0;
    this.droppedCount = 0;
  }

  async write(logFn) {
    if (this.state === 'OPEN') {
      // Check if it's time to try again
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        this.state = 'HALF_OPEN';
      } else {
        this.droppedCount++;
        return; // Drop log
      }
    }

    try {
      await logFn();
      if (this.state === 'HALF_OPEN') {
        this.state = 'CLOSED';
        this.failureCount = 0;
      }
    } catch (err) {
      this.failureCount++;
      this.lastFailureTime = Date.now();

      if (this.failureCount >= this.failureThreshold || this.state === 'HALF_OPEN') {
        this.state = 'OPEN';
        logger.warn({ droppedCount: this.droppedCount }, 'Log circuit breaker opened');
      }

      throw err;
    }
  }

  getState() {
    return {
      state: this.state,
      failureCount: this.failureCount,
      droppedCount: this.droppedCount,
      lastFailureTime: new Date(this.lastFailureTime).toISOString()
    };
  }
}

Expected output:

Log system fails 5 times → circuit opens → logs are dropped silently.
After 30 seconds → circuit half-opens → one log attempt → if successful, circuit closes; if fails, stays open.
Dropped log count is tracked and logged when circuit opens.

Log Type Taxonomy

const LOG_TAXONOMY = {
  OPERATIONAL: {
    prefix: 'op',
    description: 'Request/response logs, system metrics',
    retention: '30d',
    sampling: 0.1,
    alert: false,
    fields: ['method', 'url', 'status', 'duration', 'correlationId']
  },
  BUSINESS: {
    prefix: 'bus',
    description: 'Business events: orders, payments, signups',
    retention: '90d',
    sampling: 1.0,
    alert: true,
    fields: ['eventType', 'entityId', 'amount', 'userId']
  },
  SECURITY: {
    prefix: 'sec',
    description: 'Auth events, permission changes, admin actions',
    retention: '365d',
    sampling: 1.0,
    alert: true,
    fields: ['eventType', 'actorId', 'targetId', 'action', 'result']
  },
  ERROR: {
    prefix: 'err',
    description: 'Application errors, exceptions, failures',
    retention: '365d',
    sampling: 1.0,
    alert: true,
    fields: ['errorName', 'message', 'stack', 'code', 'service']
  },
  DEBUG: {
    prefix: 'dbg',
    description: 'Detailed debugging information',
    retention: '7d',
    sampling: 0.01,
    alert: false,
    fields: ['function', 'variables', 'state']
  }
};

function createTaxonomyLogger(logger) {
  return Object.entries(LOG_TAXONOMY).reduce((acc, [type, config]) => {
    acc[type.toLowerCase()] = (message, data = {}) => {
      if (Math.random() < config.sampling) {
        logger.info({
          _type: type,
          _prefix: config.prefix,
          ...data
        }, `[${config.prefix}] ${message}`);
      }
    };
    return acc;
  }, {});
}

const taxLogger = createTaxonomyLogger(logger);

// Usage
taxLogger.operational('Request completed', { method: 'GET', url: '/api/users', duration: '45ms' });
taxLogger.business('Order created', { eventType: 'ORDER_CREATED', amount: 2999 });
taxLogger.security('Admin login', { actorId: 'admin_1', result: 'success' });

Expected output:

[op] Request completed - sampled at 10%
[bus] Order created - sampled at 100%
[sec] Admin login - sampled at 100%
[err] Database connection failed - sampled at 100% with alerting

Common Mistakes

• Not filtering health checks — health check logs can account for 50%+ of log volume.
• Using the same sampling rate for all log types — operational logs can be sampled; business and security logs should not.
• Not having a log circuit breaker — if the logging system fails, the application should not fail with it.
• Mixing log types without taxonomy — treating business events and debug traces the same leads to confusion.
• Not reviewing and adjusting strategies — logging needs change as the application evolves.

Practice Questions

1. Why should health checks be filtered from logs?
2. How does adaptive sampling differ from static sampling?
3. What is a log circuit breaker and when should you use it?
4. Why is log taxonomy important for large systems?
5. How do logging strategies differ by log type?

Challenge

Design a logging strategy for a SaaS platform with 20 microservices. Implement: (1) health check filtering, (2) log type taxonomy (operational, business, security, error, debug), (3) adaptive sampling (errors 100%, info 10%), (4) log circuit breaker, (5) different retention per log type.

FAQ

Mini Project

Build a logging strategy system. Implement: (1) health check filter middleware, (2) log type taxonomy with different sampling rates, (3) adaptive sampler that increases rate on errors, (4) log circuit breaker for resilience, (5) strategy configuration via environment variables.

What's Next

Continue to Observability Project — a comprehensive hands-on project.