Log Monitoring: Alerting and Dashboards from Log Data

In this tutorial, you will learn about Log Monitoring: Alerting and Dashboards from Log Data. We cover key concepts, practical examples, and best practices to help you master this topic.

Log monitoring transforms raw log data into actionable insights through alerting, dashboards, and anomaly detection. Instead of manually grepping logs, automated monitoring surfaces problems: error rate spikes, latency increases, unusual patterns, and business metric changes.

flowchart LR
    Logs[Log Stream] --> Parser[Parse & Aggregate]
    Parser --> Metrics[Metrics Calculation]
    Metrics --> ErrorRate[Error Rate]
    Metrics --> Latency[Latency P50/P90/P99]
    Metrics --> Throughput[Request Throughput]
    Metrics --> Business[Business Events]
    
    ErrorRate --> Alert[Alert Rules]
    Latency --> Alert
    Throughput --> Alert
    Business --> Alert
    
    Alert -->|Threshold Breach| Notify[Notify: PagerDuty / Slack]
    ErrorRate --> Dashboard[Grafana Dashboard]
    Latency --> Dashboard
    Throughput --> Dashboard
    Business --> Dashboard

What You'll Learn

• Log-based alerting rules for common patterns
• Error rate monitoring and aggregation
• Anomaly detection in log patterns
• Grafana dashboards from Loki/Elasticsearch data

Why It Matters

Manual log analysis does not scale. Automated log monitoring alerts on problems within seconds of their occurrence, reducing MTTR from hours to minutes. Dashboards provide at-a-glance visibility into system health.

Real-World Use

An e-commerce platform monitors error rate as a percentage of total requests. A deployment that increases error rate above 1% triggers an automatic rollback. The operations dashboard shows real-time request volume, error rate by endpoint, and p99 latency.

Log Monitoring Implementation

Log-Based Alerting Rules

class LogAlertManager {
  constructor() {
    this.rules = [];
    this.alertHistory = [];
  }

  addRule(rule) {
    this.rules.push({
      ...rule,
      cooldownMs: rule.cooldownMs || 300000, // 5 min cooldown
      lastFired: 0
    });
  }

  async evaluate(logs) {
    for (const rule of this.rules) {
      const matching = logs.filter(log => rule.condition(log));
      const matchCount = matching.length;

      if (matchCount >= rule.threshold) {
        await this.fireAlert(rule, { count: matchCount, sample: matching[0] });
      }
    }
  }

  async fireAlert(rule, context) {
    const now = Date.now();
    if (now - rule.lastFired < rule.cooldownMs) return;

    rule.lastFired = now;

    const alert = {
      id: `alert-${Date.now()}`,
      rule: rule.name,
      severity: rule.severity,
      message: rule.message(context),
      timestamp: new Date().toISOString(),
      context
    };

    this.alertHistory.push(alert);

    if (rule.severity === 'critical') {
      await notifyPagerDuty(alert);
    } else {
      await notifySlack(alert);
    }
  }
}

// Define alert rules
const alertManager = new LogAlertManager();

alertManager.addRule({
  name: 'High Error Rate',
  severity: 'critical',
  threshold: 10,
  window: '1m',
  condition: (log) => log.level === 'error',
  message: (ctx) => `${ctx.count} errors in the last minute (threshold: 10)`
});

alertManager.addRule({
  name: '5xx Spike',
  severity: 'critical',
  threshold: 5,
  condition: (log) => log.status >= 500,
  message: (ctx) => `${ctx.count} 5xx responses in the last minute`
});

alertManager.addRule({
  name: 'Slow Endpoint',
  severity: 'warning',
  threshold: 3,
  condition: (log) => log.duration && parseInt(log.duration) > 5000,
  message: (ctx) => `Slow endpoint: ${ctx.sample.url} took ${ctx.sample.duration}ms`
});

Expected output:

10+ errors in 1 minute → CRITICAL alert → PagerDuty notification.
3+ slow endpoints (>5s) → WARNING alert → Slack notification.
Alerts have 5-minute cooldown to prevent noise.

Real-Time Log Metrics

class LogMetrics {
  constructor() {
    this.windows = new Map();
  }

  record(log) {
    // Per-endpoint metrics
    const endpoint = log.url || 'unknown';
    if (!this.windows.has(endpoint)) {
      this.windows.set(endpoint, {
        count: 0, errors: 0, status5xx: 0,
        durations: [], lastMinute: []
      });
    }
    const stats = this.windows.get(endpoint);
    stats.count++;
    if (log.level === 'error') stats.errors++;
    if (log.status >= 500) stats.status5xx++;
    if (log.duration) stats.durations.push(parseInt(log.duration));

    // Rolling 1-minute window
    stats.lastMinute.push({ time: Date.now(), log });
    const cutoff = Date.now() - 60000;
    stats.lastMinute = stats.lastMinute.filter(e => e.time > cutoff);

    return stats;
  }

  getMetrics() {
    const metrics = {};
    for (const [endpoint, stats] of this.windows) {
      const durations = stats.durations.slice(-1000);
      const sorted = [...durations].sort((a, b) => a - b);

      metrics[endpoint] = {
        rps: (stats.lastMinute.length / 60).toFixed(1),
        errorRate: stats.count > 0 ? ((stats.errors / stats.count) * 100).toFixed(2) + '%' : '0%',
        p50: durations.length > 0 ? sorted[Math.floor(sorted.length * 0.5)] : 0,
        p95: durations.length > 0 ? sorted[Math.floor(sorted.length * 0.95)] : 0,
        p99: durations.length > 0 ? sorted[Math.floor(sorted.length * 0.99)] : 0,
        totalRequests: stats.count
      };
    }
    return metrics;
  }
}

Expected output:

/api/orders: { rps: 15.2, errorRate: "1.2%", p50: 45, p95: 250, p99: 800, totalRequests: 15000 }
/api/payments: { rps: 8.5, errorRate: "0.3%", p50: 120, p95: 500, p99: 2000, totalRequests: 8500 }

Anomaly Detection in Log Patterns

class LogAnomalyDetector {
  constructor() {
    this.baselines = new Map();
    this.learningPeriod = 3600000; // 1 hour to establish baseline
  }

  learn(endpoint, log) {
    if (!this.baselines.has(endpoint)) {
      this.baselines.set(endpoint, {
        firstSeen: Date.now(),
        samples: [],
        hourlyCounts: new Array(24).fill(0),
        errorRates: []
      });
    }

    const baseline = this.baselines.get(endpoint);
    baseline.samples.push(log);
    baseline.hourlyCounts[new Date().getHours()]++;
    if (log.level === 'error') baseline.errorRates.push(1);
  }

  detectAnomalies(endpoint, currentLog) {
    const baseline = this.baselines.get(endpoint);
    if (!baseline || Date.now() - baseline.firstSeen < this.learningPeriod) {
      return []; // Still learning
    }

    const anomalies = [];

    // Volume anomaly: current volume vs baseline
    const currentHour = new Date().getHours();
    const baselineCount = baseline.hourlyCounts[currentHour] || 1;
    if (baseline.samples.length > baselineCount * 3) {
      anomalies.push({
        type: 'TRAFFIC_SURGE',
        severity: 'warning',
        message: `Traffic for ${endpoint} is ${Math.round(baseline.samples.length / baselineCount)}x normal`
      });
    }

    // Error rate anomaly
    const baselineErrorRate = baseline.errorRates.length / baseline.samples.length;
    if (currentLog.level === 'error' && baselineErrorRate < 0.01) {
      anomalies.push({
        type: 'ERROR_RATE_SURGE',
        severity: 'critical',
        message: `Unexpected error on ${endpoint} (baseline error rate: ${(baselineErrorRate * 100).toFixed(2)}%)`
      });
    }

    return anomalies;
  }
}

Expected output:

/api/search traffic 5x normal: warning anomaly
/api/payments error where baseline is <1%: critical anomaly

Common Mistakes

• Setting static thresholds — baseline traffic varies by time of day and day of week. Use dynamic baselines.
• Alerting on every error — aggregate errors over time Windows. A single error may not warrant an alert.
• Not setting alert cooldowns — alerts that fire every minute create noise and cause alert fatigue.
• Creating dashboards without alerting — dashboards show what happened; alerts tell you when to act.
• Monitoring logs but not business events — log monitoring should include business metrics (orders, signups, payments).

Practice Questions

1. What is the difference between log monitoring and metrics monitoring?
2. How do you set alert thresholds for error rates?
3. Why is alert cooldown important?
4. How does anomaly detection differ from threshold-based alerting?
5. What business metrics can you derive from application logs?

Challenge

Build a log monitoring system for an e-commerce API. Create: (1) alert rules for error rate (>2%), slow endpoints (>5s), and 5xx spikes, (2) a Grafana dashboard showing RPS, error rate by endpoint, and latency percentiles, (3) anomaly detection for traffic surges.

FAQ

Mini Project

Build a log monitoring system for your API. Implement: (1) real-time metrics from logs (RPS, error rate, latency), (2) alert rules for error rate >5%, (3) Grafana dashboard with 3 panels, (4) anomaly detection for traffic surges, (5) Slack notification on alerts.

What's Next

Continue to Log Rotation for managing log file sizes and retention.