Monitoring & Logging Tools — Prometheus, Grafana, ELK

In this tutorial, you'll learn about Monitoring & Logging Tools. We cover key concepts, practical examples, and best practices.

Monitoring and logging tools provide real-time visibility into application health, infrastructure performance, and error patterns — enabling teams to detect and resolve issues before users are affected.

What You'll Learn

In this tutorial, you'll learn Prometheus for metrics collection and alerting, Grafana for dashboard visualization, the ELK Stack (Elasticsearch, Logstash, Kibana) for centralized logging, Loki for log aggregation with Prometheus labels, and practical setup guides for each stack.

Why It Matters

Without monitoring, you're flying blind. Metrics tell you what's happening (CPU, memory, request rate), logs tell you why (error messages, stack traces), and dashboards correlate them. A well-monitored system has mean time to detection (MTTD) measured in minutes, not hours.

Real-World Use

Doda Browser's cloud infrastructure uses Prometheus to collect metrics from all microservices, Grafana for operational dashboards, and ELK for centralized log analysis. Alerts route to PagerDuty when error rates exceed 1% or response times spike above 500ms.

flowchart LR
  A[Application] --> B[Prometheus Exporter]
  A --> C[Filebeat / Logstash]
  B --> D[Prometheus Server]
  C --> E[Elasticsearch]
  D --> F[Alertmanager]
  D --> G[Grafana]
  E --> H[Kibana]
  F --> I[PagerDuty / Slack]
  G --> J[Dashboards]
  H --> K[Log Exploration]

Prometheus — Metrics Collection and Alerting

Prometheus scrapes metrics from HTTP endpoints and stores them as time-series data with flexible labels.

Exporters

Exporters expose metrics in Prometheus format. Common exporters include Node Exporter (system metrics), cAdvisor (container metrics), and custom application exporters.

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
  - job_name: 'api-server'
    static_configs:
      - targets: ['api:3000']

Expected behavior: Prometheus scrapes localhost:9100 (Node Exporter) and api:3000 (application metrics) every 15 seconds. Data is stored in the TSDB and available for querying via PromQL.

PromQL Queries

# Rate of HTTP requests per second (last 5 minutes)
rate(http_requests_total[5m])
# CPU usage by mode
avg by (mode) (rate(node_cpu_seconds_total[5m]))
# 95th percentile response time
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# Alert: error rate > 5%
(rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])) > 0.05

Expected output: The first query returns a single float value (requests/sec). The histogram_quantile query returns the response time in seconds at the 95th percentile. The alert expression evaluates to 1 (true) or 0 (false).

Alerting Rules

# alerts.yml
groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "API error rate above 5%"
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"

Expected behavior: When the error rate exceeds 5% for 2 consecutive minutes, Alertmanager fires a CRITICAL alert and sends it to the configured receiver (email, Slack, PagerDuty).

Grafana — Dashboard Visualization

Grafana connects to Prometheus (and many other data sources) to build interactive dashboards with panels, alerts, and annotations.

Dashboard Panel — Request Rate

{
  "title": "HTTP Request Rate",
  "type": "graph",
  "targets": [
    {
      "expr": "rate(http_requests_total[5m])",
      "legendFormat": "{{status}}",
      "refId": "A"
    }
  ],
  "gridPos": { "h": 8, "w": 12 }
}

Expected behavior: The panel shows a line graph with separate lines for each HTTP status code (2xx, 3xx, 4xx, 5xx). Hovering shows exact values at that timestamp. Time range is adjustable with the dashboard picker.

Alerting in Grafana

Grafana Alerting:
- Create alert rule from any panel
- Set conditions (e.g., `max() > 100` for 5m)
- Configure notification channel (Slack, PagerDuty, webhook)
- Alert state: Normal, Pending, Alerting

Expected behavior: When the metric exceeds the threshold, the panel border turns red with an alert icon. The configured notification channel receives the alert with a link to the dashboard panel.

ELK Stack — Centralized Logging

Elasticsearch stores and indexes logs, Logstash processes and transforms them, and Kibana provides visualization and exploration.

Logstash Configuration

# logstash.conf
input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
  geoip {
    source => "clientip"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "apache-logs-%{+YYYY.MM.dd}"
  }
}

Expected behavior: Filebeat ships Apache logs to Logstash on port 5044. Logstash parses each log line into structured fields (clientip, timestamp, response, bytes), adds geoip location data, and indexes into Elasticsearch with daily indices.

Kibana Discover

Kibana > Discover
Index pattern: apache-logs-*
Search: response: 500 AND clientip: 192.168.*
Time range: Last 24 hours

Expected behavior: Kibana shows matching log entries with a histogram at the top (event count over time) and individual log rows below. Click any entry to expand and see all parsed fields: clientip, response, bytes, geoip.location, user_agent.

Monitoring Tool Comparison

Common Errors

1. Prometheus not scraping targets — Check network connectivity, firewall rules, and that the exporter is running on the expected port. Verify with curl http://target:port/metrics.
2. High cardinality labels causing Prometheus OOM — Labels like user_id or request_id create too many time series. Keep label cardinality under 100,000 per metric.
3. Elasticsearch mapping explosion — Dynamic mapping creates too many fields. Define explicit mappings for known fields and set dynamic: strict for production indices.
4. Grafana dashboard graph showing "No data" — Check the data source is configured correctly, the metric name is correct, and the time range includes data.
5. Logstash pipeline dropping events — If the grok pattern doesn't match, events are tagged with _grokparsefailure. Add a _grokparsefailure tag handler to capture unmatched logs.

Practice Questions

1. What is the difference between Prometheus and Grafana? Prometheus is a metrics collection and alerting system with its own TSDB. Grafana is a visualization layer that queries Prometheus (and other data sources) to render dashboards.

2. How does Loki differ from the ELK Stack? Loki does not index log content — it indexes only labels (similar to Prometheus). This makes Loki cheaper and faster for log storage but less powerful for full-text search compared to Elasticsearch.

3. What is a Prometheus exporter and why is it needed? An exporter is a process that exposes metrics in Prometheus format at an HTTP endpoint. It's needed because Prometheus uses a pull model — it scrapes endpoints rather than receiving pushed data.

4. How do you create an alert in Prometheus that fires when a server is down? Rule: expr: up == 0 means the target is unreachable. Set for: 1m to avoid flapping (transient connection issues).

Challenge

Set up a complete monitoring stack using Docker Compose: Prometheus scraping Node Exporter and cAdvisor, Grafana with a pre-configured dashboard, and Loki with Promtail collecting container logs. Add a simulated high-load scenario and verify alerts fire in Grafana.

Mini Project: Monitor an API Server with Prometheus and Grafana

Build a monitoring solution for a Node.js API server:

1. Instrument the API with Prometheus client library (expose metrics at /metrics)
2. Configure Prometheus to scrape the API every 15 seconds
3. Create a Grafana dashboard with panels for: request rate, error rate, p50/p95/p99 latency, CPU/memory per instance
4. Set up an alert that fires when the error rate exceeds 3% for 5 minutes
5. Add Loki with Promtail to collect API logs
6. Create a Kibana dashboard showing logs correlated with error spikes
7. Add annotations to the Grafana dashboard for deployments (so you can see if a deploy caused a latency spike)

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.