Prometheus Alerting Rules and Alertmanager — Complete Guide with Production Alerting

In this tutorial, you'll learn about Prometheus Alerting Rules and Alertmanager. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Prometheus alerting evaluates user-defined rules against time-series metrics and triggers alerts through Alertmanager, which handles deduplication, grouping, routing, silencing, and notification delivery to Slack, PagerDuty, email, and custom Webhooks.

What You'll Learn

Why It Matters

Raw metrics are useless without intelligent alerting. Without well-designed alerting rules, teams face alert fatigue from noisy, poorly-tuned alerts that fire for transient issues. Without Alertmanager routing, every alert goes to every on-call engineer regardless of severity or team ownership. Without inhibition, a "server down" alert and 50 "service X unreachable" alerts all fire simultaneously. Proper alerting design ensures the right person is notified at the right time with the right context.

Real-World Use

DodaTech monitors 200+ services with Prometheus and routes alerts through Alertmanager: critical infrastructure alerts go to PagerDuty with 5-minute escalation, team-specific alerts route to Slack channels, and informational alerts are suppressed during maintenance windows. Alert fatigue dropped from 200+ alerts per day to under 10 actionable notifications.

flowchart TD
    A["Prometheus Server"] --> B["Recording Rules"]
    A --> C["Alerting Rules"]
    B --> D["Derived Metrics"]
    C --> E["Alerts: firing"]
    E --> F["Alertmanager"]
    F --> G["Grouping & Dedup"]
    G --> H["Route: Production"]
    G --> I["Route: Staging"]
    G --> J["Route: Infrastructure"]
    H --> K["Receiver: PagerDuty"]
    H --> L["Receiver: Slack #prod-alerts"]
    I --> M["Receiver: Slack #staging"]
    J --> N["Receiver: Slack #infra"]
    K --> O["On-Call Engineer"]
    L --> O
    style A fill:#E6522C,color:#fff
    style F fill:#E6522C,color:#fff

Recording Rules

Recording rules pre-compute frequently used or expensive queries and store them as new time series. They improve dashboard load times and reduce query costs.

# rules/recording_rules.yml
groups:
  - name: service_sli_records
    interval: 30s
    rules:
      - record: job:request_latency_seconds:p99
        expr: |
          histogram_quantile(0.99,
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)
          )

      - record: job:error_rate:ratio_5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))

      - record: instance:cpu_utilization:ratio
        expr: |
          1 - (avg without (cpu) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

      - record: instance:memory_utilization:ratio
        expr: |
          node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes
          /
          node_memory_MemTotal_bytes

Expected behavior: Every 30 seconds, Prometheus evaluates these expressions and stores the results as new time series. The p99 latency query, which is expensive to compute over large ranges, is pre-computed and served instantly to dashboards.

Alerting Rules

Alerting rules define conditions that, when true for a specified duration, fire an alert. Well-designed rules include proper severity, summary, and description annotations.

# rules/alerting_rules.yml
groups:
  - name: infrastructure_alerts
    interval: 30s
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: >
            Instance {{ $labels.instance }} of job {{ $labels.job }}
            has been unreachable for more than 5 minutes.

      - alert: HighCPUUsage
        expr: instance:cpu_utilization:ratio > 0.9
        for: 10m
        labels:
          severity: warning
          team: infrastructure
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: >
            CPU usage on {{ $labels.instance }} is at
            {{ printf "%.1f" $value }} percent for 10 minutes.

      - alert: HighMemoryUsage
        expr: instance:memory_utilization:ratio > 0.9
        for: 15m
        labels:
          severity: warning
          team: infrastructure
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: >
            Memory usage on {{ $labels.instance }} is at
            {{ printf "%.1f" $value }} percent.

      - alert: DiskSpaceFull
        expr: |
          (node_filesystem_avail_bytes{fstype!="tmpfs",mountpoint!="/etc/hostname"}
          / node_filesystem_size_bytes{fstype!="tmpfs",mountpoint!="/etc/hostname"}) < 0.1
        for: 5m
        labels:
          severity: critical
          team: infrastructure
        annotations:
          summary: "Disk space critically low on {{ $labels.instance }}"
          description: >
            Disk {{ $labels.mountpoint }} on {{ $labels.instance }}
            has less than 10 percent free space.

Expected behavior: When an instance is down for 5 continuous minutes, a critical severity alert fires. High CPU usage for 10 minutes triggers a warning. The for duration prevents alerts from firing during brief transient spikes.

# Application-level alerting rules
  - name: application_alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: job:error_rate:ratio_5m > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "Error rate above 5 percent for {{ $labels.job }}"
          description: >
            The 5-minute error rate for {{ $labels.job }} is
            {{ printf "%.2f" $value }} percent, exceeding the 5 percent threshold.

      - alert: HighLatency
        expr: job:request_latency_seconds:p99 > 2.0
        for: 5m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "P99 latency above 2 seconds for {{ $labels.job }}"
          description: >
            P99 latency for {{ $labels.job }} is
            {{ printf "%.2f" $value }} seconds, exceeding the 2 second threshold.

      - alert: NoTraffic
        expr: rate(http_requests_total[10m]) == 0
        for: 10m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "No traffic detected for {{ $labels.job }}"
          description: >
            Service {{ $labels.job }} has received zero requests
            in the last 10 minutes.

# View firing alerts
curl http://localhost:9090/api/v1/alerts

# Expected output (truncated):
# {
#   "data": {
#     "alerts": [
#       {
#         "labels": {"alertname": "InstanceDown", "instance": "web-01", "severity": "critical"}, "#         "state": "firing"", "#         "active_at": "2026-06-22T10:00:00Z"",
#         "annotations": {"summary": "Instance web-01 is down"}
#       }
#     ]
#   }
# }

Alertmanager Configuration

Alertmanager receives alerts from Prometheus and handles routing, grouping, inhibition, and silencing.

# alertmanager.yml
route:
  receiver: default
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        team: infrastructure
        severity: critical
      receiver: pagerduty-critical
      repeat_interval: 15m
    - match:
        team: infrastructure
      receiver: slack-infrastructure
    - match:
        team: backend
        severity: critical
      receiver: pagerduty-critical
    - match:
        team: backend
      receiver: slack-backend
    - match_re:
        severity: ^(warning|info)$
      receiver: slack-warnings
      group_wait: 1m
      group_interval: 15m

receivers:
  - name: default
    slack_configs:
      - channel: "#alerts"
        send_resolved: true
        title: "{{ .GroupLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

  - name: pagerduty-critical
    pagerduty_configs:
      - routing_key: "<pd-integration-key>"
        severity: critical
        description: "{{ .GroupLabels.alertname }} - {{ .Annotations.summary }}"
        details:
          firing: "{{ .Alerts.Firing | len }}"
          resolved: "{{ .Alerts.Resolved | len }}"

  - name: slack-infrastructure
    slack_configs:
      - channel: "#infra-alerts"
        send_resolved: true
        title: "{{ .GroupLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

  - name: slack-backend
    slack_configs:
      - channel: "#backend-alerts"
        send_resolved: true

  - name: slack-warnings
    slack_configs:
      - channel: "#low-priority"
        send_resolved: false

Inhibition and Silences

Inhibition suppresses lower-severity alerts when a higher-severity alert is firing. Silences temporarily mute alerts based on matchers.

# alertmanager.yml (inhibition rules)
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["instance", "job"]

  - source_match:
      alertname: InstanceDown
    target_match_re:
      severity: ^(warning|info)$
    equal: ["instance"]

Expected behavior: If an InstanceDown (critical) alert fires for web-01, all warning and info alerts for web-01 are automatically suppressed. There is no value in receiving "High CPU usage" for an instance that is already down.

# Create a silence via amtool
amtool silence add \
  alertname="HighCPUUsage" \
  instance="web-01" \
  --duration=2h \
  --comment="Maintenance window for kernel update"

# Expected output:
# d0a1b2c3-1234-5678-9abc-def012345678

Common Errors

1. Alert rules without for duration: An alert like expr: up == 0 without for: 5m fires immediately when a single scrape fails. Network glitches, Prometheus restarts, and short-lived connectivity issues trigger false alerts. Always set a for duration at least 3-5 scrape intervals.

2. Missing send_resolved on receivers: Without send_resolved: true, engineers receive the firing notification but never the resolved notification, leaving them wondering if the issue was actually fixed. Always enable resolved notifications for actionable alerts.

3. Routing tree without a default route: If an alert does not match any route, Alertmanager drops it silently. Always define a default route at the root that catches unmatched alerts and sends them to a general channel for review.

4. Repeat interval too short for non-critical alerts: Setting repeat_interval: 5m for warning alerts means engineers receive the same Slack notification every 5 minutes until the alert resolves. This creates alert fatigue and causes engineers to mute channels entirely.

5. Inhibition rules not configured: Without inhibition, a server outage triggers 50 individual alerts (CPU, memory, disk, process, mount point). The on-call engineer receives 50 notifications for a single root cause. Inhibition collapses this into one actionable alert.

Practice Questions

1. What is the difference between a recording rule and an alerting rule? Answer: A recording rule pre-computes an expression and stores the result as a new time series, used for expensive queries or computed metrics. An alerting rule evaluates an expression and fires an alert if the condition is met for the specified duration.

2. How does Alertmanager grouping prevent notification overload? Answer: Grouping collects multiple alerts with the same group labels (e.g., alertname, severity) into a single notification. Instead of 10 individual Slack messages for 10 down instances, Alertmanager sends one message listing all 10 instances.

3. What is the purpose of group_wait versus group_interval in Alertmanager routing? Answer: group_wait is the initial delay before sending the first notification for a new alert group, allowing time for additional alerts to arrive and be grouped together. group_interval is the delay between subsequent notifications for the same group.

4. How does inhibition improve the signal-to-noise ratio of alerts? Answer: Inhibition suppresses low-severity alerts when a high-severity alert for the same component is already firing. This prevents cascading alerts (a server is down, so all services on it appear down) and directs attention to the root cause.

Challenge

Design a comprehensive alerting Strategy for an e-commerce platform with 5 Microservices: write Prometheus recording rules for P99 latency, error rate, and request rate per service, write alerting rules for high error rate (>3 percent for 5 minutes), high latency (P99 > 2 seconds for 5 minutes), service down (5 minutes), disk space (<15 percent for 5 minutes), and certificate expiry (<30 days), configure Alertmanager with routes that send critical alerts to PagerDuty with 5-minute repeat interval, team-specific alerts to Slack channels, and low-severity alerts to email, add inhibition rules to suppress warning alerts when critical alerts fire for the same instance, and create a silence for a planned maintenance window.

Mini Project

Build a complete Prometheus alerting stack from scratch: deploy Prometheus with scraping targets (node_exporter on servers, application metrics endpoints), write recording rules for SLO-related metrics (availability, latency, error budget), write alerting rules for all infrastructure and application components with appropriate for durations and severity levels, deploy Alertmanager with routing to Slack (team channels), PagerDuty (critical only), and email (warnings), configure inhibition rules to prevent cascading alerts, set up a maintenance window workflow using amtool silences with expiration, create a dashboard in Grafana that shows active alerts, alert history, and notification delivery status, generate test alerts by stopping a service, and tune the for durations and thresholds based on the observed signal-to-noise ratio over a week.

Related Resources

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.