Prometheus Alerting Rules & Alertmanager Configuration

Prometheus alerting rules evaluate PromQL expressions on a schedule and fire alerts when conditions are met, while Alertmanager receives those alerts, deduplicates and groups them, and routes notifications to the right channels.

What You'll Learn

In this tutorial, you will write alerting rules for common scenarios, configure Alertmanager with receivers and routing, and test that alerts arrive at their destination correctly.

Why It Matters

Without alerting, monitoring is just a dashboard you might look at. Alerts make monitoring proactive. When a disk fills up or a service goes down, the right person gets notified in the right way. Alertmanager handles the critical job of preventing alert fatigue through grouping, inhibition, and silencing.

Real-World Use

Doda Browser uses Prometheus alerting to detect when its sync service stops responding. An alert fires if the up metric for the sync service is 0 for more than 1 minute. Alertmanager sends a Slack notification to the on-call channel and escalates to PagerDuty after 5 minutes if the issue is not acknowledged.

Step 1: Create Alerting Rules

Alerting rules are defined in a separate file referenced from the Prometheus config. Create rules.yml:

groups:
  - name: infrastructure
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute."

The for field ensures the condition must persist for 1 minute before firing, preventing flapping alerts.

Step 2: Reference Rules in Prometheus Config

Update <a href="/devops/prometheus-grafana/">Prometheus</a>.yml to include the rules file:

rule_files:
  - "rules.yml"

Restart or reload Prometheus. Verify rules are loaded at /rules in the web UI.

Step 3: Install and Configure Alertmanager

Download Alertmanager and create a configuration file:

route:
  receiver: team-ops
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: team-ops
    slack_configs:
      - api_url: https://hooks.slack.com/services/...
        channel: "#alerts"

Start Alertmanager with alertmanager --config.file=alertmanager.yml.

Step 4: Configure Prometheus to Send Alerts

Add the Alertmanager address to <a href="/devops/prometheus-grafana/">Prometheus</a>.yml:

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

Restart Prometheus. Alerts now flow from Prometheus rules to Alertmanager to your notification channel.

Step 5: Test the Alert Pipeline

Stop a monitored service to trigger the InstanceDown alert. After 1 minute:

• Prometheus shows the alert as FIRING in the web UI at /alerts
• Alertmanager shows the alert and its routing at /#/alerts
• Slack receives the notification in the configured channel

Common Mistakes

1. Missing for Duration

Omitting for causes alerts to fire immediately on transient spikes, creating noise. Always set a for duration.

2. No Alertmanager Configuration

Alerting rules without Alertmanager configured in Prometheus go nowhere. Ensure alerting.alertmanagers is set.

3. Not Grouping Related Alerts

Without proper grouping, a network outage that affects 50 instances fires 50 separate notifications. Configure group_by labels to collapse them.

4. Wrong Severity Labels

Too many critical alerts desensitize the team. Use severity levels (critical, warning, info) and route them differently.

5. No Repeat Interval

Without repeat_interval, acknowledged alerts never remind you again. Set a reasonable repeat interval like 4 hours for ongoing issues.

Practice Questions

1. What is the purpose of the for field in an alerting rule? It specifies how long a condition must persist before the alert fires, preventing flapping from transient issues.

2. What does Alertmanager do with alerts it receives? It groups, deduplicates, inhibits, silences, and routes alerts to configured receivers (Slack, email, PagerDuty, etc.).

3. How do you prevent alert fatigue? Use grouping to collapse related alerts, set appropriate severity levels, define repeat intervals, and use silences for planned maintenance.

4. What is the difference between group_wait and group_interval? group_wait is how long to wait before sending the first notification for a new group. group_interval is the minimum time between notifications for an existing group.

5. Challenge: Create an alert that fires when CPU usage exceeds 80% for 5 minutes, routed to a warning channel, and a separate critical alert for 95% CPU that pages the on-call engineer.

What's Next

Learn about exporters that collect metrics from databases, hardware, and applications to feed into your alerting pipeline.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.