Grafana Alert Rule Not Firing Fix

In this tutorial, you'll learn about Grafana Alert Rule Not Firing Fix. We cover key concepts, practical examples, and best practices.

Your Grafana alert rule shows Normal even when the condition is clearly met — the alert query returns the expected value but the alert doesn't fire. The evaluation interval is too long, the condition is wrong, or a silence is active.

The Problem

# WRONG — condition that never triggers
min(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100) < 10

The alert rule shows Normal with a value of 5.2 (below 10 threshold), but the alert never fires. The query is correct but the for duration hasn't elapsed yet, or the alert evaluation evaluates at a different time than expected.

Step-by-Step Fix

1. Check the evaluation interval

Alert rule: Low Memory
Condition: min(node_memory_MemAvailable_bytes / ...
Evaluate every: 1m
For: 5m

The alert must be in the Alerting state for the full for duration (5 minutes) before it fires. Spikes shorter than 5 minutes are ignored. If the condition only holds for 3 minutes, the alert never fires.

2. Test the alert query in Explore

Run the query in Grafana Explore at the same time the alert evaluates. Verify the query returns values that match the condition. If the query returns No data, the alert never triggers.

3. Check for active silences

Alerting > Silences

An active silence matching the alert's labels prevents it from firing. Silences can be scheduled or indefinite. Check Notification Policies for mute timings.

4. Fix the alert condition syntax

# WRONG — comparing to a string
avg(cpu_usage) > "80"

# RIGHT — comparing to a number
avg(cpu_usage) > 80

5. Use pending state to verify

When you update an alert rule, Grafana shows Pending for the for duration. If it stays Pending and never becomes Firing, the condition is intermittent. Check the alert rule evaluation log in Alerting > Alert Rules > [Rule] > State history.

Expected output:

Alert rule: Low Memory
State: Firing (since 2m ago)
Query value: 5.2 (threshold: 10)
Labels: {severity="critical", team="platform"}
✓ Alert notification sent to Slack

Prevention Tips

• Test alert queries in Explore before creating rules
• Start with for: 0m for testing, then increase
• Check Alertmanager notification configuration
• Use the State history tab to debug no-fire scenarios
• Verify alert rule evaluation interval matches your needs

Common Mistakes with alert not firing

1. Forgetting deriving (Show, Eq) on custom data types needed for debugging
2. Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable
3. Using head and tail instead of pattern matching, causing runtime errors on empty lists

These mistakes appear frequently in real-world GRAFANA code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ