Rate Limit Alerting — Proactive Notification of Traffic Anomalies and Abuse

In this tutorial, you will learn about Rate Limit Alerting. We cover key concepts, practical examples, and best practices to help you master this topic.

Rate limit alerting notifies operations teams when abnormal rate limiting events occur, enabling proactive response to traffic anomalies, abuse patterns, and approaching quota limits before they impact users.

What You'll Learn

• What rate limit events should trigger alerts
• How to configure Prometheus Alertmanager for rate limit alerts
• How to implement Webhook-based alert notifications

Why It Matters

When a partner's API key starts hitting limits, they eventually contact support. By then, the issue has already affected their operations. Proactive alerting lets you notify partners before they hit limits, suggest upgrades, or investigate potential abuse before it escalates.

Real-World Use

DodaTech's alerting system detects when a partner's daily quota reaches 80%, sends a warning email, at 90% sends a Slack notification to the partner success team, and at 100% automatically offers an in-app upgrade prompt.

flowchart LR
    A["Rate Limit\nEvent"] --> B{"Threshold\ncheck"}
    B -->|"80% quota"| C["Email warning\nto partner"]
    B -->|"90% quota"| D["Slack alert\nto success team"]
    B -->|"100% quota"| E["429 + upgrade\nprompt"]
    B -->|"Traffic spike\n>500% normal"| F["PagerDuty\nto on-call engineer"]
    style A fill:#dbeafe,stroke:#2563eb
    style F fill:#fecaca,stroke:#dc2626

Alert Configuration

class RateLimitAlertManager:
    def __init__(self):
        self.thresholds = {
            "quota_warning": 0.80,
            "quota_critical": 0.95,
            "rate_spike_multiplier": 5.0,
            "concurrent_attack_min": 100
        }

    def check_quota_threshold(self, api_key, tier, used, limit):
        ratio = used / limit if limit > 0 else 1.0

        if ratio >= self.thresholds["quota_critical"]:
            self.send_alert(
                "quota_critical",
                api_key=api_key,
                tier=tier,
                usage_ratio=ratio
            )
        elif ratio >= self.thresholds["quota_warning"]:
            self.send_alert(
                "quota_warning",
                api_key=api_key,
                tier=tier,
                usage_ratio=ratio
            )

    def check_traffic_anomaly(self, api_key, current_rate, historical_rate):
        if historical_rate > 0:
            spike_ratio = current_rate / historical_rate
            if spike_ratio >= self.thresholds["rate_spike_multiplier"]:
                self.send_alert(
                    "traffic_spike",
                    api_key=api_key,
                    spike_ratio=spike_ratio,
                    current_rate=current_rate
                )

Alert Channels

import smtplib
import json
import requests
from email.mime.text import MIMEText

class AlertChannels:
    def send_email(self, to_email, subject, body):
        msg = MIMEText(body)
        msg['Subject'] = subject
        msg['To'] = to_email
        s = smtplib.SMTP('smtp.dodatech.com')
        s.send_message(msg)
        s.quit()

    def send_slack(self, webhook_url, message):
        payload = {"text": f"*Rate Limit Alert*\n{message}"}
        requests.post(webhook_url, json=payload)

    def send_pagerduty(self, routing_key, summary, severity="warning"):
        payload = {
            "routing_key": routing_key,
            "event_action": "trigger",
            "payload": {
                "summary": summary,
                "source": "rate-limiter",
                "severity": severity,
                "group": "api-rate-limiting"
            }
        }
        headers = {"Content-Type": "application/json"}
        requests.post(
            "https://events.pagerduty.com/v2/enqueue",
            json=payload,
            headers=headers
        )

    def send_webhook(self, url, alert_data):
        requests.post(url, json=alert_data, timeout=5)

Webhook Notifications for Partners

@app.route('/api/webhooks/rate-limit-alert', methods=['POST'])
def partner_rate_alert_webhook():
    """Endpoint for partners to receive rate limit alerts"""
    data = request.json
    alert_type = data.get('type')
    api_key = data.get('api_key')

    if alert_type == 'quota_warning':
        # Partner has used 80% of quota
        handle_quota_warning(api_key, data)
    elif alert_type == 'quota_exceeded':
        # Partner has exceeded quota
        handle_quota_exceeded(api_key, data)

    return jsonify({"status": "received"}), 200

Common Mistakes

1. Alerting Too Frequently

Rate limit events can be high-volume. Aggregating before alerting prevents alert fatigue. Only alert on trends, not individual events.

2. Not Differentiating Alert Severities

A single client hitting limits is informational. A 10x traffic spike across all clients is critical. Use severity levels.

3. Ignoring False Positives

Traffic spikes from legitimate promotions trigger alerts. Implement a review Process to tune thresholds.

4. Not Including Context in Alerts

An alert saying "rate limit exceeded" is useless. Include: which key, which endpoint, current rate, limit, and suggested action.

5. Alerting Without Auto-Remediation

Where possible, automate the response: temporarily increase limits, block obvious abuse, or route traffic to a dedicated instance.

Practice Questions

1. What rate limit metrics should trigger alerts?
2. How do you prevent alert fatigue?
3. What information should an alert include?
4. How would you alert a partner about approaching quota?
5. What is auto-remediation for rate limit alerts?

Answers

1. Quota thresholds, traffic spikes, abuse patterns, and latency spikes. 2. Aggregate events before alerting and use severity levels. 3. API key, endpoint, current rate, limit, threshold breached, and suggested action. 4. Send a webhook to their registered alert endpoint. 5. Automatically increase limits, block abuse, or provision additional resources.

Challenge

Build an alerting system that: monitors rate limit metrics in real-time, supports configurable thresholds per client, sends alerts via email, Slack, and PagerDuty, provides a dashboard to view recent alerts, and includes auto-remediation actions like temporarily increasing limits for good customers.

FAQ

Mini Project

Create a rate limit alerting service with: a rules engine for defining alert conditions, multiple notification channels (email, Slack, PagerDuty, webhook), a web UI for configuring thresholds per client, and an auto-remediation engine that can temporarily adjust limits or block IPs based on alert rules.

What's Next

• Learn about rate limit bypass prevention techniques
• Explore rate limiting for NGINX module configuration
• Continue to distributed rate limiting with Redis cluster