Cron Alerting — Automated Alerts for Cron Job Failures and Anomalies

In this tutorial, you will learn about Cron Alerting. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron alerting strategies: configure automated alerts for cron job failures, duration anomalies, missed executions, and success rate degradation. Integrate with Slack, email, and PagerDuty for reliable notification.

What You Learn

You will learn how to set up alerting for Cron Jobs: defining alert rules for failures, duration anomalies, missed executions, configuring notification channels, and implementing escalation policies.

Why It Matters

A cron job that silently fails is worse than no cron job at all. Alerting turns failures into actionable notifications. Without alerting, a failed database backup is discovered only when a restore is attempted and the backup is missing.

Real-World Use

DodaTech alerting rules for cron: Page PagerDuty if a critical job fails, send Slack message to #cron-alerts for any job failure, send email digest of all failures every 6 hours. Consecutive failures escalate: 1 failure -> Slack, 2 consecutive -> PagerDuty low, 3 consecutive -> PagerDuty high.

Alert Rule Engine

import time
import json
from datetime import datetime

class CronAlertRule:
    def __init__(self, name, condition_fn, severity='warning', message_template=''):
        self.name = name
        self.condition = condition_fn
        self.severity = severity
        self.message = message_template
        self.firing = False

class CronAlertManager:
    def __init__(self):
        self.rules = []
        self.alerts = []

    def add_rule(self, rule):
        self.rules.append(rule)

    def evaluate(self, job_metrics):
        for rule in self.rules:
            try:
                triggered = rule.condition(job_metrics)
                if triggered and not rule.firing:
                    rule.firing = True
                    self._fire_alert(rule, job_metrics)
                elif not triggered:
                    rule.firing = False
            except Exception as e:
                print(f"Alert rule '{rule.name}' evaluation error: {e}")

    def _fire_alert(self, rule, metrics):
        alert = {
            'rule': rule.name,
            'severity': rule.severity,
            'job': metrics.get('job_name', 'unknown'),
            'message': rule.message.format(**metrics),
            'timestamp': datetime.now().isoformat(),
            'metrics': metrics
        }
        self.alerts.append(alert)
        channel = {'warning': 'Slack', 'critical': 'PagerDuty', 'info': 'email'}
        print(f"[{rule.severity.upper()}] Alert: {alert['message']} -> {channel.get(rule.severity, 'Slack')}")

def failure_condition(m):
    return m.get('success', 1) == 0

def duration_condition(m):
    max_dur = m.get('expected_max_duration', 300)
    return m.get('duration_seconds', 0) > max_dur * 2

def missed_condition(m):
    age_min = m.get('age_minutes_since_last_run', 0)
    expected = m.get('expected_interval_minutes', 1440)
    return age_min > expected * 2

manager = CronAlertManager()
manager.add_rule(CronAlertRule("JobFailure", failure_condition, "critical", "Job {job_name} failed"))
manager.add_rule(CronAlertRule("JobDuration", duration_condition, "warning", "Job {job_name} took {duration_seconds}s (max: {expected_max_duration}s)"))
manager.add_rule(CronAlertRule("JobMissed", missed_condition, "critical", "Job {job_name} has not run in {age_minutes_since_last_run} min"))

manager.evaluate({'job_name': 'db-backup', 'success': 0, 'duration_seconds': 45, 'expected_max_duration': 60})
manager.evaluate({'job_name': 'cache-warm', 'success': 1, 'duration_seconds': 120, 'expected_max_duration': 30})

Expected output:

[CRITICAL] Alert: Job db-backup failed -> PagerDuty
[WARNING] Alert: Job cache-warm took 120s (max: 30s) -> Slack

Escalation Policy

import time
from datetime import datetime, timedelta

class EscalationPolicy:
    def __init__(self, name, levels=None):
        self.name = name
        self.levels = levels or []
        self.consecutive_failures = {}

    def add_level(self, after_minutes, channel, severity):
        self.levels.append({'after': after_minutes, 'channel': channel, 'severity': severity})

    def evaluate(self, job_name, consecutive_failures):
        failure_count = consecutive_failures.get(job_name, 0)
        for level in reversed(self.levels):
            if failure_count >= level['after']:
                print(f"Escalation [{self.name}/{job_name}]: level={level['after']} failures -> {level['channel']} ({level['severity']})")
                return level
        return None

policy = EscalationPolicy("cron-escalation")
policy.add_level(1, "Slack", "warning")
policy.add_level(2, "PagerDuty", "critical")
policy.add_level(5, "Phone Call", "emergency")

failures = {'db-backup': 1, 'cache-warm': 3, 'report-gen': 6}
for job, count in failures.items():
    policy.evaluate(job, failures)

Expected output:

Escalation [cron-escalation/db-backup]: level=1 failures -> Slack (warning)
Escalation [cron-escalation/cache-warm]: level=2 failures -> PagerDuty (critical)
Escalation [cron-escalation/report-gen]: level=5 failures -> Phone Call (emergency)

Common Mistakes

1. Alerting Fatigue

Alerting on every single failure desensitizes the team. Use: alert on consecutive failures (2+), not single. Use different channels for different severities: Slack for warnings, PagerDuty for critical.

2. No Escalation Path

A Slack message sent at 3 AM may not be seen until 9 AM. Implement escalation: Slack -> PagerDuty -> phone call. Each level has a different notification channel and response expectation.

3. Alerting Without Context

"Job failed" tells the on-call engineer nothing. Include: job name, hostname, exit code, duration, last N log lines, and a link to the full logs. Context reduces mean time to resolution.

4. No Maintenance Window Support

If a cron job is intentionally disabled for maintenance, all alert rules should be silenced. Implement maintenance Windows: specify start/end time and affected jobs. Suppress alerts during the window.

5. Alert Rules Never Reviewed

Alert rules that were useful 6 months ago may be noisy or irrelevant now. Review alert rules monthly. Tune thresholds based on observed patterns. Remove rules that never fire or always fire.

Practice Questions

1. What is a good default alert severity for cron job failures?

Warning: first failure of non-critical job, duration exceeds 2x expected. Critical: consecutive failures (2+) of any job, no successful run in 2x interval, critical job failure. Emergency: 5+ consecutive failures, no run in 5x interval.

2. How do you implement escalation policies for cron alerts?

Define escalation tiers: tier 1 (Slack, immediate), tier 2 (PagerDuty, after 2 consecutive failures), tier 3 (phone call, after 5 consecutive failures). Each tier has a higher severity and faster expected response time.

3. How do you prevent alert fatigue from cron failures?

Require consecutive failures before alerting (2+). Use maintenance windows. Implement daily/weekly digests for non-urgent alerts. Route different severities to different channels. Review and tune thresholds monthly.

4. What information should every cron alert include?

Job name, hostname, exit code, duration, expected duration, last N log lines, timestamp, next scheduled run, and a link to the job's runbook. For failure alerts, include the error message and stack trace.

Challenge

Build a cron alerting system: (1) alert rules: failure (consecutive > 1), duration (> 2x expected), missed (no run in 2x interval), (2) escalation: 1 failure -> Slack, 2 consecutive -> PagerDuty, 5 consecutive -> phone call, (3) notification channels: Slack (with attachment showing key fields), email (HTML with full context), PagerDuty (with dedup key per job), (4) maintenance windows: YAML config with start/end and affected jobs, suppress alerts during maintenance, (5) digest: daily email summary of all alerts with pass/fail counts, (6) dashboard: alert history, escalation status per job.

FAQ

Mini Project: Cron Alerting System

Build a comprehensive alerting system: (1) alert rules engine: failure (consecutive > 1), duration (> 2x expected P95), missed (no run in 2x interval), stale (no metrics for 3x interval), (2) escalation tiers: tier 1 -> Slack (#cron-alerts), tier 2 -> PagerDuty (after 2 consecutive), tier 3 -> phone call (after 5 consecutive), (3) notification formatter: Slack with colorful attachments, email HTML with table, PagerDuty with dedup key, (4) maintenance windows: YAML config, suppress during window, (5) alert suppression: silence duplicate alerts for 1 hour, aggregate multiple failures of the same job into one alert, (6) dashboard: alert history by job, escalation status, MTTR (mean time to resolve), (7) monthly review: alert frequency report, false positive rate, tuning recommendations.

What's Next

Now that you understand cron alerting, explore SLA compliance for cron jobs, then learn about capacity planning for cron.