Cron Notifications — Alerting and Reporting for Scheduled Job Results

In this tutorial, you will learn about Cron Notifications. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron notification strategies for alerting on job failures and reporting results: MAILTO configuration, Slack webhook integration, email reports with formatted output, webhook callbacks, and escalation to PagerDuty for critical failures.

What You Learn

You will learn how to configure cron notifications using MAILTO for email delivery, custom notification scripts for Slack and Webhooks, structured reporting with JSON output, and escalation paths for failed jobs.

Why It Matters

Silent cron failures are the most dangerous type of production incident. A backup that silently fails for weeks, a cleanup job that stops running, a report that never generates — all go unnoticed without proper notifications. Notifications turn silent failures into actionable alerts.

Real-World Use

DodaTech's cron notification system sends job results to three destinations: Slack channel #cron-jobs (all results), email to on-call engineer (failures only), and PagerDuty (critical job failures or consecutive failures). Each notification includes job name, exit code, duration, and a log snippet.

MAILTO Configuration

# Crontab with MAILTO
MAILTO=ops@dodatech.com
MAILFROM=cron@dodatech.com
CONTENT_TYPE=text/plain

# Every cron job sends output to ops@dodatech.com
0 3 * * * /usr/local/bin/db-backup.sh

# Override MAILTO for specific jobs
0 4 * * * MAILTO=backup-team@dodatech.com /usr/local/bin/verify-backup.sh

# Suppress email for successful jobs
0 5 * * * /usr/local/bin/cleanup-temp.sh > /dev/null 2>&1

Slack Notification Script

import json
import subprocess
import sys

class CronNotifier:
    def __init__(self, webhook_url=None):
        self.webhook_url = webhook_url

    def send_slack(self, job_name, status, duration, details=""):
        color = "#36a64f" if status == "SUCCESS" else "#ff0000"
        icon = ":white_check_mark:" if status == "SUCCESS" else ":x:"
        message = {
            "attachments": [{
                "color": color,
                "title": f"{icon} Cron Job: {job_name}",
                "fields": [
                    {"title": "Status", "value": status, "short": True},
                    {"title": "Duration", "value": f"{duration:.1f}s", "short": True},
                    {"title": "Details", "value": details or "No details", "short": False},
                ],
                "footer": "Cron Notification System",
                "ts": int(__import__('time').time())
            }]
        }
        return json.dumps(message, indent=2)

notifier = CronNotifier()
payload = notifier.send_slack("db-backup", "SUCCESS", 12.5, "Backed up 3 databases (15 GB)")
print(payload)

Expected output:

{
  "attachments": [
    {
      "color": "#36a64f",
      "title": ":white_check_mark: Cron Job: db-backup",
      "fields": [
        {"title": "Status", "value": "SUCCESS", "short": true},
        {"title": "Duration", "value": "12.5s", "short": true},
        {"title": "Details", "value": "Backed up 3 databases (15 GB)", "short": false}
      ],
      "footer": "Cron Notification System",
      "ts": 1719532800
    }
  ]
}

Common Mistakes

1. No MAILTO Configured

Without MAILTO, cron output is mailed to the local user's system mailbox, which nobody reads. Always set MAILTO to a team email address or use a notification script.

2. Excessive Notification Volume

A successful job that runs every 5 minutes sends 288 emails per day. Suppress success notifications for high-frequency jobs. Only notify on failures, or send a daily digest.

3. Notifications Without Context

A notification saying "Job failed" is useless. Include: job name, hostname, exit code, last N log lines, duration, and a link to the full log.

4. No Escalation Path

If the on-call engineer misses the notification, the failure goes unnoticed. Implement escalation: Slack -> email -> PagerDuty -> phone call for critical jobs.

5. Notification Script Failures

If the notification script itself fails, you never know about the cron job failure. Test notification paths separately. Use a simple fallback: if Slack webhook fails, fall back to mail.

Practice Questions

1. How do you configure cron to send email notifications?

Set MAILTO=email@example.com at the top of the crontab. Cron sends job output (stdout and stderr) to this address. Suppress output with > /dev/null 2>&1 for jobs that don't need email.

2. How do you implement Slack notifications for Cron Jobs?

Write a notification script that sends a JSON payload to the Slack webhook URL. Call it at the end of each cron job, passing the exit code, duration, and log file path as arguments.

3. What should a cron failure notification include?

Job name, hostname, exit code, duration, timestamp, last 20 lines of output, and a link to the full log file. For critical jobs, include escalation instructions.

4. How do you prevent notification fatigue?

Suppress success notifications for jobs that run more than once per hour. Send a daily digest of all job results. Only alert on failures and skipped jobs. Implement severity levels per job.

Challenge

Build a cron notification system: (1) notification dispatcher that sends to Slack, email, and PagerDuty based on severity, (2) suppression rules: suppress success for high-frequency jobs, aggregate multiple failures into a single notification, (3) escalation: if a critical job fails, send to Slack immediately, page on-call after 5 minutes if not acknowledged, call after 15 minutes, (4) daily digest: email summary of all cron jobs with pass/fail counts and duration statistics, (5) notification testing: scheduled test notification every Monday at 9 AM to verify all channels work.

FAQ

Mini Project: Complete Cron Notification System

Build a comprehensive notification system: (1) notification dispatcher supporting Slack webhook, email (SMTP), and PagerDuty Events API, (2) severity levels: INFO (success), WARNING (first failure), ERROR (consecutive failure), CRITICAL (system-level failure), (3) suppression engine: suppress success notifications for jobs running more than once per hour, aggregate duplicate failures into single notification, (4) escalation: unacknowledged critical alerts escalate every 5 minutes via increasing severity channels, (5) daily digest: HTML email with tables showing all job results, durations, and trends, (6) heartbeat monitoring: expected heartbeat per critical job, alert if heartbeat is more than 2x the expected interval.

What's Next

Now that you understand cron notifications, explore scheduling backups with cron, then learn about database maintenance scheduling.