Error Handling in Cron Jobs
In this tutorial, you will learn about Error Handling in Cron Jobs. We cover key concepts, practical examples, and best practices to help you master this topic.
Handle cron job failures gracefully: exit codes, error notification, retry mechanisms, graceful degradation, alerting, and building robust cron scripts that recover from errors.
What You Learn
You will learn how to handle errors in cron jobs using exit codes, implement retry logic for transient failures, send alerts on failure, degrade gracefully when dependencies are unavailable, and build robust cron scripts.
Why It Matters
Cron jobs run unattended. A silent failure can corrupt data, miss backups, or leave the system in an inconsistent state for hours. Proper error handling ensures failures are detected, reported, and recovered automatically.
Real-World Use
DodaTech's backup cron job retries 3 times on network failures, alerts PagerDuty if all retries fail, degrades to a partial backup if the primary storage is full, and sends a summary email with backup status every morning.
Exit Codes and Error Checking
# Exit codes in scripts determine success/failure
# 0 = success, non-0 = failure
# Check exit code in shell
/usr/local/bin/backup.sh
if [ $? -ne 0 ]; then
echo "Backup failed with exit code $?"
exit 1
fi
# Using && and || for inline error handling
/usr/local/bin/backup.sh && echo "Success" || echo "Failed"
# Capture exit code from pipe
/usr/local/bin/backup.sh 2>&1 | tee -a /var/log/cron/backup.log
EXIT_CODE=${PIPESTATUS[0]}
# Set exit on error in scripts
#!/bin/bash
set -e # Exit on any error
set -u # Exit on undefined variable
set -o pipefail # Exit on pipe failure
echo "This runs"
false # This will exit the script
echo "This never runs"
Retry Logic for Cron Jobs
#!/usr/bin/env python3
"""Cron job with retry logic for transient failures."""
import time
import sys
import logging
class CronRetry:
def __init__(self, max_retries=3, base_delay=5, backoff=2):
self.max_retries = max_retries
self.base_delay = base_delay
self.backoff = backoff
self.logger = logging.getLogger('cron-retry')
def execute(self, func, *args, **kwargs):
last_exception = None
for attempt in range(1, self.max_retries + 1):
try:
result = func(*args, **kwargs)
if attempt > 1:
self.logger.info(f"Succeeded on retry {attempt}")
return result
except TransientError as e:
last_exception = e
if attempt < self.max_retries:
delay = self.base_delay * (self.backoff ** (attempt - 1))
self.logger.warning(f"Attempt {attempt} failed: {e}. Retrying in {delay}s")
time.sleep(delay)
else:
self.logger.error(f"All {self.max_retries} attempts failed")
raise PermanentError(f"Failed after {self.max_retries} retries: {e}")
except PermanentError:
raise
except Exception as e:
raise PermanentError(f"Non-retryable error: {e}")
class TransientError(Exception):
pass
class PermanentError(Exception):
pass
def fetch_remote_data():
"""Simulate an operation that may fail transiently."""
import random
if random.random() < 0.6:
raise TransientError("Network timeout")
return {"data": "success"}
retrier = CronRetry(max_retries=3, base_delay=2)
try:
result = retrier.execute(fetch_remote_data)
print(f"Success: {result}")
except PermanentError as e:
print(f"Failed: {e}")
sys.exit(1)
Expected output (on first try failure, second try success):
Attempt 1 failed: Network timeout. Retrying in 2s
Success: {'data': 'success'}
Alerting on Failure
# Simple email alert on failure
0 3 * * * /usr/local/bin/backup.sh || \
echo "Backup failed on $(hostname) at $(date)" | \
mail -s "CRON ALERT: Backup Failed" admin@dodatech.com
# Webhook alert using curl
0 3 * * * /usr/local/bin/backup.sh || \
curl -s -X POST \
-H "Content-Type: application/json" \
-d '{"text":"Backup failed on '$(hostname)'"}' \
https://hooks.slack.com/services/xxx/yyy/zzz
# PagerDuty alert
0 3 * * * /usr/local/bin/backup.sh || \
curl -s -X POST \
-H "Content-Type: application/json" \
-d '{"routing_key":"YOUR_KEY","event_action":"trigger","payload":{"summary":"Backup failed","source":"'$(hostname)'","severity":"error"}}' \
https://events.pagerduty.com/v2/enqueue
# Comprehensive alert with diagnostics
#!/bin/bash
# /usr/local/bin/alert-on-failure.sh
ALERT_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
HOST=$(hostname)
DATE=$(date)
read -r INPUT
EXIT_CODE=$?
if [ $EXIT_CODE -ne 0 ]; then
PAYLOAD=$(cat <<EOF
{
"text": "CRON FAILURE on ${HOST} at ${DATE}",
"attachments": [{
"color": "danger",
"text": "Exit code: ${EXIT_CODE}\\nCommand: ${COMMAND}\\nOutput: ${INPUT}"
}]
}
EOF
)
curl -s -X POST -H "Content-Type: application/json" -d "$PAYLOAD" "$ALERT_URL"
fi
# Usage in crontab:
# 0 3 * * * backup.sh 2>&1 | alert-on-failure.sh
Graceful Degradation
#!/usr/bin/env python3
"""Cron job with graceful degradation."""
import sys
import logging
class GracefulCron:
def __init__(self):
self.logger = logging.getLogger('graceful-cron')
self.partial_success = False
def check_dependencies(self):
"""Check if required services are available."""
deps = {
'database': self._check_db(),
'storage': self._check_storage(),
'api': self._check_api(),
}
return deps
def _check_db(self):
try:
# Simulate DB check
return True
except Exception:
self.logger.warning("Database unavailable, using cache")
return False
def _check_storage(self):
import os
stat = os.statvfs('/backups')
free_gb = stat.f_frsize * stat.f_bavail / (1024**3)
if free_gb < 1:
self.logger.warning(f"Low disk space: {free_gb:.1f}GB free")
return False
return True
def _check_api(self):
import socket
try:
socket.create_connection(("api.dodatech.com", 443), timeout=5)
return True
except OSError:
self.logger.warning("API unavailable")
return False
def run_backup(self):
deps = self.check_dependencies()
if not any(deps.values()):
self.logger.error("All dependencies unavailable, skipping backup")
return False
if not deps['database']:
self.logger.info("Running partial backup (cached data only)")
self._backup_cache()
self.partial_success = True
else:
self._backup_full()
if not deps['storage']:
self.logger.warning("Primary storage full, using secondary")
self._backup_secondary()
self.logger.info("Backup completed (partial: %s)", self.partial_success)
return True
def _backup_full(self):
self.logger.info("Full backup completed")
def _backup_cache(self):
self.logger.info("Cache backup completed")
def _backup_secondary(self):
self.logger.info("Secondary backup completed")
cron = GracefulCron()
success = cron.run_backup()
if not success:
sys.exit(1)
Timeout Protection
#!/bin/bash
# /usr/local/bin/cron-timeout.sh
# Kill a cron job if it runs too long
JOB_COMMAND="$@"
TIMEOUT=300 # 5 minutes
LOG_FILE="/var/log/cron/timeout-kill.log"
# Run with timeout
timeout $TIMEOUT $JOB_COMMAND
EXIT_CODE=$?
if [ $EXIT_CODE -eq 124 ]; then
echo "[$(date)] TIMEOUT: Command killed after ${TIMEOUT}s: $JOB_COMMAND" >> "$LOG_FILE"
logger -p cron.err "Cron job timed out after ${TIMEOUT}s: ${JOB_COMMAND}"
fi
exit $EXIT_CODE
# Usage in crontab:
# 0 3 * * * /usr/local/bin/cron-timeout.sh /usr/local/bin/backup.sh
Common Mistakes
1. Ignoring Exit Codes
A script may appear to succeed but fail internally. Always check exit codes explicitly, especially in piped commands.
2. No Alerting on Failure
Failures go unnoticed until someone checks. Always configure alerting: email, Webhook, PagerDuty, or a monitoring system.
3. No Retry for Transient Errors
Network timeouts and temporary service disruptions are common. Without retries, these transient failures become permanent data loss.
4. Hardcoded Assumptions
Scripts assume databases are up, disks are not full, APIs respond. Check these assumptions and degrade gracefully.
5. No Timeout
A stuck cron job (hung database connection, infinite loop) never exits. Use timeout to ensure jobs terminate.
Practice Questions
1. What does exit code 124 mean from the timeout command?
The command was killed because it exceeded the time limit. 124 is the timeout command's exit code for a timed-out Process.
2. How do you send an alert when a cron job fails?
Use a conditional: run the command, check exit code with $?, and send a webhook or email on non-zero exit.
3. What is graceful degradation?
Instead of failing entirely when a dependency is unavailable, the script performs a reduced version of its task (e.g., partial backup, cached data only).
4. Why should you use set -e in bash scripts?
It causes the script to exit immediately if any command fails (returns non-zero exit code), preventing silent cascading failures.
Challenge
Build a cron job with: 3 retries with 30-second backoff for network operations, timeout of 10 minutes, graceful degradation if database is down (use cache), Slack webhook alert on permanent failure, and a summary email with exit code and duration.
FAQ
Mini Project: Robust Cron Wrapper
#!/bin/bash
# /usr/local/bin/robust-cron.sh
# A robust wrapper for any cron job
COMMAND="$@"
JOB_NAME=$(basename "$(echo "$COMMAND" | awk '{print $1}')")
LOG_DIR="/var/log/cron"
LOG_FILE="${LOG_DIR}/${JOB_NAME}.log"
MAX_RETRIES=3
RETRY_DELAY=30
TIMEOUT=600
ALERT_URL="https://hooks.slack.com/services/YOUR/WEBHOOK"
mkdir -p "$LOG_DIR"
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Starting ${JOB_NAME}" >> "$LOG_FILE"
for RETRY in $(seq 1 $MAX_RETRIES); do
timeout $TIMEOUT $COMMAND >> "$LOG_FILE" 2>&1
EXIT_CODE=$?
if [ $EXIT_CODE -eq 0 ]; then
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Completed successfully" >> "$LOG_FILE"
exit 0
elif [ $EXIT_CODE -eq 124 ]; then
echo "[$(date)] TIMEOUT after ${TIMEOUT}s (attempt ${RETRY}/${MAX_RETRIES})" >> "$LOG_FILE"
else
echo "[$(date)] Failed with code ${EXIT_CODE} (attempt ${RETRY}/${MAX_RETRIES})" >> "$LOG_FILE"
fi
if [ $RETRY -lt $MAX_RETRIES ]; then
sleep $RETRY_DELAY
fi
done
# All retries failed
ERROR_MSG="${JOB_NAME} failed after ${MAX_RETRIES} attempts on $(hostname)"
logger -p cron.err "$ERROR_MSG"
curl -s -X POST -H "Content-Type: application/json" \
-d "{\"text\":\"${ERROR_MSG}\"}" "$ALERT_URL" > /dev/null 2>&1
exit 1
What's Next
Now that you understand error handling in cron, explore locking and concurrency control to prevent overlapping executions, then learn about distributed cron scheduling for multi-server environments.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro