Cron Failure Analysis — Systematic Post-Mortem for Failed Scheduled Jobs

In this tutorial, you will learn about Cron Failure Analysis. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron failure analysis: conduct systematic post-mortems for failed Cron Jobs, classify failure causes, identify preventive measures, track failure patterns over time, and build a cron failure knowledge base.

What You Learn

You will learn how to analyze cron job failures: systematic post-mortem Process, failure classification, root cause analysis, preventive measure implementation, and failure pattern tracking.

Why It Matters

Each cron job failure is a learning opportunity. Without systematic analysis, the same failure repeats. Post-mortems turn failures into improvements: each incident reduces the probability of recurrence.

Real-World Use

DodaTech tracks every cron job failure in a post-mortem system. Each failure is classified into one of 12 categories (resource exhaustion, dependency failure, configuration error, code bug, etc.). The most common failure type triggers a quarterly improvement initiative.

Failure Classifier

from datetime import datetime
import json

class CronFailure:
    CATEGORIES = [
        'resource_exhaustion', 'dependency_failure', 'configuration_error',
        'code_bug', 'network_error', 'permission_error', 'timeout',
        'data_quality', 'external_api', 'hardware_failure', 'security', 'unknown'
    ]

    def __init__(self, job_name, category, summary, severity='minor', timestamp=None):
        assert category in self.CATEGORIES, f"Invalid category: {category}"
        self.job_name = job_name
        self.category = category
        self.summary = summary
        self.severity = severity
        self.timestamp = timestamp or datetime.now()
        self.root_cause = ''
        self.action_items = []
        self.resolved = False

class FailureAnalyzer:
    def __init__(self):
        self.failures = []

    def add_failure(self, failure):
        self.failures.append(failure)

    def analyze(self):
        categories = {}
        for f in self.failures:
            categories[f.category] = categories.get(f.category, 0) + 1
        return dict(sorted(categories.items(), key=lambda x: x[1], reverse=True))

    def get_top_failures(self, n=3):
        analysis = self.analyze()
        print(f"Top {n} failure categories:")
        for i, (cat, count) in enumerate(list(analysis.items())[:n], 1):
            pct = (count / len(self.failures)) * 100
            print(f"  {i}. {cat}: {count} occurrences ({pct:.0f}%)")

    def suggest_improvements(self):
        analysis = self.analyze()
        suggestions = {
            'resource_exhaustion': 'Increase server capacity or stagger job schedules',
            'dependency_failure': 'Add dependency health checks and auto-retry',
            'configuration_error': 'Add schema validation for cron config YAML',
            'timeout': 'Increase timeout or optimize job performance',
            'network_error': 'Add retry with exponential backoff',
        }
        for cat, count in analysis.items():
            if count > 0 and cat in suggestions:
                print(f"  {cat} ({count}): {suggestions[cat]}")

analyzer = FailureAnalyzer()
analyzer.add_failure(CronFailure("daily-backup", "timeout", "Backup exceeded 1-hour timeout"))
analyzer.add_failure(CronFailure("daily-backup", "timeout", "Backup exceeded 1-hour timeout"))
analyzer.add_failure(CronFailure("cache-warm", "dependency_failure", "Redis unavailable"))
analyzer.add_failure(CronFailure("report-gen", "configuration_error", "Wrong output path"))
analyzer.add_failure(CronFailure("health-check", "network_error", "DNS resolution failed"))

analyzer.get_top_failures(3)
analyzer.suggest_improvements()

Expected output:

Top 3 failure categories:
  1. timeout: 2 occurrences (40%)
  2. dependency_failure: 1 occurrences (20%)
  3. configuration_error: 1 occurrences (20%)
  timeout (2): Increase timeout or optimize job performance
  dependency_failure (1): Add dependency health checks and auto-retry
  configuration_error (1): Add schema validation for cron config YAML

Post-Mortem Report Generator

import json
from datetime import datetime

class PostMortemReport:
    def __init__(self, incident_id, job_name, date, severity):
        self.incident_id = incident_id
        self.job_name = job_name
        self.date = date.isoformat() if isinstance(date, datetime) else date
        self.severity = severity
        self.summary = ''
        self.timeline = []
        self.root_cause = ''
        self.action_items = []

    def add_timeline_entry(self, time_str, event):
        self.timeline.append({'time': time_str, 'event': event})

    def add_action(self, description, owner, due_date, priority='medium'):
        self.action_items.append({
            'action': description,
            'owner': owner,
            'due': due_date.isoformat() if isinstance(due_date, datetime) else due_date,
            'priority': priority,
            'status': 'open',
        })

    def generate(self):
        report = {
            'incident_id': self.incident_id,
            'job': self.job_name,
            'date': self.date,
            'severity': self.severity,
            'summary': self.summary,
            'timeline': self.timeline,
            'root_cause': self.root_cause,
            'action_items': self.action_items,
        }
        print(f"Post-Mortem: {self.incident_id} - {self.job_name}")
        print(f"  Severity: {self.severity}")
        print(f"  Root cause: {self.root_cause[:60]}...")
        print(f"  Actions: {len(self.action_items)}")
        return report

report = PostMortemReport("PM-2026-001", "daily-backup", datetime(2026, 6, 28), "critical")
report.summary = "Daily database backup failed due to disk space exhaustion on backup volume"
report.add_timeline_entry("02:00", "Backup started")
report.add_timeline_entry("02:45", "Backup failed: disk full")
report.add_timeline_entry("02:50", "On-call paged")
report.add_timeline_entry("03:15", "Old backups cleaned, disk space recovered")
report.add_timeline_entry("03:30", "Backup re-run successful")
report.root_cause = "Backup volume had only 5% free space. The retention policy cleanup script had not run for 3 days due to a bug."
report.add_action("Fix retention cleanup script bug", "alice@dodatech.com", datetime(2026, 7, 5), "high")
report.add_action("Add disk space alert at 80%", "bob@dodatech.com", datetime(2026, 7, 1), "critical")
report.add_action("Increase backup volume by 200GB", "infra@dodatech.com", datetime(2026, 7, 15), "medium")

report.generate()

Expected output:

Post-Mortem: PM-2026-001 - daily-backup
  Severity: critical
  Root cause: Backup volume had only 5% free space. The retention policy...
  Actions: 3

Common Mistakes

1. No Post-Mortem for Failures

Every cron job failure should have a post-mortem. Without one, the same failure repeats. At minimum, document: what happened, why it happened, and what prevents recurrence. Even minor failures have lessons.

2. Blaming Without Root Cause Analysis

"The backup failed because disk was full" is a symptom, not a root cause. The root cause is "the retention cleanup script had a bug and did not run for 3 days." Dig deeper: ask "why" five times to find the true root cause.

3. Action Items Without Owners

A post-mortem with action items but no owners is worthless. Every action item must have an owner, a due date, and a verification method. The cron platform should track action item completion and alert on overdue items.

4. No Failure Trend Analysis

A single timeout failure is an incident. Five timeout failures in a month is a pattern that requires systemic change. Track failure categories over time. If one category dominates, prioritize fixing it.

5. No Knowledge Base

Every post-mortem contains valuable knowledge that should be shared. Maintain a cron failure knowledge base: searchable by job name, failure category, and root cause. New team members read the KB to learn from past incidents.

Practice Questions

1. What should a cron job post-mortem include?

Incident ID, job name, date, severity, timeline (what happened when), root cause (5 whys), impact (data loss, delay, cost), action items (with owners and due dates), and lessons learned.

2. How do you find the root cause of a cron failure?

Ask "why" five times: Why did backup fail? Disk full. Why was disk full? Retention cleanup did not run. Why did cleanup not run? Script bug. Why was the bug not caught? No test coverage. Why no coverage? No CI/CD for cron scripts.

3. How do you track failure patterns over time?

Classify each failure into a category. Track category frequency over time (weekly, monthly). Create a dashboard showing failure rate trends. Investigate categories with increasing trends.

4. What is the most effective way to prevent cron failure recurrence?

Implement automation that prevents the failure class. For resource failures: add monitoring and auto-scaling. For dependency failures: add health checks and retries. For configuration failures: add validation and CI/CD.

Challenge

Build a failure analysis system: (1) failure classification: 12 categories with sub-types, (2) post-mortem template: incident ID, job, date, severity, timeline, root cause (5 whys), impact, action items, (3) trend analyzer: weekly/monthly failure rate by category, top-N categories, trend direction (increasing/decreasing/stable), (4) action item tracker: owner, due date, status (open/in-progress/verified/closed), alerts on overdue items, (5) knowledge base: searchable database of all post-mortems with category, job, root cause, and prevention tags, (6) integration: alerting system that auto-creates a post-mortem when a job fails, (7) dashboard: failure trends, category distribution, action item status.

FAQ

Mini Project: Failure Analysis System

Build a cron failure analysis system: (1) failure classification into 12 categories with automated detection based on error patterns, (2) post-mortem generator with structured template, timeline, 5-whys root cause analysis, and action items, (3) trend analyzer: failure rate per category over 7/30/90 days, category distribution pie chart, trend arrows (up/down/stable), (4) action tracker: owner, due date, priority, status, automated reminders for overdue items, (5) knowledge base: searchable by job name, category, root cause, prevention, (6) integration: auto-create post-mortem from alert for critical jobs, (7) dashboard: failure rate chart, category breakdown, open actions, knowledge base search.

What's Next

Now that you understand cron failure analysis, build the complete cron project that combines all concepts into a production-ready cron management system.