Cron Metrics and Monitoring — Measuring Cron Job Health and Performance

In this tutorial, you will learn about Cron Metrics and Monitoring. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron metrics and monitoring: instrument cron jobs to emit execution metrics, track duration and success rates, set up Prometheus exporters, build Grafana dashboards, and monitor cron job health across your infrastructure.

What You Learn

You will learn how to monitor cron jobs: instrumenting jobs for metrics collection, exposing Prometheus metrics, building monitoring dashboards, tracking SLA Compliance, and alerting on cron job health issues.

Why It Matters

Without monitoring, you do not know if cron jobs are running, how long they take, or if they are succeeding. A silently failing cron job that runs every 15 minutes fails 96 times before someone notices. Monitoring turns silent failures into visible problems.

Real-World Use

DodaTech monitors 400+ cron jobs with Prometheus. Each job exposes: duration seconds, success/failure, start timestamp, and records processed. A dashboard shows: jobs failing, jobs taking longer than expected, and jobs that have not run recently. Alerting catches any job that fails more than twice consecutively.

Cron Metrics Exporter

import time
import json
from datetime import datetime

class CronMetrics:
    def __init__(self, job_name):
        self.job_name = job_name
        self.start_time = None
        self.metrics = {}

    def start(self):
        self.start_time = time.time()
        self.metrics['start_timestamp'] = time.time()

    def finish(self, success=True, records_processed=0, error=None):
        duration = time.time() - self.start_time
        self.metrics.update({
            'duration_seconds': round(duration, 3),
            'success': 1 if success else 0,
            'failure': 0 if success else 1,
            'records_processed': records_processed,
            'error': error or '',
            'end_timestamp': time.time(),
        })
        return self

    def expose_prometheus(self):
        labels = f'job="{self.job_name}"'
        return '\n'.join([
            f'# HELP cron_job_duration_seconds Duration of last cron job execution',
            f'# TYPE cron_job_duration_seconds gauge',
            f'cron_job_duration_seconds{{{labels}}} {self.metrics["duration_seconds"]}',
            f'# HELP cron_job_success Whether last cron job execution succeeded (1=success, 0=failure)',
            f'# TYPE cron_job_success gauge',
            f'cron_job_success{{{labels}}} {self.metrics["success"]}',
            f'# HELP cron_job_records_processed Number of records processed in last execution',
            f'# TYPE cron_job_records_processed gauge',
            f'cron_job_records_processed{{{labels}}} {self.metrics["records_processed"]}',
            f'# HELP cron_job_start_timestamp Start time of last execution (Unix epoch)',
            f'# TYPE cron_job_start_timestamp gauge',
            f'cron_job_start_timestamp{{{labels}}} {self.metrics["start_timestamp"]}',
        ])

    def expose_json(self):
        return json.dumps({'job': self.job_name, 'metrics': self.metrics})

metrics = CronMetrics("daily-backup")
metrics.start()
time.sleep(0.1)
metrics.finish(success=True, records_processed=15000)
print(metrics.expose_prometheus())

Expected output:

# HELP cron_job_duration_seconds Duration of last cron job execution
# TYPE cron_job_duration_seconds gauge
cron_job_duration_seconds{job="daily-backup"} 0.1
# HELP cron_job_success Whether last cron job execution succeeded (1=success, 0=failure)
# TYPE cron_job_success gauge
cron_job_success{job="daily-backup"} 1
# HELP cron_job_records_processed Number of records processed in last execution
# TYPE cron_job_records_processed gauge
cron_job_records_processed{job="daily-backup"} 15000
# HELP cron_job_start_timestamp Start time of last execution (Unix epoch)
# TYPE cron_job_start_timestamp gauge
cron_job_start_timestamp{job="daily-backup"} 1719532800.0

Cron SLA Monitor

import time
from datetime import datetime, timedelta
import statistics

class CronSLAMonitor:
    def __init__(self, job_name, expected_interval_minutes, max_duration_minutes):
        self.job_name = job_name
        self.expected_interval = expected_interval_minutes
        self.max_duration = max_duration_minutes
        self.executions = []

    def record_execution(self, duration_minutes, success):
        self.executions.append({
            'timestamp': datetime.now(),
            'duration_minutes': duration_minutes,
            'success': success,
        })
        if len(self.executions) > 100:
            self.executions.pop(0)

    def check_sla(self):
        now = datetime.now()

        recent = [e for e in self.executions if now - e['timestamp'] < timedelta(hours=24)]
        if not recent:
            return {'sla_ok': False, 'reason': 'No executions in last 24 hours'}

        last_run = recent[-1]
        last_run_age = (now - last_run['timestamp']).total_seconds() / 60

        duration_ok = last_run['duration_minutes'] <= self.max_duration
        interval_ok = last_run_age <= self.expected_interval * 1.5
        success_ok = all(e['success'] for e in recent[-5:])

        sla_ok = duration_ok and interval_ok and success_ok

        return {
            'sla_ok': sla_ok,
            'last_run_age_minutes': round(last_run_age, 1),
            'last_duration_minutes': last_run['duration_minutes'],
            'duration_ok': duration_ok,
            'interval_ok': interval_ok,
            'success_ok': success_ok,
        }

monitor = CronSLAMonitor("daily-backup", expected_interval_minutes=1440, max_duration_minutes=60)
monitor.record_execution(duration_minutes=25, success=True)
monitor.record_execution(duration_minutes=30, success=True)
monitor.record_execution(duration_minutes=22, success=True)

sla = monitor.check_sla()
print(f"SLA OK: {sla['sla_ok']}")
print(f"  Last run age: {sla['last_run_age_minutes']} min")
print(f"  Last duration: {sla['last_duration_minutes']} min")

Expected output:

SLA OK: True
  Last run age: 0.0 min
  Last duration: 22 min

Common Mistakes

1. No Monitoring at All

The most common mistake. A cron job that fails silently is a ticking time bomb. Every cron job must expose at least: whether it ran, how long it took, and whether it succeeded.

2. Only Monitoring Success/Failure

Knowing a job succeeded ignores performance degradation. If a backup that took 30 minutes suddenly takes 90 minutes, something is wrong. Monitor duration trends and alert on significant changes.

3. No Time-Since-Last-Run Alert

If a cron job fails to start at all (cron daemon down, system rebooted, schedule misconfigured), no metrics are emitted. A "time since last successful run" metric with an alert if it exceeds 2x the expected interval catches this.

4. Metrics Without Labels

A Prometheus metric cron_job_duration_seconds{} without a job label cannot distinguish between jobs. Always include labels: job name, hostname, environment, and job type.

5. No Historical Retention

If you only keep the last metric value, you cannot see trends. Push metrics to Prometheus or a time-series database. Retain at least 30 days of data for trend analysis and SLA reporting.

Practice Questions

1. What metrics should every cron job expose?

Duration seconds, success/failure (1/0), start timestamp, records processed, and hostname. These five metrics give complete visibility into cron job health and performance.

2. How do you expose cron metrics to Prometheus?

Write metrics to a file in Prometheus text format, or push to Pushgateway for short-lived jobs. Use a file-based approach: the cron job writes metrics to a well-known path, and a node_exporter textfile collector reads them.

3. What is a good SLA for a daily cron job?

Daily jobs should run within 24 hours +/- 30 minutes (interval SLA), complete within the expected duration (duration SLA), and succeed 99.5%+ of the time (reliability SLA). Alert on any SLA breach.

4. How do you detect a cron job that stopped running?

Monitor "time since last successful execution". Alert if the age exceeds 2x the expected interval. For a daily job, alert if no success in 48 hours. For an hourly job, alert if no success in 2 hours.

Challenge

Build a cron monitoring system: (1) metrics collector: each cron job writes Prometheus-format metrics to a textfile after execution, (2) Prometheus node_exporter textfile collector scrapes these files, (3) key metrics: duration_seconds, success, failure, records_processed, start_timestamp, (4) SLA definitions per job: expected interval, max duration, required success rate, (5) Grafana dashboard: job status grid (green/red), duration trends, success rate over time, time since last run, (6) alerting: no successful run in 2x interval, duration > 2x expected, 3 consecutive failures, (7) monthly SLA report: uptime percentage per job, total execution count, average duration.

FAQ

Mini Project: Cron Monitoring System

Build a cron monitoring system: (1) metrics library: each job calls start()/finish() to record duration, success, records processed, (2) Prometheus exposition: write to textfile in /var/lib/node_exporter/textfile/cron.prom, (3) Prometheus recording rules: compute success rate over 7 days, average duration, p95 duration, (4) Grafana dashboard: job status grid, duration trends per job, success rate heatmap (24h x 7d), time-since-last-run table, SLA compliance per job, (5) alerting rules: CronJobMissing (no data for 2x interval), CronJobSlow (duration > 2x expected), CronJobFailing (3 consecutive failures), (6) SLA reporter: monthly summary with uptime, average duration, total executions per job.

What's Next

Now that you understand cron metrics and monitoring, explore cron alerting strategies, then learn about SLA compliance for cron jobs.