Cron Dependency Management — Ordering and Coordinating Dependent Cron Jobs

In this tutorial, you will learn about Cron Dependency Management. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn cron dependency management: coordinate dependent Cron Jobs so they run in the correct order, check that required data and services are available before executing, prevent overlapping dependent runs, and handle partial dependency failures gracefully.

What You Learn

You will learn how to manage dependencies between cron jobs: dependency declaration and validation, ordered job execution, upstream data readiness checks, and failure propagation.

Why It Matters

A report generation job that runs before the data aggregation job produces stale data. A cache warming job that runs before the database finishes maintenance caches wrong data. Dependency management prevents these order-dependent failures.

Real-World Use

DodaTech's cron dependency graph specifies that "aggregate-data" runs at 2 AM, "generate-reports" depends on aggregate-data and runs at 3 AM, "distribute-reports" depends on generate-reports and runs at 4 AM. Each job checks that its upstream dependency completed successfully before starting.

Dependency Graph

import time
from datetime import datetime

class CronDependencyNode:
    def __init__(self, name, execute_fn, dependencies=None):
        self.name = name
        self.execute = execute_fn
        self.dependencies = dependencies or []
        self.completed = False
        self.failed = False
        self.duration = 0

class CronDependencyGraph:
    def __init__(self):
        self.nodes = {}

    def add_node(self, node):
        self.nodes[node.name] = node

    def can_run(self, node_name):
        node = self.nodes.get(node_name)
        if not node:
            return False
        for dep_name in node.dependencies:
            dep = self.nodes.get(dep_name)
            if not dep or not dep.completed or dep.failed:
                return False
        return True

    def get_ready_nodes(self):
        ready = []
        for name, node in self.nodes.items():
            if not node.completed and not node.failed and self.can_run(name):
                ready.append(node)
        return ready

    def run_ready(self):
        ready = self.get_ready_nodes()
        for node in ready:
            start = time.time()
            print(f"[{datetime.now().strftime('%H:%M:%S')}] Running: {node.name}")
            try:
                node.execute()
                node.completed = True
                node.duration = time.time() - start
                print(f"  Completed ({node.duration:.1f}s)")
            except Exception as e:
                node.failed = True
                print(f"  FAILED: {e}")

    def run_all(self):
        while True:
            ready = self.get_ready_nodes()
            if not ready:
                remaining = [n for n in self.nodes.values() if not n.completed and not n.failed]
                if remaining:
                    print(f"Deadlock or missing dependency for: {[n.name for n in remaining]}")
                break
            self.run_ready()

def aggregate():
    time.sleep(0.05)
def generate():
    time.sleep(0.05)
def distribute():
    time.sleep(0.05)
def archive():
    time.sleep(0.05)

graph = CronDependencyGraph()
graph.add_node(CronDependencyNode("aggregate-data", aggregate))
graph.add_node(CronDependencyNode("generate-reports", generate, dependencies=["aggregate-data"]))
graph.add_node(CronDependencyNode("distribute-reports", distribute, dependencies=["generate-reports"]))
graph.add_node(CronDependencyNode("archive-data", archive, dependencies=["aggregate-data"]))

graph.run_all()

Expected output:

[00:00:00] Running: aggregate-data
  Completed (0.1s)
[00:00:00] Running: generate-reports
  Completed (0.1s)
[00:00:00] Running: archive-data
  Completed (0.1s)
[00:00:00] Running: distribute-reports
  Completed (0.1s)

Upstream Data Readiness Check

import time
from datetime import datetime, timedelta

class UpstreamChecker:
    def __init__(self):
        self.job_timestamps = {}

    def record_completion(self, job_name):
        self.job_timestamps[job_name] = datetime.now()

    def is_data_fresh(self, upstream_job, max_age_minutes=60):
        if upstream_job not in self.job_timestamps:
            print(f"  DATA NOT READY: {upstream_job} has never run")
            return False
        age = datetime.now() - self.job_timestamps[upstream_job]
        if age > timedelta(minutes=max_age_minutes):
            print(f"  DATA STALE: {upstream_job} ran {age.total_seconds() / 60:.0f} minutes ago (max: {max_age_minutes})")
            return False
        print(f"  Data fresh: {upstream_job} ran {age.total_seconds() / 60:.1f} min ago")
        return True

    def wait_for_upstream(self, upstream_job, timeout_minutes=30):
        deadline = datetime.now() + timedelta(minutes=timeout_minutes)
        while datetime.now() < deadline:
            if self.is_data_fresh(upstream_job, max_age_minutes=120):
                return True
            print(f"  Waiting for {upstream_job}...")
            time.sleep(0.5)
        print(f"  TIMEOUT: {upstream_job} did not complete within {timeout_minutes} minutes")
        return False

checker = UpstreamChecker()
checker.record_completion("aggregate-data")
checker.is_data_fresh("aggregate-data", max_age_minutes=60)

Expected output:

  Data fresh: aggregate-data ran 0.0 min ago

Common Mistakes

1. Assuming Jobs Run on Time

A job scheduled at 2 AM may not finish until 4 AM if the data volume is high. The downstream job scheduled at 3 AM starts with stale data. Use completion markers instead of schedules: check that upstream completed before starting downstream.

2. Circular Dependencies

Job A needs Job B to finish, and Job B needs Job A to finish. Neither runs. Detect circular dependencies at configuration time. Use a directed acyclic graph (DAG) and reject any configuration with cycles.

3. No Timeout for Dependency Wait

A downstream job that waits indefinitely for an upstream job that never completes holds resources forever. Implement a timeout: if the upstream does not complete within N minutes, alert and exit. The next cron run will retry.

4. Cascading Failures Without Isolation

If the aggregate-data job fails, the generate-reports and distribute-reports jobs also fail. This is expected, but the failure notifications should distinguish: aggregate-data failed vs. downstream jobs skipped due to dependency failure.

5. Dependency Check Without Heartbeat

A job that hangs (Process running but no progress) appears as "running" to dependency checks. Implement heartbeats: each job writes a heartbeat timestamp every N seconds. If the heartbeat stops, assume the job is hung.

Practice Questions

1. How do you declare dependencies between cron jobs?

Store dependency information in configuration: each job lists its upstream dependencies. At runtime, the job checks that all upstream jobs have completed successfully before executing. Completion is tracked via a shared state store.

2. How do you detect circular dependencies?

Build a dependency graph and detect cycles using topological sort. If the graph cannot be topologically sorted, there is a cycle. Reject circular dependency configurations at deployment time.

3. What is the difference between schedule-based and completion-based job ordering?

Schedule-based uses time to order jobs: job A at 2 AM, job B at 3 AM. Completion-based checks that job A finished before starting job B. Completion-based is more reliable when job duration varies.

4. How do you handle a dependency that never completes?

Implement a timeout (e.g., upstream should complete within 4 hours). If the timeout expires, page the on-call engineer. The downstream job can run with stale data after a warning, or skip entirely and mark itself as failed.

Challenge

Build a dependency management system: (1) dependency graph with DAG validation (reject cycles), (2) completion tracking via shared state (Redis or database), (3) data freshness check: verify upstream data is not older than N minutes, (4) timeout handling: wait for upstream with timeout (configurable per dependency), (5) failure propagation: if upstream fails, skip downstream and mark as "skipped (dependency failure)", (6) heartbeat monitoring: each job writes heartbeat, alert if heartbeat stops, (7) dashboard: dependency graph visualization, job status, freshness indicators.

FAQ

Mini Project: Cron Dependency Orchestrator

Build a dependency-aware cron orchestrator: (1) DAG configuration in YAML: each job lists dependencies, timeout, retry count, (2) completion tracking in Redis: SET/GET with TTL (24 hours for daily jobs), (3) dependency validator: at startup, check all upstream deps completed within expected timeframe, (4) readiness check: verify upstream data freshness (age < max_age_minutes), (5) timeout: configurable wait per dependency (30 min default), alert on timeout, (6) failure handling: skip downstream if upstream failed, mark as "skipped - dependency failure", (7) dashboard: DAG visualization with status colors (green=completed, red=failed, yellow=running, gray=pending).

What's Next

Now that you understand cron dependency management, explore retry patterns for cron jobs, then learn about cron metrics and monitoring.