Apache Airflow Scheduler Not Running Fix

In this tutorial, you'll learn about Apache Airflow Scheduler Not Running Fix. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

DAGs are not running even though they are set to schedule_interval — the scheduler Process crashed, its heartbeat stopped, or it cannot connect to the metadata database.

Step-by-Step Fix

1. Check if the scheduler is running

ps aux | grep "airflow scheduler"

Expected output shows a running Process:

airflow  12345  0.5  0.8  123456  7890 ?  S  10:00  0:01 airflow scheduler

2. Restart the scheduler

airflow scheduler -D

The -D flag runs it as a daemon.

3. Check scheduler logs

cat ~/airflow/logs/scheduler/latest/*.log | tail -50

Expected output shows heartbeats:

[2024-01-15 10:30:00] {scheduler_job.py:123} INFO - Starting the scheduler
[2024-01-15 10:30:05] {scheduler_job.py:456} INFO - Heartbeat sent

4. Verify database connection

airflow db check

Expected output: Connection successful.

5. Wrong vs Right approach

# Wrong — only running webserver without scheduler
airflow webserver -p 8080
# DAGs appear but never run

# Right — run both scheduler and webserver
airflow scheduler -D
airflow webserver -p 8080 -D
# DAGs start executing on schedule

Common Mistakes

Prevention

• Run scheduler as a systemd service with auto-restart.
• Monitor scheduler heartbeats with Airflow's built-in health endpoint.
• Use a production-grade database (PostgreSQL) for the metadata DB.
• Set up alerting on scheduler Process failure.

DodaTech Tools

Doda Browser's Airflow health monitor tracks scheduler heartbeats and alerts on failures. DodaZIP archives scheduler logs for forensic analysis. Durga Antivirus Pro monitors for anomalous scheduler behavior that could indicate resource exhaustion.

Common Mistakes with scheduler not running

1. Using foldl instead of foldl' causing stack overflow on large lists
2. Forgetting deriving (Show, Eq) on custom data types needed for debugging
3. Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable

These mistakes appear frequently in real-world AIRFLOW code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ