Airflow Backfill Error Fix

In this tutorial, you'll learn about Airflow Backfill Error Fix. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Running a backfill produces no output:

airflow dags backfill my_dag --start-date 2024-01-01 --end-date 2024-01-05

Backfill completed. 0 runs created.

The backfill created zero runs because the scheduler already created runs for those dates (catchup was True), or the DAG's start_date is after the backfill range. Backfills only create runs for intervals that don't already have a DAG run.

Step-by-Step Fix

1. Force backfill with --reset-dagruns

WRONG — backfill doesn't create runs for existing dates:

airflow dags backfill my_dag --start-date 2024-01-01 --end-date 2024-01-05
# 0 runs created — runs already exist

RIGHT — reset existing runs:

airflow dags backfill my_dag \
    --start-date 2024-01-01 \
    --end-date 2024-01-05 \
    --reset-dagruns  # Deletes and recreates runs

2. Use --rerun-failed-tasks for partial retries

WRONG — full backfill reruns successful tasks unnecessarily:

airflow dags backfill my_dag --start-date 2024-01-01 --end-date 2024-06-01

RIGHT — only rerun failed tasks:

airflow dags backfill my_dag \
    --start-date 2024-01-01 \
    --end-date 2024-06-01 \
    --rerun-failed-tasks  # Only re-executes failed tasks

3. Check DAG catchup setting

WRONG — catchup=False prevents backfill from creating past runs:

with DAG(
    dag_id="my_dag",
    start_date=datetime(2024, 1, 1),
    schedule="@daily",
    catchup=False,  # Skips past dates!
):

RIGHT — temporarily enable catchup or use backfill with --trickle:

# Backfill works even with catchup=False
# But it will only create runs that don't exist
airflow dags backfill my_dag --start-date 2024-01-01 --end-date 2024-01-05

4. Run backfill sequentially

WRONG — running backfill in parallel causes database locks:

airflow dags backfill my_dag \
    --start-date 2024-01-01 \
    --end-date 2024-06-01 \
    --delay-on-limit 60  # Pause between runs

RIGHT — use sequential backfill for large ranges:

# Run one day at a time
for date in $(seq -w 1 31); do
    airflow dags backfill my_dag \
        --start-date "2024-01-$date" \
        --end-date "2024-01-$date" \
        --rerun-failed-tasks
done

5. Backfill for specific DAG intervals

WRONG — using dates that don't align with the schedule:

# DAG runs hourly, backfill uses daily dates
airflow dags backfill my_dag \
    --start-date 2024-01-01 \
    --end-date 2024-01-02
# This covers 24 intervals (if hourly)

RIGHT — align dates with schedule:

# For @hourly schedule:
airflow dags backfill my_dag \
    --start-date 2024-01-01T00:00:00 \
    --end-date 2024-01-01T23:00:00

# For @daily schedule:
airflow dags backfill my_dag \
    --start-date 2024-01-01 \
    --end-date 2024-01-31

6. Backfill with custom parameters

airflow dags backfill my_dag \
    --start-date 2024-01-01 \
    --end-date 2024-01-05 \
    -t "my_task"  # Only run specific task
    --ignore-first-depends-on-past \
    --ignore-dependencies

Check backfill status:

airflow dags list-runs my_dag \
    --state failed \
    --start-date 2024-01-01 \
    --end-date 2024-01-05

Expected output: backfill creates and runs DAG runs for the specified interval.

Prevention

• Use --reset-dagruns for full re-runs of past intervals.
• Use --rerun-failed-tasks for partial backfills.
• Test backfill with a single day before running large ranges.
• Set catchup=False for production DAGs, use backfill explicitly when needed.
• Monitor backfill progress in the UI.

Common Mistakes with backfill error

1. Non-exhaustive pattern matches that compile with warnings then crash at runtime
2. Misunderstanding that String is [Char] with poor performance for large text operations
3. Using foldl instead of foldl' causing stack overflow on large lists

These mistakes appear frequently in real-world AIRFLOW code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ