Airflow Pool Slots Exhausted Fix

In this tutorial, you'll learn about Airflow Pool Slots Exhausted Fix. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Tasks remain in "Scheduled" state indefinitely:

Task is in 'scheduled' state but not picked up by any worker.

The task is stuck because the pool it belongs to has no available slots. Each pool has a fixed number of slots. When all slots are occupied by running tasks, new tasks wait in the queue until a slot opens up.

Step-by-Step Fix

1. Check pool utilization

RIGHT — view pool status:

Airflow UI > Admin > Pools

This shows:

Pool: default_pool
Slots: 128
Used Slots: 128 (Running)
Queued Slots: 5
Free Slots: 0

If free slots = 0, no new tasks can run in that pool.

2. Increase pool slots

WRONG — slot count too low for the number of tasks:

from airflow.models import Pool

# Create or update pool
Pool.create_or_update_pool(
    pool_name="data_pipeline",
    slots=4,  # Too few for parallel tasks
    description="Data pipeline tasks"
)

RIGHT — increase slots:

Pool.create_or_update_pool(
    pool_name="data_pipeline",
    slots=16,
    description="Data pipeline tasks"
)

3. Assign tasks to different pools

WRONG — all tasks use the default pool:

task1 = DummyOperator(task_id="task1", pool="default_pool")
task2 = DummyOperator(task_id="task2", pool="default_pool")
# Both compete for default_pool slots

RIGHT — separate by priority:

high_priority = DummyOperator(task_id="critical", pool="high_priority")
low_priority = DummyOperator(task_id="backfill", pool="background")

Create pools:

Admin > Pools > Add:
- high_priority: 10 slots
- background: 50 slots
- default_pool: 128 slots

4. Reduce task concurrency per DAG

WRONG — too many tasks running in parallel for the same DAG:

with DAG(
    dag_id="my_dag",
    concurrency=64,  # 64 tasks at once — may exhaust pools
    ...
):

RIGHT — limit concurrency:

with DAG(
    dag_id="my_dag",
    concurrency=4,  # Only 4 simultaneous tasks
    max_active_tasks=4,
    max_active_runs=1,
    ...
):

5. Release stuck pool slots

WRONG — tasks fail but slots remain occupied:

# A task that fails still holds its slot until marked as failed

RIGHT — use pool_slots parameter:

task = PythonOperator(
    task_id="heavy_task",
    pool="data_pipeline",
    pool_slots=2,  # Task claims 2 slots (useful for resource-heavy tasks)
)

For stuck slots, clear the task instance:

UI > DAG > Task > Clear > Clear the task instance

6. Monitor with Airflow metrics

# Prometheus metrics for pool monitoring
from airflow.models import Pool

def check_pools():
    for pool in Pool.get_pools():
        used = pool.used_slots()
        free = pool.open_slots()
        total = pool.slots
        print(f"{pool.pool}: {used}/{total} used, {free} free")

Expected output: tasks start running after pool slots become available.

Prevention

• Monitor pool utilization regularly through the Airflow UI.
• Set realistic slot counts based on your worker capacity.
• Use separate pools for high and low priority workloads.
• Set task concurrency lower than pool slots to prevent deadlocks.
• Alert on pool slot exhaustion via Airflow's built-in metrics.

Common Mistakes with pool slots

1. Overlapping type class instances that cause GHC to reject the program with ambiguous dispatch errors
2. Non-exhaustive pattern matches that compile with warnings then crash at runtime
3. Misunderstanding that String is [Char] with poor performance for large text operations

These mistakes appear frequently in real-world AIRFLOW code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ