Airflow Kubernetes Executor Error Fix

In this tutorial, you'll learn about Airflow Kubernetes Executor Error Fix. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Tasks using the KubernetesExecutor fail to start:

Pod Launching failed: [Pod is in phase: Pending]
[Pod is in phase: Failed]

The KubernetesExecutor creates a worker pod for each task. Pods can get stuck in Pending state due to resource constraints, missing ConfigMaps, or incorrect image pull secrets. Failed pods usually indicate an error in the worker startup Process.

Step-by-Step Fix

1. Check pod status with kubectl

WRONG — only looking at Airflow UI:

RIGHT — check actual pod status:

# List pods for the namespace
kubectl get pods -n airflow

# Describe the stuck pod
kubectl describe pod my-task-pod -n airflow

# Check pod logs
kubectl logs my-task-pod -n airflow

Pod Status: Pending
Events:
  - 0/3 nodes are available: 1 Insufficient memory, 2 Insufficient CPU

2. Adjust pod resource requests

WRONG — resource requests too high for the cluster:

from airflow.kubernetes.pod import PodDefaults

executor_config = {
    "requests": {
        "memory": "4Gi",   # Too high for available nodes
        "cpu": "2"         # Too high
    }
}

task = PythonOperator(
    task_id="my_task",
    python_callable=my_func,
    executor_config=executor_config,
)

RIGHT — match resource availability:

executor_config = {
    "requests": {
        "memory": "512Mi",
        "cpu": "0.5"
    },
    "limits": {
        "memory": "1Gi",
        "cpu": "1"
    }
}

3. Verify Kubernetes configuration

WRONG — missing or incorrect kube config:

[kubernetes_executor]
kubernetes_kubeconfig = /home/airflow/.kube/config  # File doesn't exist

RIGHT — configure in-cluster or with a valid kubeconfig:

For in-cluster (Airflow running inside Kubernetes):

[kubernetes_executor]
in_cluster = True
kubernetes_kubeconfig =

For external connection:

[kubernetes_executor]
in_cluster = False
kubernetes_kubeconfig = /path/to/kubeconfig
kubernetes_namespace = airflow

Test connection:

kubectl --kubeconfig=/path/to/kubeconfig get nodes

4. Configure image pull secrets

WRONG — using a private image without credentials:

executor_config = {
    "image": "my-private-registry.com/airflow-worker:latest",
    # Missing image_pull_secrets
}

RIGHT — add pull secret:

kubectl create secret docker-registry regcred \
    --docker-server=my-private-registry.com \
    --docker-username=user \
    --docker-password=pass \
    -n airflow

executor_config = {
    "image": "my-private-registry.com/airflow-worker:latest",
    "image_pull_secrets": "regcred",
}

5. Set node affinity and tolerations

WRONG — tasks scheduled on nodes without required components:

executor_config = {
    "node_selector": {
        "type": "airflow-worker"  # Node label doesn't exist
    }
}

RIGHT — use labels and tolerations:

executor_config = {
    "node_selector": {
        "kubernetes.io/os": "linux"
    },
    "tolerations": [{
        "key": "dedicated",
        "operator": "Equal",
        "value": "airflow",
        "effect": "NoSchedule]
    }]
}

6. Mount ConfigMap for DAGs

executor_config = {
    "volumes": [{
        "name": "dags",
        "configMap": {
            "name": "airflow-dags]
        }
    }],
    "volume_mounts": [{
        "name": "dags",
        "mountPath": "/opt/airflow/dags]
    }]
}

Expected output: worker pods start successfully and execute tasks.

Prevention

• Set realistic pod resource requests that cluster nodes can satisfy.
• Use in_cluster=True when Airflow runs inside Kubernetes.
• Create and configure image pull secrets for private registries.
• Monitor pod startup times and failure reasons via kubectl.
• Test pod configuration with a simple task before production use.

Common Mistakes with Kubernetes executor

1. Misunderstanding that String is [Char] with poor performance for large text operations
2. Using foldl instead of foldl' causing stack overflow on large lists
3. Forgetting deriving (Show, Eq) on custom data types needed for debugging

These mistakes appear frequently in real-world AIRFLOW code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ