Airflow XCom Not Passing Data Fix

In this tutorial, you'll learn about Airflow XCom Not Passing Data Fix. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

A downstream task cannot read the XCom value from an upstream task:

def push_data(**kwargs):
    return {"key": "value"}

def pull_data(**kwargs):
    ti = kwargs["ti"]
    data = ti.xcom_pull(task_ids="push_data")  # Returns None

XCom values are not automatically pushed unless the task returns a value or explicitly calls xcom_push. Only the return value of the Python callable is automatically pushed. Values stored without a key can only be pulled by index (default key is "return_value").

Step-by-Step Fix

1. Use the return value for automatic XCom push

WRONG — not returning a value:

def push_data(**kwargs):
    data = {"key": "value"}
    # No return — nothing pushed to XCom

RIGHT — return the value:

def push_data(**kwargs):
    return {"key": "value"}  # Automatically pushed to XCom with key "return_value"

Pull it:

def pull_data(**kwargs):
    ti = kwargs["ti"]
    data = ti.xcom_pull(task_ids="push_data")
    # data = {"key": "value"}

2. Push XCom with explicit keys

WRONG — pushing without a key makes retrieval ambiguous:

def push_data(**kwargs):
    ti = kwargs["ti"]
    ti.xcom_push(key="user_data", value={"name": "Alice"})

RIGHT — use consistent keys:

def push_data(**kwargs):
    ti = kwargs["ti"]
    ti.xcom_push(key="user_data", value={"name": "Alice"})

def pull_data(**kwargs):
    ti = kwargs["ti"]
    data = ti.xcom_pull(task_ids="push_data", key="user_data")
    # data = {"name": "Alice"}

3. Keep XCom values small

WRONG — pushing large data (>48KB) to XCom:

def push_data(**kwargs):
    return [i for i in range(1000000)]  # Huge list — will be slow

RIGHT — store large data in a file or database:

def push_data(**kwargs):
    large_data = [i for i in range(1000000)]
    # Write to a file in a shared storage
    with open("/tmp/large_data.json", "w") as f:
        json.dump(large_data, f)
    # Push the file path instead
    return "/tmp/large_data.json"

def pull_data(**kwargs):
    ti = kwargs["ti"]
    file_path = ti.xcom_pull(task_ids="push_data")
    with open(file_path) as f:
        return json.load(f)

Or configure a larger XCom backend:

# airflow.cfg
[core]
xcom_backend = airflow.models.xcom.BaseXCom
xcom_max_length = 100000  # Bytes (default 48000)

4. Fix XCom in dynamically mapped tasks

WRONG — pulling XCom from mapped tasks without index:

def pull_from_mapped(**kwargs):
    ti = kwargs["ti"]
    # Mapped tasks generate multiple XComs
    data = ti.xcom_pull(task_ids="mapped_task")
    # Returns a list of all return values

RIGHT — handle mapped XComs:

def pull_from_mapped(**kwargs):
    ti = kwargs["ti"]
    all_results = ti.xcom_pull(task_ids="mapped_task")
    for result in all_results:
        process(result)

5. Check XCom in the UI

Admin > XComs

This shows all XCom entries. Filter by:

• DAG ID
• Task ID
• Key
• Date range

Verify that the upstream task actually pushed the XCom.

Expected output: downstream task receives the expected data from XCom.

Prevention

• Always return values from Python callables that should pass data.
• Use explicit keys when pushing multiple XComs from one task.
• Keep XCom values small — use file storage for large data.
• Check the XCom UI to verify data is being stored.
• Use XCom with the include_prior_dates parameter for cross-DAG pulls.

Common Mistakes with xcom error

1. Using head and tail instead of pattern matching, causing runtime errors on empty lists
2. Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks
3. Using return to exit a function early instead of wrapping a pure value in the monad

These mistakes appear frequently in real-world AIRFLOW code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ

data = ti.xcom_pull(dag_id="other_dag", task_ids="some_task", key="return_value")