Airflow Variable Error Fix

In this tutorial, you'll learn about Airflow Variable Error Fix. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Reading a variable from Airflow fails:

from airflow.models import Variable

value = Variable.get("my_config")  # KeyError: my_config

Variable.get() raises a KeyError when the variable doesn't exist. Variables are stored in the Airflow database and must be created through the UI, CLI, or API before they can be read. The default_var parameter can prevent the KeyError.

Step-by-Step Fix

1. Use default_var to prevent KeyError

WRONG — no default, raises exception:

value = Variable.get("my_config")
# KeyError if not set

RIGHT — provide a default:

value = Variable.get("my_config", default_var=None)
# Returns None if not set, no exception

Or use deserialize_json for JSON variables:

config = Variable.get("my_config", default_var={}, deserialize_json=True)

2. Create variables through CLI or UI

WRONG — hardcoding sensitive values:

api_key = "sk-1234567890"  # Hardcoded in the DAG file

RIGHT — use Airflow Variables:

# Create via CLI
airflow variables set my_config '{"key": "value"}'

# Create from JSON file
airflow variables import variables.json

Or via the UI: Admin > Variables > Add.

3. Handle JSON Serialization

WRONG — storing JSON as a plain string:

# Variable value: '{"key": "value"}'
json_var = Variable.get("json_config")
print(json_var["key"])  # TypeError: string indices must be integers

RIGHT — use deserialize_json:

json_var = Variable.get("json_config", deserialize_json=True)
print(json_var["key"])  # 'value'

4. Set variables programmatically in DAGs

from airflow.models import Variable

# Set a default if not exists
if Variable.get("my_config", default_var=None) is None:
    Variable.set("my_config", {"setting": "default"}, serialize_json=True)

5. Export and import variables

# Export all variables to JSON
airflow variables export /path/to/variables.json

# Import from JSON
airflow variables import /path/to/variables.json

JSON format:

{
    "my_config": "{\"key\": \"value\"}",
    "api_key": "sk-xxx",
    "env": "production"
}

6. Use Secrets backend for sensitive variables

# airflow.cfg
[secrets]
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {"connections_prefix": "/airflow/connections", "variables_prefix": "/airflow/variables"}

Then create variables in AWS Parameter Store under /airflow/variables/:

aws ssm put-parameter \
    --name "/airflow/variables/db_password" \
    --value "supersecret" \
    --type SecureString

Expected output: Variable.get("my_config") returns the expected value.

Prevention

• Always provide default_var for Variable.get() calls.
• Create variables before the DAG runs that need them.
• Use deserialize_json=True for structured data.
• Use Secrets backend (AWS SSM, GCP Secret Manager) for sensitive values.
• Export variables to version control as JSON files.

Common Mistakes with variable error

1. Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable
2. Using head and tail instead of pattern matching, causing runtime errors on empty lists
3. Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks

These mistakes appear frequently in real-world AIRFLOW code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ