Celery Task Expiration: Soft Time Limits, Hard Time Limits, and Deadline Management

In this tutorial, you will learn about Celery Task Expiration: Soft Time Limits, Hard Time Limits, and Deadline Management. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery task expiration manages task lifecycle boundaries through time limits that terminate long-running tasks, expiration dates that discard stale messages, and deadline-based scheduling that prevents backlog processing of outdated work items.

flowchart TD
    T[Task Submitted] --> E{Has Expiration?}
    E -->|Yes, expired| D[Discarded by Broker]
    E -->|No| W[Worker Executes]
    W --> S{Soft Time Limit?}
    S -->|Exceeded| Exc[SoftTimeLimitExceeded
Graceful Shutdown]
    S -->|OK| H{Hard Time Limit?}
    H -->|Exceeded| Kill[Process Killed
Forcefully]
    H -->|OK| Done[Task Complete]

What You'll Learn

• Soft and hard time limits
• Task expiration with ETA
• Queue-level message TTL
• Deadline-based task scheduling
• Graceful vs forced termination

Why It Matters

Without expiration controls, runaway tasks consume worker resources indefinitely, stale tasks Process outdated data, and backlogs grow unbounded. Time limits protect system stability by bounding worst-case execution time.

Real-World Use

DodaTech's file scanner sets a 30-second hard limit on individual file scans. If malware analysis hangs on a corrupted file, the hard limit kills the process and the file is quarantined. Task expiration discards scan requests older than 1 hour, preventing backlog processing.

Soft and Hard Time Limits

from celery import Celery
from celery.exceptions import SoftTimeLimitExceeded
import time

app = Celery('timelimits', broker='redis://localhost:6379/0')

@app.task(bind=True, soft_time_limit=5, time_limit=10)
def analyze_file(self, file_path):
    try:
        time.sleep(4)
        result = f"Analysis complete for {file_path}"
        print(result)
        return result
    except SoftTimeLimitExceeded:
        print(f"Soft time limit reached for {file_path}, saving partial results")
        return {"file": file_path, "status": "partial", "progress": 0.8}

@app.task(bind=True, soft_time_limit=5, time_limit=10)
def analyze_slow_file(self, file_path):
    time.sleep(15)
    result = f"Analysis complete for {file_path}"
    print(result)
    return result

task1 = analyze_file.delay("doc1.pdf")
task2 = analyze_slow_file.delay("malicious.exe")
print(f"Tasks submitted: {task1.id}, {task2.id}")

Expected output:

Tasks submitted: id1, id2
Analysis complete for doc1.pdf
Soft time limit reached for malicious.exe, saving partial results
[2026-06-28 10:00:10: WARNING] Hard time limit exceeded for analyze_slow_file
[2026-06-28 10:00:10: ERROR] Task killed by hard time limit

Task Expiration with ETA

from celery import Celery
from datetime import datetime, timedelta
import time

app = Celery('expiration', broker='redis://localhost:6379/0')

@app.task
def generate_report(report_id):
    result = f"Report {report_id} generated"
    print(result)
    return result

@app.task
def process_sensor_data(sensor_id, timestamp):
    result = f"Sensor {sensor_id}: data from {timestamp} processed"
    print(result)
    return result

future_time = datetime.utcnow() + timedelta(seconds=5)
stale_time = datetime.utcnow() - timedelta(hours=2)

task1 = generate_report.apply_async(
    args=["DAILY-001"],
    expires=datetime.utcnow() + timedelta(hours=1)
)

task2 = process_sensor_data.apply_async(
    args=["sensor-01", "2026-06-28T08:00:00"],
    expires=stale_time
)

print(f"Task 1 (valid): {task1.id}")
print(f"Task 2 (expired): {task2.id}")

Expected output:

Task 1 (valid): id1
Task 2 (expired): id2
[2026-06-28 10:00:00: WARNING] Task process_sensor_data discarded: expired
Report DAILY-001 generated

Queue-Level Message TTL

from celery import Celery

app = Celery('queue_ttl', broker='redis://localhost:6379/0')

app.conf.task_default_queue = 'default'
app.conf.task_queue_max_priority = 10

app.conf.broker_transport_options = {
    'queue_name_prefix': 'celery_',
    'visibility_timeout': 3600,
    'max_retries': 3,
}

queue_ttl_settings = {
    'x-message-ttl': 300000,
    'x-max-length': 10000,
    'x-max-length-bytes': 1073741824,
}

app.conf.task_queues = {
    'high_priority': {
        'exchange': 'high_priority',
        'routing_key': 'high_priority',
        'queue_arguments': {
            'x-message-ttl': 60000,
            'x-max-priority': 10,
        },
    },
    'low_priority': {
        'exchange': 'low_priority',
        'routing_key': 'low_priority',
        'queue_arguments': {
            'x-message-ttl': 3600000,
            'x-max-priority': 5,
        },
    },
}

print("Queue TTL configured: high=60s, low=3600s")

Expected output:

Queue TTL configured: high=60s, low=3600s

Graceful Shutdown on Time Limit

from celery import Celery
from celery.exceptions import SoftTimeLimitExceeded
import time
import json

app = Celery('graceful', broker='redis://localhost:6379/0')

@app.task(bind=True, soft_time_limit=10, time_limit=15)
def export_large_dataset(self, query_params):
    progress = {"status": "in_progress", "records": 0}
    try:
        for batch in range(20):
            time.sleep(1)
            progress["records"] = (batch + 1) * 100
            self.update_state(state='PROGRESS', meta=progress)
            print(f"Batch {batch + 1}/20 complete ({progress['records']} records)")
    except SoftTimeLimitExceeded:
        print("Soft time limit reached, saving progress")
        self.update_state(
            state='PARTIAL',
            meta={
                "status": "partial",
                "records_exported": progress["records"],
                "next_offset": progress["records"],
            }
        )
        return {"status": "partial", "records": progress["records"]}

task = export_large_dataset.delay({"table": "logs", "date": "2026-06-28"})
print(f"Export task submitted: {task.id}")

Expected output:

Export task submitted: id
Batch 1/20 complete (100 records)
Batch 2/20 complete (200 records)
...
Batch 10/20 complete (1000 records)
Soft time limit reached, saving progress
Task export_large_dataset: state=PARTIAL

Common Mistakes

• Setting soft and hard limits too close -- with soft=10 and hard=11, the graceful handler has only 1 second to clean up. Leave at least 30% headroom between soft and hard limits.
• Forgetting that hard limit kills the process -- hard limit sends SIGKILL to the worker process. The task cannot catch this or clean up. Always use soft limits for cleanup logic.
• Relying on expiration for scheduling -- expiration only discards tasks. For delayed execution, use ETA (estimated time of arrival). Expiration is for discarding, not delaying.
• Expiration vs Time Limit confusion -- expiration discards tasks before they execute (based on age). Time limits terminate tasks during execution (based on run time). They address different problems.
• Not setting queue-level TTL -- without queue TTL, messages accumulate indefinitely if no worker consumes them. Set a reasonable queue TTL as a safety net for abandoned queues.

Practice Questions

1. What is the difference between soft_time_limit and time_limit in Celery?
2. How does task expiration differ from task ETA?
3. What happens to a worker process when a hard time limit is exceeded?
4. Why should you set soft and hard time limits with adequate spacing?
5. How does queue-level message TTL protect against queue growth?

Challenge

Build a task timeout management system that: (1) registers tasks with soft_limit=30s and hard_limit=45s, (2) uses a custom base class that logs when SoftTimeLimitExceeded is raised and saves partial state to Redis, (3) implements an exponential backoff for tasks that hit time limits (retry with countdown*2 up to 3 retries), and (4) alerts to Slack when any task exceeds hard limit. Test with a task that sleeps 60 seconds.

FAQ

Mini Project

Build a task lifecycle manager that: (1) accepts tasks with configurable soft/hard limits and expiration, (2) runs a watchdog that monitors task execution time and logs warnings at 50% of soft limit, (3) implements graceful degradation on soft limit (save state, set retry), (4) sends email notification on hard limit kills with the task context, and (5) provides a dashboard showing time limit hit rates and average task duration per queue.

What's Next

Continue with Task Revocation to learn how to cancel running tasks. Then explore Worker Shutdown for graceful worker lifecycle management.