Celery Soft Time Limits: Graceful Task Timeout Handling and Partial Results

In this tutorial, you will learn about Celery Soft Time Limits: Graceful Task Timeout Handling and Partial Results. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery soft time limits raise a catchable SoftTimeLimitExceeded exception that lets tasks save partial results, clean up resources, and exit gracefully before the hard time limit forcibly terminates the worker Process.

flowchart LR
    Start[Task Starts] --> Timer{Time < Soft Limit?}
    Timer -->|Yes| Work[Execute Work]
    Work -->|Checkpoint| Save[Save Progress]
    Save --> Timer
    Timer -->|No| Exc[SoftTimeLimitExceeded]
    Exc --> Catch[Catch Exception]
    Catch --> Cleanup[Release Resources]
    Cleanup --> Partial[Return Partial Result]
    Partial -->|Hard Limit| Kill[Process Killed]

What You'll Learn

• Soft time limit configuration and mechanics
• Catching SoftTimeLimitExceeded gracefully
• Saving partial progress before timeout
• Combining soft and hard limits
• Resource cleanup on timeout

Why It Matters

Hard time limits kill tasks without warning, losing all work. Soft time limits give tasks a chance to checkpoint progress, release database connections, and return partial results -- saving hours of recomputation for long-running operations.

Real-World Use

DodaTech's video transcoding platform uses soft time limits to save partially transcoded segments when a task approaches its deadline. The next retry resumes from the saved progress instead of starting over, reducing average transcoding time by 60%.

Catching Soft Time Limit

from celery import Celery
from celery.exceptions import SoftTimeLimitExceeded
import time

app = Celery('softlimits', broker='redis://localhost:6379/0')

@app.task(bind=True, soft_time_limit=10, time_limit=15)
def process_large_file(self, file_id):
    processed_chunks = 0
    try:
        for chunk in range(50):
            time.sleep(0.3)
            processed_chunks += 1
            progress = int((processed_chunks / 50) * 100)
            self.update_state(
                state='PROGRESS',
                meta={'current': processed_chunks, 'total': 50, 'pct': progress}
            )
            print(f"Chunk {processed_chunks}/50 processed")

    except SoftTimeLimitExceeded:
        print(f"\nSoft limit reached after {processed_chunks} chunks")
        self.update_state(
            state='PARTIAL',
            meta={
                'file_id': file_id,
                'processed_chunks': processed_chunks,
                'total_chunks': 50,
                'progress_pct': int((processed_chunks / 50) * 100),
            }
        )
        return {
            'status': 'partial',
            'file_id': file_id,
            'processed_chunks': processed_chunks,
        }

task = process_large_file.delay("VID-042")
print(f"Processing file {task.id}")

Expected output:

Processing file id
Chunk 1/50 processed
Chunk 2/50 processed
...
Chunk 33/50 processed
Soft limit reached after 33 chunks

Resource Cleanup on Timeout

from celery import Celery
from celery.exceptions import SoftTimeLimitExceeded
import time

app = Celery('cleanup', broker='redis://localhost:6379/0')

class DatabaseConnection:
    def __init__(self, name):
        self.name = name
        self.connected = True
        print(f"Connected to {name}")

    def close(self):
        if self.connected:
            self.connected = False
            print(f"Closed connection to {self.name}")

    def query(self, sql):
        time.sleep(2)
        return f"Result from {self.name}: {sql}"

@app.task(bind=True, soft_time_limit=5, time_limit=8)
def export_data(self, source_name):
    conn = DatabaseConnection(source_name)
    try:
        result = conn.query("SELECT * FROM large_table")
        print(f"Query result: {result}")
        return {"source": source_name, "data": result}
    except SoftTimeLimitExceeded:
        print(f"Timeout exporting from {source_name}, cleaning up...")
        return {"source": source_name, "status": "timeout"}
    finally:
        conn.close()

task = export_data.delay("production_db")
print(f"Export task: {task.id}")

Expected output:

Export task: id
Connected to production_db
Timeout exporting from production_db, cleaning up...
Closed connection to production_db

Partial Results with Retry

from celery import Celery
from celery.exceptions import SoftTimeLimitExceeded
import time
import json

app = Celery('partial', broker='redis://localhost:6379/0')
cache = {}

@app.task(bind=True, soft_time_limit=15, time_limit=20, max_retries=2)
def process_batch(self, batch_id, total_items):
    completed = cache.get(f'batch:{batch_id}', {}).get('completed', 0)

    try:
        for item in range(completed, total_items):
            time.sleep(1)
            completed = item + 1
            cache[batch_id] = {'completed': completed, 'total': total_items}
            print(f"Item {completed}/{total_items} complete")

        cache.pop(batch_id, None)
        print(f"Batch {batch_id} fully completed")
        return {"batch_id": batch_id, "status": "complete", "items": total_items}

    except SoftTimeLimitExceeded:
        print(f"Batch {batch_id} interrupted at {completed}/{total_items}")
        remaining = total_items - completed
        countdown = min(remaining * 2, 60)

        self.retry(countdown=countdown, max_retries=2)
        return {"batch_id": batch_id, "status": "retry", "completed": completed}

task = process_batch.delay("BATCH-001", 25)
print(f"Batch task: {task.id}")

Expected output:

Batch task: id
Item 1/25 complete
Item 2/25 complete
...
Item 15/25 complete
Batch BATCH-001 interrupted at 15/25
Item 16/25 complete
...
Item 25/25 complete
Batch BATCH-001 fully completed

Common Mistakes

• Not allowing enough time between soft and hard limits -- with soft=10s and hard=11s, the cleanup code has only 1 second. Set hard at least 30-50% higher than soft (e.g., soft=10, hard=15).
• Catching SoftTimeLimitExceeded but re-raising -- if you catch and re-raise, the hard limit still kills the process. Either handle completely or let the exception propagate naturally.
• Not saving progress in a persistent store -- partial progress saved in memory vanishes if the worker restarts. Use Redis, database, or filesystem for durable progress tracking.
• Ignoring SoftTimeLimitExceeded in finally blocks -- cleanup code in finally blocks runs after the exception. Ensure cleanup handles the interrupted state correctly (e.g., partial transactions).
• Setting soft limit too low -- tasks that legitimately run longer trigger false timeouts. Profile actual execution times and set limits at p99 + 50% buffer.

Practice Questions

1. Why should you always set both soft and hard time limits together?
2. How do you persist partial progress so it survives worker restarts?
3. What happens if cleanup code in a finally block also hits the hard time limit?
4. How does soft time limit interact with task retries?
5. Can you set different soft limits for different tasks in the same worker?

Challenge

Build a resume-able ETL pipeline handler: (1) define a base task class that saves progress to Redis after every 10 records, (2) on SoftTimeLimitExceeded, save checkpoint details and retry with countdown=30, (3) on retry, read checkpoint from Redis and resume from the last completed record, (4) implement a hard limit safety net at 2x the soft limit, and (5) handle the case where retry count is exhausted (final partial report).

FAQ

Mini Project

Build a resilient batch processing framework that: (1) processes items from a Redis list with configurable batch size, (2) saves progress to Redis after every item (list index + result), (3) catches SoftTimeLimitExceeded to save checkpoint and retry from the checkpoint, (4) implements circuit breaker: if 3 consecutive timeouts occur on the same batch, skip the problematic item, (5) generates a completion report showing items processed per batch, timeouts, and skips.

What's Next

Continue with Task Revocation to learn how to cancel running tasks from external systems. Then explore Worker Shutdown for controlling worker lifecycle during deployments.