Django Celery Chain Callback Fix

In this tutorial, you'll learn about Django Celery Chain Callback Fix. We cover key concepts, practical examples, and best practices.

The Problem

Some workflows require running tasks in sequence — process image, then generate thumbnail, then notify user. Launching them independently misses the dependency chain.

Quick Fix

Wrong — launching tasks independently

def upload_view(request):
    process_image.delay(image_id)       # Task 1
    generate_thumbnail.delay(image_id)   # Task 2 — runs before Task 1 finishes
    notify_user.delay(user_id)           # Task 3 — runs before either finishes

Output: Thumbnail generation may process a non-existent file. Notification fires before anything is ready.

Correct — Celery chain

from celery import chain

def upload_view(request):
    workflow = chain(
        process_image.s(image_id),
        generate_thumbnail.s(),
        notify_user.s(user_id)
    )
    result = workflow()
    return Response({'task_group_id': result.id})

Output: Each task runs only after the previous one succeeds. Thumbnail is generated after processing completes.

Using pipe operator

workflow = process_image.s(image_id) | generate_thumbnail.s() | notify_user.s(user_id)

Passing results between tasks

@app.task
def process_image(image_id):
    image = Image.objects.get(id=image_id)
    image.process()
    return {'image_id': image_id, 'format': 'webp'}

@app.task
def generate_thumbnail(result):
    # result is the return value of process_image
    thumbnail = create_thumbnail(result['image_id'])
    return {**result, 'thumbnail_id': thumbnail.id}

@app.task
def notify_user(result, user_id):
    send_notification(user_id, f"Image {result['image_id']} ready")

Callbacks with error handling

workflow = (
    process_image.s(image_id) |
    generate_thumbnail.s() |
    notify_user.s(user_id)
) | cleanup_on_failure.s()  # Runs even if previous fails? No — use chord for that

Callbacks with link_error

process_image.apply_async(
    args=(image_id,),
    link=generate_thumbnail.s(),
    link_error=handle_error.s()
)

Prevention

• Use chain() or the pipe | operator for sequential task execution.
• Use group() for parallel tasks, chord() for parallel + callback.
• Use link_error to handle failures in callback tasks.

Common Mistakes with celery chain callback

1. Non-exhaustive pattern matches that compile with warnings then crash at runtime
2. Misunderstanding that String is [Char] with poor performance for large text operations
3. Using foldl instead of foldl' causing stack overflow on large lists

These mistakes appear frequently in real-world DJANGO code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ