Celery Canvas: Complex Workflows with Chains, Groups, Chords, and Maps

In this tutorial, you will learn about Celery Canvas: Complex Workflows with Chains, Groups, Chords, and Maps. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery canvas provides workflow composition primitives — chains, groups, chords, and maps — that let you combine individual tasks into complex execution graphs with sequential, parallel, and callback patterns for distributed processing.

flowchart TD
    Chain[Chain: A >> B >> C] --> A[Task A] --> B[Task B] --> C[Task C]
    Group[Group: A | B | C] --> A2[Task A]
    Group --> B2[Task B]
    Group --> C2[Task C]
    Chord[Chord: Group + Callback] --> G[Group]
    G --> Callback[Callback Task]
    Map[Map: Apply to List] --> M1[Item 1]
    Map --> M2[Item 2]
    Map --> M3[Item 3]

What You'll Learn

• Chain: sequential task execution with result passing
• Group: parallel task execution
• Chord: group with a callback on completion
• Map and Starmap: distributed mapping

Why It Matters

Without canvas, you must manually coordinate multi-step workflows using task callbacks, polling, or external orchestrators. Canvas primitives let you define complex workflows declaratively, with Celery handling all coordination, error propagation, and result collection.

Real-World Use

DodaZIP's file processing pipeline uses a chord: a group of resize tasks (one per image in an album) runs in parallel, and when all complete, a callback generates a thumbnail index. Without chords, the application would need a database poller to detect when all resizes finished.

Chain: Sequential Execution

Pass results from one task to the next:

from celery import Celery, chain

app = Celery('canvas', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    result = x + y
    print(f"add({x}, {y}) = {result}")
    return result

@app.task
def multiply(x, y):
    result = x * y
    print(f"multiply({x}, {y}) = {result}")
    return result

@app.task
def format_result(value):
    result = f"Final result: {value}"
    print(result)
    return result

workflow = chain(
    add.s(10, 20),
    multiply.s(2),
    format_result.s()
)

result = workflow()
print(f"Chain result: {result}")

Expected output:

add(10, 20) = 30
multiply(30, 2) = 60
Final result: 60
Chain result: Final result: 60

Group: Parallel Execution

Run multiple tasks in parallel:

from celery import Celery, group
import time

app = Celery('canvas', broker='redis://localhost:6379/0')

@app.task
def process_file(file_path):
    time.sleep(0.1)
    result = f"Processed: {file_path}"
    print(result)
    return result

files = ["doc1.pdf", "doc2.pdf", "doc3.pdf", "doc4.pdf"]

parallel_group = group(
    process_file.s(f) for f in files
)

start = time.time()
result = parallel_group()
elapsed = time.time() - start

print(f"\nAll files processed in {elapsed:.2f}s")
print(f"Results: {result}")

Expected output:

Processed: doc1.pdf
Processed: doc2.pdf
Processed: doc3.pdf
Processed: doc4.pdf

All files processed in 0.12s
Results: [Result, Result, Result, Result]

Chord: Group with Callback

Execute a callback when all group tasks complete:

from celery import Celery, chord

app = Celery('canvas', broker='redis://localhost:6379/0')

@app.task
def scrape_page(url):
    content = f"Content from {url}"
    print(f"Scraped: {url}")
    return content

@app.task
def aggregate_results(contents):
    total_length = sum(len(c) for c in contents)
    result = f"Aggregated {len(contents)} pages, total {total_length} chars"
    print(result)
    return result

urls = [
    "https://example.com/page1",
    "https://example.com/page2",
    "https://example.com/page3",
]

workflow = chord(
    group(scrape_page.s(url) for url in urls),
    aggregate_results.s()
)

result = workflow()
print(f"Chord result: {result}")

Expected output:

Scraped: https://example.com/page1
Scraped: https://example.com/page2
Scraped: https://example.com/page3
Aggregated 3 pages, total 129 chars
Chord result: Aggregated 3 pages, total 129 chars

Common Mistakes

• Using chains when tasks are independent — chains force sequential execution. If tasks don't depend on each other, use a group for parallel execution.
• Forgetting that chord callbacks receive a list of results — the callback gets an ordered list of each group task's return value. Design the callback signature to accept a list.
• Mixing chains and groups without understanding the result type — chains return a single value (last task). Groups return a list of values. When combining, ensure your functions expect the right input type.
• Not handling errors in groups — if one task in a group fails, the chord callback still fires with partial results unless you set link_error for error callbacks.
• Overusing chains for long workflows — a chain of 20 tasks ties up a worker for the duration. Consider breaking long chains into smaller workflows with intermediate persistence.

Practice Questions

1. What is the difference between a chain and a group in Celery canvas?
2. When would you use a chord instead of manually collecting group results?
3. How does a chain pass results between tasks?
4. What happens when a task in a group fails before a chord callback?
5. How do you handle errors in canvas workflows?

Challenge

Build a document processing workflow using Celery canvas. Workflow: (1) download a document (parallel for 5 docs), (2) convert each to text (parallel, each depends on its download), (3) analyze text for sentiment (parallel), (4) aggregate all sentiments into a report (single callback after all analyses complete). Use chains within a group with a chord callback.

FAQ

Mini Project

Build a data pipeline using Celery canvas that: (1) fetches data from 3 APIs in parallel (group), (2) transforms each API response (chain: fetch >> transform), (3) merges all transformed data into a single report (chord callback), and (4) saves the report to disk. Include error handling: if one API fails, the callback receives partial data and marks the report as partial.

What's Next

Continue with Celery Subtasks and Signatures to learn about task signatures and partial arguments. Then explore Complex Workflow Patterns for advanced workflow Orchestration.