Introduction to Celery

In this tutorial, you will learn about Introduction to Celery. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery is a distributed task queue for Python that executes background tasks asynchronously, enabling offloading expensive operations from the main application flow.

What You Learn

You will learn what Celery is, how it compares to other task queues, its core components (broker, worker, result backend), and when to use Celery in your applications.

Why It Matters

Web applications must respond to HTTP requests quickly. Tasks like sending emails, processing images, or generating reports take seconds or minutes. Celery moves these tasks to background workers, keeping the HTTP response fast and the user experience smooth.

Real-World Use

Doda Browser uses Celery for malware analysis. When a user uploads a file, the web request returns immediately with a "scanning" status while Celery workers analyze the file in the background. Results are stored and retrieved when ready.

What is Celery?

flowchart LR
    P[Web App] -->|Task| B[Broker
Redis/RabbitMQ]
    B -->|Task| W1[Worker 1]
    B -->|Task| W2[Worker 2]
    B -->|Task| W3[Worker 3]
    W1 -->|Result| R[Result Backend]
    P -->|Check| R
    style B fill:#f90,color:#fff
    style R fill:#6a0,color:#fff

Celery has three core components:

• Broker: Stores tasks (Redis or RabbitMQ)
• Worker: Executes tasks asynchronously
• Result Backend: Stores task results for retrieval

Minimal Celery Application

# tasks.py
from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y

# client.py
from tasks import add

result = add.delay(4, 4)
print(f"Task ID: {result.id}")
print(f"Result: {result.get(timeout=10)}")

Expected output:

Task ID: 550e8400-e29b-41d4-a716-446655440000
Result: 8

Core Components Explained

Broker: Holds tasks until workers consume them. Redis is fast and simple. RabbitMQ is more feature-rich with complex routing. Choose RabbitMQ for production that needs routing flexibility; Redis for simplicity and speed.

Worker: Runs tasks in separate processes. Workers can run on the same machine or across a fleet. Each worker can handle multiple tasks concurrently based on concurrency settings.

Result Backend: Stores task return values. Common backends include Redis, database, and S3. The result backend is optional. Many applications only need the broker and workers.

When to Use Celery

Celery vs Other Task Queues

Common Mistakes

1. Using Celery for Simple Async

If you only need to run one function asynchronously, Python's asyncio or threading may suffice. Celery is for distributed, reliable task execution.

2. Running Workers Without a Broker

Celery requires a running broker. A common mistake is starting workers without Redis or RabbitMQ running. Workers wait forever for tasks.

3. Blocking Tasks

Long-running CPU-bound tasks block the worker Process. Use @app.task(acks_late=True) and increase worker concurrency to handle multiple tasks per worker.

4. Not Setting Task Time Limits

A task that hangs infinitely consumes a worker forever. Always set task_time_limit and task_soft_time_limit in Celery config.

5. Using pickle Serializer in Production

Pickle is convenient but dangerous. Untrusted data can execute arbitrary code. Use JSON or msgpack in production.

Practice Questions

1. What are the three core components of Celery?

Broker (stores tasks), Worker (executes tasks), Result Backend (stores task results). The broker and result backend can be the same service (Redis).

2. What is the difference between delay() and apply_async()?

delay() is a shortcut for apply_async(). apply_async allows additional options like countdown, queue, routing_key, and eta for fine-grained task control.

3. Can Celery run without a result backend?

Yes. The result backend is optional. Use it only when you need to retrieve task return values. Many Celery setups run without a result backend.

4. What brokers does Celery support?

Redis, RabbitMQ, Amazon SQS, and Apache Kafka (experimental). Redis is the most common for simplicity and speed.

Challenge

Design a Celery-based system for a video processing pipeline: upload, transcode, generate thumbnails, analyze content, and notify the user. Map each step to a Celery task with appropriate routing, retry policies, and result backend usage.

FAQ

Mini Project: First Celery App

# tasks.py
from celery import Celery

app = Celery('first_app',
             broker='redis://localhost:6379/0',
             backend='redis://localhost:6379/0')

app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
    timezone='UTC',
    enable_utc=True,
    task_time_limit=300,
    task_soft_time_limit=240,
)

@app.task
def reverse_string(s):
    return s[::-1]

@app.task
def word_count(text):
    return len(text.split())

@app.task
def to_uppercase(text):
    return text.upper()

# run.py
from tasks import reverse_string, word_count, to_uppercase
import time

tasks = [
    reverse_string.delay('hello'),
    word_count.delay('The quick brown fox jumps over the lazy dog'),
    to_uppercase.delay('celery is awesome'),
]

for task in tasks:
    result = task.get(timeout=10)
    print(f"{task.name}: {result}")

print(f"\nAll tasks completed")

Expected output:

tasks.reverse_string: olleh
tasks.word_count: 9
tasks.to_uppercase: CELERY IS AWESOME

All tasks completed

What's Next

Now that you understand what Celery is, move on to installation and setup to get Celery running, then configure broker setup with Redis and RabbitMQ.