Broker Setup for Celery — Complete Guide

In this tutorial, you will learn about Broker Setup for Celery. We cover key concepts, practical examples, and best practices to help you master this topic.

Configure Redis and RabbitMQ as Celery brokers, understand their trade-offs, set up connection pooling, and handle broker failover in production.

What You Learn

You will learn how to configure Redis and RabbitMQ as Celery brokers, choose between them based on your needs, set up connection pooling, and handle broker connection failures.

Why It Matters

The broker is the backbone of Celery. It holds all pending tasks. If the broker goes down, no tasks can be submitted or executed. Choosing the right broker and configuring it correctly is critical for reliability.

Real-World Use

DodaTech uses RabbitMQ as the Celery broker in production for its routing flexibility and delivery guarantees. Development environments use Redis for simplicity and speed. The broker configuration is environment-aware.

Broker Comparison

flowchart TB
    subgraph "Broker Choice"
        direction LR
        R[Redis] --- RMQ[RabbitMQ]
    end
    R --> |Pros| RP[Fast, simple,
good visibility]
    R --> |Cons| RC[No routing,
limited persistence]
    RMQ --> |Pros| RMP[Advanced routing,
delivery guarantees]
    RMQ --> |Cons| RMC[More complex,
heavier setup]
    style R fill:#f90,color:#fff
    style RMQ fill:#6a0,color:#fff

Redis Broker Configuration

# celery_app.py
from celery import Celery

# Simple Redis configuration
app = Celery('redis_app',
             broker='redis://localhost:6379/0',
             backend='redis://localhost:6379/1')

# Redis with password
app2 = Celery('secure_app',
              broker='redis://:password@localhost:6379/0')

# Redis over TLS
app3 = Celery('tls_app',
              broker='rediss://:password@redis.example.com:6379/0')

# Redis with connection pool settings
app.conf.update(
    broker_pool_limit=10,
    broker_connection_timeout=30,
    broker_connection_retry=True,
    broker_connection_retry_on_startup=True,
    broker_connection_max_retries=10,
)

@app.task
def sample_task(data):
    return f"Processed: {data}"

RabbitMQ Broker Configuration

# RabbitMQ broker setup
from celery import Celery

# Simple RabbitMQ
app = Celery('rabbit_app',
             broker='amqp://guest:guest@localhost:5672//',
             backend='rpc://')

# RabbitMQ with virtual host
app2 = Celery('multi_tenant',
              broker='amqp://user:pass@localhost:5672/production_vhost')

# RabbitMQ cluster
app3 = Celery('cluster_app',
              broker='amqp://user:pass@node1:5672,node2:5672,node3:5672/production_vhost')

# RabbitMQ over TLS
app4 = Celery('tls_app',
              broker='amqps://user:pass@rabbitmq.example.com:5671/production_vhost')

Connection Pooling

from celery import Celery

app = Celery('pool_app', broker='redis://localhost:6379/0')

# Configure connection pool
app.conf.update(
    broker_pool_limit=20,
    broker_connection_timeout=30,
    broker_connection_retry=True,
    broker_connection_max_retries=0,  # infinite retries
)

@app.task(bind=True, max_retries=3)
def task_with_retry(self, data):
    try:
        return f"Result: {data}"
    except Exception as exc:
        raise self.retry(exc=exc, countdown=60)

Broker URL Formats

Sentinel Configuration for Redis

For high-availability Redis, use Sentinel:

from celery import Celery

app = Celery('sentinel_app')

# Redis Sentinel configuration
app.conf.update(
    broker_url='redis-sentinel://sentinel1:26379,sentinel2:26379,sentinel3:26379/0',
    broker_transport_options={
        'master_name': 'mymaster',
        'sentinels': [
            ('sentinel1', 26379),
            ('sentinel2', 26379),
            ('sentinel3', 26379),
        ],
    }
)

Broker Visibility Timeout

Redis broker has a visibility timeout for unacknowledged tasks:

app.conf.update(
    broker_transport_options={
        'visibility_timeout': 3600,  # 1 hour
    }
)

Tasks that are not acknowledged within the visibility timeout are redelivered. Set this to match your longest expected task runtime.

Common Mistakes

1. Using Same Redis Database for Broker and Result Backend

Use different database numbers (e.g., broker=db0, result=db1). Mixing them causes key collisions and unpredictable behavior.

2. Not Setting broker_connection_retry_on_startup

Without this setting, Celery may fail to start if the broker is temporarily unavailable. Always enable it for production.

3. Ignoring Broker Connection Timeouts

Default timeouts may be too short for slow networks. Set broker_connection_timeout to 30-60 seconds for production environments.

4. Using Redis Without Persistence

Redis stores everything in memory by default. Configure Redis persistence (AOF or RDB) to avoid losing tasks on restart.

5. Not Monitoring Broker Memory

Redis stores all unprocessed tasks in memory. Monitor memory usage and set a maxmemory policy to prevent OOM crashes.

Practice Questions

1. What is the main advantage of RabbitMQ over Redis as a Celery broker?

RabbitMQ provides advanced routing (exchanges, bindings), delivery guarantees, and supports complex task routing patterns. Redis is simpler and faster but lacks routing features.

2. How do you configure a Redis broker with a password?

Use the URL format redis://:password@host:6379/0. The colon before the password is required.

3. What is broker_connection_retry_on_startup used for?

It tells Celery to retry connecting to the broker on startup instead of failing immediately. Essential for environments where the broker may start after the worker.

4. What is the visibility timeout in Redis broker?

The time a task remains invisible to other workers after being fetched. If the worker does not ack within this time, the task is redelivered.

Challenge

Design a broker configuration for a multi-region Celery deployment. Workers in US, EU, and Asia each have a local Redis broker. Tasks are routed based on region. Configure connection pooling, timeouts, and retries for each region.

FAQ

Mini Project: Dual Broker Setup

import os
from celery import Celery

env = os.environ.get('APP_ENV', 'development')

if env == 'production':
    broker_url = 'amqp://user:pass@rabbitmq:5672/production_vhost'
    backend_url = 'redis://redis:6379/1'
elif env == 'staging':
    broker_url = 'amqp://user:pass@rabbitmq-staging:5672/staging_vhost'
    backend_url = 'redis://redis-staging:6379/1'
else:
    broker_url = 'redis://localhost:6379/0'
    backend_url = 'redis://localhost:6379/1'

app = Celery('multi_env', broker=broker_url, backend=backend_url)

app.conf.update(
    task_serializer='json',
    accept_content=['json'],
    result_serializer='json',
    timezone='UTC',
    enable_utc=True,
    broker_connection_retry_on_startup=True,
    broker_connection_max_retries=10,
    task_track_started=True,
)

@app.task
def process_data(data_id):
    return f"Processed {data_id} in {env} environment"

# Run with different environments
APP_ENV=development celery -A multi_env_app worker --loglevel=info
APP_ENV=production celery -A multi_env_app worker --loglevel=info

Expected output:

[2026-06-28 10:00:00: INFO/MainProcess] Connected to redis://localhost:6379/0
[2026-06-28 10:00:00: INFO/MainProcess] celery@hostname ready.

What's Next

Now that your broker is configured, learn about defining tasks with different options, then explore running the Celery worker with various settings.