Celery Broker High Availability: Resilient Redis and RabbitMQ Configurations

In this tutorial, you will learn about Celery Broker High Availability: Resilient Redis and RabbitMQ Configurations. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery broker high availability ensures task processing continues through broker failures using Redis Sentinel for automatic failover, RabbitMQ mirrored queues, multi-broker URL fallback, and connection retry configurations that survive network partitions and broker outages.

flowchart TD
    W[Workers] --> MC{Master Broker}
    MC -->|Primary| R1[Redis Primary]
    MC -->|Failover| R2[Redis Replica
Promoted]
    R1 -->|Replication| R2
    R1 -->|Monitor| S1[Sentinel 1]
    R1 -->|Monitor| S2[Sentinel 2]
    R1 -->|Monitor| S3[Sentinel 3]
    S1 -->|Failover Decision| R2
    R2 -->|New Primary| W

What You'll Learn

• Redis Sentinel configuration for Celery
• RabbitMQ mirror queue setup
• Broker connection retry and fallback
• Multi-broker URL configuration
• Monitoring broker health

Why It Matters

The broker is the single most critical component in Celery. If the broker goes down, no tasks can be submitted or processed. High availability configuration ensures Celery continues working through broker maintenance, failures, and network issues.

Real-World Use

DodaTech runs Redis Sentinel across 3 availability zones. When a Redis primary fails, automatic failover completes in under 10 seconds. Celery workers reconnect to the new primary with zero task loss. Before HA, a single Redis outage stopped all task processing for 30+ minutes.

Redis Sentinel Configuration

from celery import Celery

app = Celery('ha', broker='redis-sentinel://sentinel1:26379,sentinel2:26379,sentinel3:26379/0')

app.conf.broker_transport_options = {
    'master_name': 'mymaster',
    'sentinels': [
        ('sentinel1', 26379),
        ('sentinel2', 26379),
        ('sentinel3', 26379),
    ],
    'socket_timeout': 0.5,
    'retry_on_timeout': True,
}

app.conf.broker_connection_retry = True
app.conf.broker_connection_retry_on_startup = True
app.conf.broker_connection_max_retries = 10

@app.task
def ha_task(item_id):
    result = f"HA task {item_id} processed"
    print(result)
    return result

task = ha_task.delay(42)
print(f"Task submitted via Sentinel-managed Redis: {task.id}")

Start worker:

celery -A ha worker --loglevel=info

Expected output during failover:

[2026-06-28 10:00:00: INFO] Connected to redis-sentinel://...
[2026-06-28 10:00:30: WARNING] Connection to broker lost, reconnecting...
[2026-06-28 10:00:31: INFO] Connected to new master redis-sentinel://...
Task HA task 42 processed

Multi-Broker Fallback

from celery import Celery

app = Celery('ha', broker=[
    'redis://redis-primary:6379/0',
    'redis://redis-secondary:6379/0',
    'redis://redis-backup:6379/0',
])

app.conf.broker_failover_strategy = 'round-robin'

app.conf.broker_transport_options = {
    'max_retries': 3,
    'interval_start': 0,
    'interval_step': 0.2,
    'interval_max': 2,
}

@app.task
def fallback_task(data_id):
    result = f"Data {data_id} processed (broker: {app.conf.broker_url})"
    print(result)
    return result

for i in range(5):
    fallback_task.delay(i)
print("Tasks submitted with broker failover")

Expected output:

Tasks submitted with broker failover
Data 0 processed (broker: redis://redis-primary:6379/0)
Data 1 processed (broker: redis://redis-primary:6379/0)
[2026-06-28 10:00:05: WARNING] Primary broker unavailable, switching to secondary
Data 2 processed (broker: redis://redis-secondary:6379/0)

Broker Health Monitoring

from celery import Celery
from celery.signals import worker_connected, worker_disconnected
import time

app = Celery('ha', broker='redis://localhost:6379/0')

broker_health = {
    'connected': False,
    'last_connected': None,
    'last_disconnected': None,
    'reconnect_count': 0,
}

@worker_connected.connect
def on_connect(sender=None, **kwargs):
    broker_health['connected'] = True
    broker_health['last_connected'] = time.time()
    print(f"[BROKER] Connected to broker")

@worker_disconnected.connect
def on_disconnect(sender=None, **kwargs):
    broker_health['connected'] = False
    broker_health['last_disconnected'] = time.time()
    broker_health['reconnect_count'] += 1
    print(f"[BROKER] Disconnected from broker (reconnects: {broker_health['reconnect_count']})")

@app.task
def health_check_task():
    latency = measure_broker_latency()
    report = {
        'broker_connected': broker_health['connected'],
        'uptime_seconds': time.time() - (broker_health['last_connected'] or time.time()),
        'reconnect_count': broker_health['reconnect_count'],
        'latency_ms': latency,
    }
    print(f"Broker health: {report}")
    return report

def measure_broker_latency():
    import redis
    try:
        r = redis.Redis(host='localhost', port=6379, db=0, socket_timeout=2)
        start = time.time()
        r.ping()
        return (time.time() - start) * 1000
    except:
        return -1

report = health_check_task.delay()
print(f"Health check: {report.id}")

Expected output:

[BROKER] Connected to broker
Broker health: {'broker_connected': True, 'uptime_seconds': 100.5, 'reconnect_count': 0, 'latency_ms': 1.2}
Health check: task-id

Common Mistakes

• Single broker URL in production -- a single Redis instance is a single point of failure. Use Sentinel, Cluster, or multi-broker configurations for production. Celery stops processing when the single broker fails.
• No broker_connection_retry_on_startup -- without this setting, if the broker is unavailable when the worker starts, the worker fails immediately. Enable it so the worker waits for the broker to become available.
• Not testing failover scenarios -- Sentinels may detect failures but promote replicas that are behind on replication. Run chaos engineering exercises: kill the primary Redis and verify Celery workers reconnect and resume processing.
• Incorrect Sentinel configuration -- workers must connect via Sentinel, not directly to Redis replicas. Direct connections bypass Sentinel and do not redirect during failover. Always use the redis-sentinel:// URL scheme.
• Ignoring broker resource monitoring -- the broker needs CPU, memory, and network monitoring. A broker running at 90% memory can fail during failover. Monitor broker resources and alert before saturation.

Practice Questions

1. How does Redis Sentinel provide high availability for Celery?
2. How do you configure Celery to use multiple brokers for failover?
3. What happens to in-flight tasks during a broker failover?
4. How do you monitor broker health from Celery workers?
5. Why is broker_connection_retry_on_startup important?

Challenge

Build a broker HA test harness: (1) deploy a 3-node Redis Sentinel cluster with 1 primary and 2 replicas, (2) configure Celery workers to connect via Sentinel, (3) submit 1000 tasks and kill the Redis primary mid-processing, (4) measure failover time (last successful task to first task on new primary), (5) verify zero task loss by comparing submitted vs completed task counts, (6) test Network Partition scenarios (disconnect Sentinel from primary, partition worker network), and (7) generate a report showing recovery time, task replay rate, and worker reconnection behavior.

FAQ

Mini Project

Build a resilient Celery HA deployment: (1) 3-node Redis Sentinel cluster with 1 primary, 2 replicas, and 3 sentinels across availability zones, (2) Celery workers configured with redis-sentinel:// broker URL, (3) automatic broker health check that exposes Sentinel status as Prometheus metrics, (4) chaos-monkey script that randomly kills the primary Redis and validates recovery, (5) Grafana dashboard showing failover events, reconnection times, and queue depth during failover, (6) alerting rule that fires if failover takes longer than 30 seconds.

What's Next

Continue with Multi-Datacenter Deployment to learn geo-distributed Celery configurations. Then explore Security Configuration for securing Celery in production.