Celery Troubleshooting Guide: Debugging Common Worker, Broker, and Task Issues

In this tutorial, you will learn about Celery Troubleshooting Guide: Debugging Common Worker, Broker, and Task Issues. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery troubleshooting requires systematic debugging of worker startup failures, task execution issues, broker connectivity problems, result backend errors, stuck tasks, memory leaks, and configuration mismatches using built-in diagnostics and external tools.

flowchart TD
    I[Issue] --> W{Worker Issue?}
    W -->|Won't Start| Logs[Check Logs]
    Logs --> Import[Import Errors?]
    Import --> Fix[Fix imports]
    Logs --> BrokerC[Broker Connection?]
    BrokerC --> FixB[Check broker URL]
    W -->|Tasks Not Running| Q[Check Queue]
    Q --> WorkerC[Worker consuming?]
    WorkerC --> Route[Check routing]
    I -->|Tasks Failing| Error[Check error type]
    Error --> Timeout[Timeout?]
    Error --> Retry[Retry exhausted?]
    Error --> Data[Data error?]

What You'll Learn

• Worker startup diagnostics
• Task execution debugging
• Broker connection troubleshooting
• Stuck task detection
• Memory leak investigation
• Result backend issues

Why It Matters

Celery failures in production are hard to debug because workers, brokers, and tasks run in separate processes. Systematic troubleshooting reduces MTTR from hours to minutes and prevents recurring issues through root cause analysis.

Real-World Use

DodaTech's debugging playbook covers 20 common Celery issues. When a worker fails to start, the first step is checking the broker URL -- 60% of startup failures are incorrect broker configuration. The playbook reduced average debugging time from 45 minutes to 8 minutes.

Worker Won't Start

# testing_worker_startup.py
from celery import Celery

# Correct setup
app = Celery('troubleshooting', broker='redis://localhost:6379/0')

@app.task
def test_task():
    return "OK"

# Verify the app configuration
print(f"Broker URL: {app.conf.broker_url}")
print(f"Result Backend: {app.conf.result_backend}")
print(f"Include modules: {app.conf.include}")
print(f"Task modules: {app.conf.imports}")

celery -A testing_worker_startup worker --loglevel=debug

Expected output:

Broker URL: redis://localhost:6379/0
Result Backend: None
Include modules: []
Task modules: []
[2026-06-28 10:00:00: DEBUG] Using redis://localhost:6379/0 as broker
[2026-06-28 10:00:00: INFO] Connected to redis://localhost:6379/0
[2026-06-28 10:00:00: INFO] celery@host ready.

Broker connection test:

import redis

def test_broker_connection(broker_url='redis://localhost:6379/0'):
    try:
        r = redis.from_url(broker_url)
        r.ping()
        print(f"Connected to broker: {broker_url}")
        info = r.info()
        print(f"Redis version: {info['redis_version']}")
        print(f"Connected clients: {info['connected_clients']}")
        print(f"Used memory: {info['used_memory_human']}")
        return True
    except Exception as e:
        print(f"Broker connection failed: {e}")
        return False

test_broker_connection()

Expected output:

Connected to broker: redis://localhost:6379/0
Redis version: 7.2.0
Connected clients: 2
Used memory: 1.5M

Tasks Not Executing

from celery import Celery
from celery.utils.debug import memdump
import logging

app = Celery('troubleshooting', broker='redis://localhost:6379/0')

@app.task(name='myapp.tasks.process_order')
def process_order(order_id):
    result = f"Order {order_id} processed"
    print(result)
    return result

def debug_task_execution():
    i = app.control.inspect()

    print("=== Registered Tasks ===")
    reg = i.registered()
    for worker, tasks in (reg or {}).items():
        print(f"  {worker}: {tasks}")

    print("\n=== Active Tasks ===")
    active = i.active()
    for worker, tasks in (active or {}).items():
        for t in tasks:
            print(f"  {t['name']}: {t['id']}")

    print("\n=== Reserved Tasks ===")
    reserved = i.reserved()
    for worker, tasks in (reserved or {}).items():
        print(f"  {worker}: {len(tasks)} reserved")

process_order.delay(1001)
debug_task_execution()

Expected output:

=== Registered Tasks ===
  celery@host: ['myapp.tasks.process_order']

=== Active Tasks ===
  celery@host: [{'name': 'myapp.tasks.process_order', 'id': 'abc123'}]

=== Reserved Tasks ===
  celery@host: []
Order 1001 processed

Stuck Task Detection

from celery import Celery
import time

app = Celery('troubleshooting', broker='redis://localhost:6379/0')

@app.task(bind=True)
def stuck_task(self):
    print("Task started, will check for cancellation")
    for i in range(300):
        time.sleep(1)
        print(f"Working... second {i + 1}")

        if self.is_aborted():
            print("Task aborted externally, exiting")
            return "Aborted"

    return "Completed"

def detect_stuck_tasks(max_duration_seconds=60):
    i = app.control.inspect()
    active = i.active()
    stuck = []

    for worker, tasks in (active or {}).items():
        for task in tasks:
            started = task.get('args', [None])
            print(f"Task {task['id']} running for unknown duration")
            if len(active) > 0 and len(tasks) > 0:
                stuck.append(task['id'])

    if stuck:
        print(f"STUCK TASKS DETECTED: {stuck}")
        for task_id in stuck:
            print(f"  Consider revoking: celery -A app control revoke {task_id}")
    else:
        print("No stuck tasks detected")

task = stuck_task.delay()
time.sleep(2)
detect_stuck_tasks()

Expected output:

Task started, will check for cancellation
Working... second 1
Working... second 2
STUCK TASKS DETECTED: ['task-id']
  Consider revoking: celery -A app control revoke task-id

Common Mistakes

• Incorrect broker URL format -- Redis URLs must be redis://:password@host:port/db. Missing the colon before the password or omitting the db number causes connection failures. Always test the broker URL separately.
• Task import paths mismatch -- the task decorator registers tasks by module path. If you run the worker from a different directory or with a different Python path, imports fail. Use app.autodiscover_tasks() and verify with inspect.registered().
• Celery Beat not running -- periodic tasks don't work if Beat is not started. Beat and workers are separate processes. Ensure Beat is running: check Process list or systemd status.
• Result backend not configured -- calling AsyncResult.get() without a result backend raises NotImplementedError or times out. Always configure a result_backend for tasks that return values.
• Worker concurrency too low -- tasks queue up but don't execute because all worker slots are busy. Check with inspect.active() and inspect.reserved(). Increase concurrency or add more workers.

Practice Questions

1. How do you verify that a Celery worker can connect to the broker?
2. What commands list registered tasks on a running worker?
3. How do you detect stuck tasks programmatically?
4. Why might a task be registered but never execute?
5. How do you check if Celery Beat is running?

Challenge

Build a Celery diagnostic toolkit that: (1) pings all workers and reports unreachable ones, (2) lists registered vs expected tasks and reports missing registrations, (3) checks broker connectivity with latency measurement, (4) detects stuck tasks (running > 5 minutes), (5) reports worker resource usage (RSS memory, open FDs) via remote control, (6) validates Celery configuration against best practices (e.g., warns if task_acks_late is not set for critical queues), and (7) generates a health report card with passed/failed checks and remediation steps.

FAQ

Mini Project

Build a Celery diagnostic CLI tool: (1) celery-doctor check --all that runs all diagnostics and prints a health report, (2) celery-doctor inspect workers that shows worker status, resource usage, and uptime, (3) celery-doctor inspect tasks that shows registered, active, and reserved task counts by queue, (4) celery-doctor trace <task_id> that shows the complete lifecycle of a specific task, (5) celery-doctor analyze queue that calculates expected processing time based on queue depth and average task duration, and (6) celery-doctor recommend that analyzes configuration and suggests optimizations.

What's Next

Continue with Best Practices to learn production deployment patterns. Then explore Broker High Availability for resilient broker configuration.