Celery Supervisor: Process Management for Celery Workers with Supervisor

In this tutorial, you will learn about Celery Supervisor: Process Management for Celery Workers with Supervisor. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery Supervisor configuration manages worker processes with automatic restart on failure, multiple worker groups, centralized logging, event-driven process control, and graceful rolling restarts for production Celery deployments.

flowchart LR
    S[Supervisord] --> P1[Celery Worker 1
High Priority]
    S --> P2[Celery Worker 2
Default Queue]
    S --> P3[Celery Worker 3
Bulk Processing]
    S --> P4[Celery Beat]
    S -->|Auto-Restart| P1
    S -->|Auto-Restart| P2
    S -->|Auto-Restart| P3
    S -->|Logs| L[Log Files]
    P1 -->|stderr| L
    P2 -->|stderr| L

What You'll Learn

• Supervisor configuration for Celery workers
• Multiple worker programs per host
• Auto-restart on crash
• Log management and rotation
• Graceful restart without downtime

Why It Matters

Celery workers crash due to memory issues, unhandled exceptions, and resource exhaustion. Without a process manager, crashes go unnoticed until tasks start failing. Supervisor ensures workers stay running and logs are captured for debugging.

Real-World Use

DodaTech runs 4 Celery worker programs per host under Supervisor: one for high-priority tasks, two for default queues, and one for bulk processing. When a worker crashes due to memory pressure, Supervisor restarts it within 1 second with zero operator intervention.

Supervisor Configuration

; /etc/supervisor/conf.d/celery.conf
[group:celery]
programs=worker-high,worker-default,worker-bulk,beat

[program:celery-worker-high]
command=celery -A tasks worker -Q high --concurrency=8 --loglevel=info
directory=/opt/app
user=celeryuser
numprocs=1
autostart=true
autorestart=true
startretries=3
stopwaitsecs=300
stopasgroup=true
killasgroup=true
stdout_logfile=/var/log/celery/worker-high.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10
stderr_logfile=/var/log/celery/worker-high.error.log
stderr_logfile_maxbytes=50MB
environment=
    CELERY_BROKER_URL="redis://localhost:6379/0",
    CELERY_RESULT_BACKEND="redis://localhost:6379/0"

[program:celery-worker-default]
command=celery -A tasks worker -Q default --concurrency=4 --loglevel=info
directory=/opt/app
user=celeryuser
numprocs=2
autostart=true
autorestart=true
stopwaitsecs=300
stdout_logfile=/var/log/celery/worker-default.log
stdout_logfile_maxbytes=50MB
stdout_logfile_backups=10

[program:celery-worker-bulk]
command=celery -A tasks worker -Q bulk --concurrency=2 --loglevel=info
directory=/opt/app
user=celeryuser
numprocs=1
autostart=true
autorestart=true
stopwaitsecs=600
stdout_logfile=/var/log/celery/worker-bulk.log

[program:celery-beat]
command=celery -A tasks beat --loglevel=info --schedule=/var/run/celery/beat-schedule
directory=/opt/app
user=celeryuser
numprocs=1
autostart=true
autorestart=true
stopwaitsecs=30
stdout_logfile=/var/log/celery/beat.log

Apply configuration:

supervisorctl reread
supervisorctl update
supervisorctl status

Expected output:

celery:worker-high        RUNNING   pid 12345, uptime 0:10:32
celery:worker-default:0   RUNNING   pid 12346, uptime 0:10:32
celery:worker-default:1   RUNNING   pid 12347, uptime 0:10:32
celery:worker-bulk        RUNNING   pid 12348, uptime 0:10:32
celery:beat               RUNNING   pid 12349, uptime 0:10:32

Rolling Restart

supervisorctl stop celery:worker-high
supervisorctl start celery:worker-high

Rolling restart script:

#!/bin/bash
# rolling-restart.sh
PROGRAMS=("celery:worker-high" "celery:worker-default:0"
          "celery:worker-default:1" "celery:worker-bulk")

for program in "${PROGRAMS[@]}"; do
    echo "Restarting $program..."
    supervisorctl stop "$program"
    sleep 2
    supervisorctl start "$program"
    sleep 5
    if supervisorctl status "$program" | grep -q RUNNING; then
        echo "$program restarted successfully"
    else
        echo "ERROR: $program failed to start"
        exit 1
    fi
done

echo "Rolling restart complete"

Expected output:

Restarting celery:worker-high...
celery:worker-high: stopped
celery:worker-high: started
celery:worker-high restarted successfully
Restarting celery:worker-default:0...
...
Rolling restart complete

Event Listener

; /etc/supervisor/conf.d/celery-events.conf
[eventlistener:celery-monitor]
command=python /opt/app/monitor_events.py
events=PROCESS_STATE
buffer_size=100

[eventlistener:celery-crash-alert]
command=python /opt/app/crash_alert.py
events=PROCESS_STATE_FATAL
buffer_size=10

# monitor_events.py
import sys

def write_stdout(s):
    sys.stdout.write(s)
    sys.stdout.flush()

def write_stderr(s):
    sys.stderr.write(s)
    sys.stderr.flush()

def main():
    while True:
        write_stdout('READY\n')
        line = sys.stdin.readline()
        headers = dict(x.split(':') for x in line.split())
        payload = sys.stdin.read(int(headers.get('len', 0)))
        event = headers.get('eventname', '')
        print(f"[MONITOR] Event: {event}")
        print(f"[MONITOR] Payload: {payload.strip()}")
        write_stdout('RESULT 2\nOK')

if __name__ == '__main__':
    main()

Common Mistakes

• Not setting stopwaitsecs high enough -- Supervisor sends SIGTERM and waits stopwaitsecs before SIGKILL. If set too low (default 10s), workers with long-running tasks are killed forcefully. Set to max task duration + 60s buffer.
• Using stopasgroup=false -- without stopasgroup, Supervisor sends SIGTERM to the main Celery process but not its child worker processes. The parent exits but children become orphans. Always set stopasgroup=true and killasgroup=true.
• Running workers as root -- Supervisor runs as root by default. Configure the user= directive to run workers as a non-privileged user. This limits damage if a worker is compromised.
• Not rotating logs -- without log rotation, Supervisor log files grow indefinitely, filling disks. Set stdout_logfile_maxbytes and stdout_logfile_backups for automatic rotation.
• Using numprocs for workers with different configurations -- numprocs creates identical copies. For different concurrency or queue settings, use separate program definitions instead of numprocs.

Practice Questions

1. Why is stopasgroup=true important for Celery workers under Supervisor?
2. How does autorestart differ from startretries?
3. How do you implement rolling restarts with Supervisor?
4. What information can Supervisor event listeners capture?
5. How do you configure per-worker log files?

Challenge

Build a Supervisor management system for Celery: (1) configuration template that generates per-host Supervisor config from a YAML definition, (2) rolling restart script that respects queue draining (stops workers for queue X, waits for in-flight to finish, restarts), (3) event listener that sends Slack alerts when a worker enters FATAL state, (4) log aggregation that collects all Celery worker logs into a central location with daily rotation, (5) a supervisorctl wrapper that provides one-command deployment of new worker configurations.

FAQ

Mini Project

Build a Supervisor-based Celery deployment toolkit: (1) config generator that produces per-environment Supervisor configs from environment variables, (2) rolling restart with health check verification between each worker restart, (3) event listener that records worker lifecycle events to a database for auditing, (4) log shipping configuration that sends Celery logs to Elasticsearch via Filebeat, (5) alert integration that notifies on-call when a worker restarts more than 3 times in 5 minutes, and (6) a status dashboard showing all Celery processes under Supervisor management.

What's Next

Continue with Systemd Integration to learn running Celery as a systemd service. Then explore Monitoring with Prometheus for metrics-driven worker monitoring.