Celery Monitoring: Prometheus, Grafana, and Flower for Worker Observability

In this tutorial, you will learn about Celery Monitoring: Prometheus, Grafana, and Flower for Worker Observability. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery monitoring uses Prometheus for metrics collection and alerting, Grafana for real-time dashboards of queue depth and worker throughput, and Flower for web-based task inspection and worker management in production Celery deployments.

flowchart LR
    CW[Celery Worker] -->|Metrics| PE[Prometheus Exporter]
    PE -->|Scrape| Prom[Prometheus]
    Prom -->|Data Source| Grafana[Grafana Dashboard]
    Prom -->|Alerts| Alert[Alertmanager]
    Flower[Flower Web UI] -->|Monitor| W[Celery Workers]
    Flower -->|Inspect| T[Tasks]
    CW -->|Events| Flower

What You'll Learn

• Prometheus metrics for Celery
• Grafana dashboards
• Flower web-based monitoring
• Custom metric instrumentation
• Alerting rules for worker health

Why It Matters

Without monitoring, you cannot know if workers are healthy, queues are growing, or tasks are failing. Flower provides per-task visibility, Prometheus enables historical trends and alerting, and Grafana unifies it all into operational dashboards.

Real-World Use

DodaTech runs a Grafana dashboard showing 50+ Celery workers across 3 clusters. The ops team monitors queue depth, task latency p50/p99, worker memory usage, and task error rates in real time. Alerts fire when any queue exceeds 10000 tasks or error rate exceeds 5%.

Flower Setup

from celery import Celery

app = Celery('monitoring', broker='redis://localhost:6379/0')

@app.task
def sample_task(x):
    return x * 2

sample_task.delay(21)
sample_task.delay(42)

Start Flower:

celery -A monitoring flower --port=5555 --loglevel=info

Expected output:

[2026-06-28 10:00:00: INFO] Flower started on port 5555
[2026-06-28 10:00:00: INFO] Visit http://localhost:5555

Flower features via API:

curl http://localhost:5555/api/workers
curl http://localhost:5555/api/tasks
curl http://localhost:5555/api/queues/length

Expected output:

{"celery@host": {"status": "online", "active": 2, "processed": 150}}

Prometheus Metrics

from celery import Celery
from celery.signals import task_prerun, task_postrun, task_failure
import time

app = Celery('metrics', broker='redis://localhost:6379/0')

task_duration = {}
task_count = {}
task_errors = {}

@task_prerun.connect
def track_start(sender=None, task_id=None, **kwargs):
    task_duration[task_id] = time.time()
    task_name = sender.name if sender else 'unknown'
    task_count[task_name] = task_count.get(task_name, 0) + 1

@task_postrun.connect
def track_end(sender=None, task_id=None, **kwargs):
    start = task_duration.pop(task_id, None)
    if start:
        elapsed = time.time() - start
        task_name = sender.name if sender else 'unknown'
        print(f"[METRIC] task_duration_seconds{{task=\"{task_name}\"}} {elapsed:.3f}")
        print(f"[METRIC] task_count_total{{task=\"{task_name}\"}} {task_count.get(task_name, 0)}")

@task_failure.connect
def track_error(sender=None, task_id=None, **kwargs):
    task_name = sender.name if sender else 'unknown'
    task_errors[task_name] = task_errors.get(task_name, 0) + 1
    print(f"[METRIC] task_errors_total{{task=\"{task_name}\"}} {task_errors[task_name]}")

@app.task
def process(item):
    time.sleep(0.1)
    return f"Processed {item}"

for i in range(5):
    process.delay(i)
time.sleep(1)

Expected output:

[METRIC] task_duration_seconds{task="metrics.process"} 0.105
[METRIC] task_count_total{task="metrics.process"} 1
[METRIC] task_duration_seconds{task="metrics.process"} 0.102
[METRIC] task_count_total{task="metrics.process"} 5

Grafana Dashboard Queries

# Task throughput (tasks/sec)
rate(celery_task_succeeded_total[5m])

# Queue depth
celery_queue_length{queue="default"}

# Task duration p99
histogram_quantile(0.99, rate(celery_task_runtime_bucket[5m]))

# Worker memory usage (if cAdvisor or node_exporter)
container_memory_usage_bytes{container="celery-worker"}

# Error rate
rate(celery_task_failed_total[5m]) / rate(celery_task_succeeded_total[5m]) * 100

# Active workers
count(celery_worker_online == 1)

Expected output: A Grafana dashboard showing panels for: queue depth gauge, task throughput graph, error rate line, worker count stat, and task duration heatmap.

Common Mistakes

• Not setting up broker monitoring -- queue depth monitoring requires broker-level metrics (Redis INFO, RabbitMQ API). Don't rely solely on Celery metrics. Monitor broker memory, connection count, and message rates directly.
• Over-scraping Celery metrics -- Celery's Prometheus exporter or events can generate many metrics. High-cardinality labels (task_id) explode Prometheus storage. Keep task_id out of metric labels, use task_name only.
• Flower consuming too many resources -- Flower uses Celery events, which can be high-volume. Enable Flower in production with --max-tasks=10000 and --persistent=false to limit memory growth.
• No alerting on queue growth -- queue depth in Grafana is reactive, not proactive. Set Prometheus alerting rules for queue depth thresholds. Alert when depth exceeds 5-minute sustained maximum.
• Ignoring task runtime outliers -- average task duration hides slow tasks. Monitor p99 and p999 task duration. Use Prometheus histograms for duration distribution rather than summary metrics.

Practice Questions

1. What is Flower and how does it monitor Celery?
2. How do you export Celery metrics to Prometheus?
3. What Grafana queries would show task error rates?
4. Why should you avoid high-cardinality labels in Prometheus metrics?
5. How do you set up alerting for queue depth?

Challenge

Build a complete monitoring stack for Celery: (1) deploy Prometheus with a Celery exporter sidecar on each worker pod, (2) create Grafana dashboard panels for queue depth (by queue), task throughput (5m rate), error rate (%), task duration histograms (p50, p95, p99), worker count and memory, (3) set up Prometheus alerting rules for: queue depth > 1000 for 5 minutes, error rate > 5% for 2 minutes, zero workers online for 1 minute, (4) deploy Flower with authentication and resource limits, (5) integrate alerts with PagerDuty or Slack.

FAQ

Mini Project

Build a comprehensive monitoring solution: (1) instrument a Celery app with Prometheus client metrics (task duration histogram, error counter, active tasks gauge, queue depth gauge via Redis INFO), (2) create a Grafana dashboard with 8 panels: queue depth per queue, throughput rate, error rate, worker count, latency heatmap (p50/p95/p99), worker resource usage, broker health, and task success/failure breakdown by task name, (3) configure Alertmanager with routes for critical (PagerDuty) and warning (Slack) alerts, (4) deploy Flower for per-task debugging with persistent mode and 1-hour retention.

What's Next

Continue with Alerting and Troubleshooting to learn production issue response strategies. Then explore Performance Optimization for tuning Celery for high throughput.