Celery Kubernetes Deployment: Orchestrating Celery Workers and Beat on Kubernetes

In this tutorial, you will learn about Celery Kubernetes Deployment: Orchestrating Celery Workers and Beat on Kubernetes. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery Kubernetes deployment uses Deployments for stateless worker pods, StatefulSets for Beat's persistent schedule, HorizontalPodAutoscaler for worker scaling based on queue depth, and Redis from Helm charts for broker and result backend services.

flowchart TD
    Ingress -->|Requests| App[Web App Pods]
    App -->|Tasks| Redis[Redis Service
Broker + Backend]
    Redis --> W[Celery Worker Deployment]
    W -->|HPA| S[HPA: Scale by Queue]
    S -->|Scale Up| W
    Beat[Celery Beat StatefulSet] -->|Schedule| Redis
    CM[ConfigMap] --> W
    CM --> Beat
    Secrets[K8s Secrets] --> W
    Secrets --> Beat

What You'll Learn

• Worker Deployment configuration
• Beat StatefulSet with persistent schedule
• HorizontalPodAutoscaler for workers
• ConfigMaps and Secrets for configuration
• Graceful shutdown in Kubernetes

Why It Matters

Kubernetes provides automated deployment, scaling, and self-healing for containerized applications. Celery on Kubernetes gives you declarative configuration, rolling updates without downtime, and autoscaling based on queue load.

Real-World Use

DodaTech migrated from Docker Compose to Kubernetes for 500+ Celery workers across 10 environments. Kubernetes autoscaling adjusts worker count from 10 to 200 based on queue depth, and rolling updates deploy new code without losing a single task.

Worker Deployment

# celery-worker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: celery-worker
spec:
  replicas: 4
  selector:
    matchLabels:
      app: celery-worker
  template:
    metadata:
      labels:
        app: celery-worker
    spec:
      containers:
      - name: worker
        image: dodatech/celery-worker:latest
        command: ["celery", "-A", "tasks", "worker"]
        args: ["--loglevel=info", "--concurrency=8"]
        env:
        - name: CELERY_BROKER_URL
          value: "redis://redis-service:6379/0"
        - name: CELERY_RESULT_BACKEND
          value: "redis://redis-service:6379/0"
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
          limits:
            cpu: 2000m
            memory: 2Gi
        livenessProbe:
          exec:
            command: ["celery", "-A", "tasks", "status"]
          initialDelaySeconds: 30
          periodSeconds: 30
        readinessProbe:
          exec:
            command: ["celery", "-A", "tasks", "inspect", "ping"]
          initialDelaySeconds: 10
          periodSeconds: 10
        lifecycle:
          preStop:
            exec:
              command: ["celery", "-A", "tasks", "control", "shutdown"]
      terminationGracePeriodSeconds: 300

Deploy:

kubectl apply -f celery-worker.yaml
kubectl get pods -l app=celery-worker

Expected output:

NAME                             READY   STATUS    RESTARTS   AGE
celery-worker-7d8f9c6b4c-abc     1/1     Running   0          10s
celery-worker-7d8f9c6b4c-def     1/1     Running   0          10s
celery-worker-7d8f9c6b4c-ghi     1/1     Running   0          10s
celery-worker-7d8f9c6b4c-jkl     1/1     Running   0          10s

Horizontal Pod Autoscaler

# celery-hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: celery-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: celery-worker
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: celery_queue_depth
        selector:
          matchLabels:
            queue: default
      target:
        type: AverageValue
        averageValue: 100
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
      - type: Pods
        value: 4
        periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 120
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60

kubectl apply -f celery-hpa.yaml
kubectl get hpa celery-worker-hpa -w

Expected output:

NAME                 REFERENCE                  TARGETS      MINPODS   MAXPODS   REPLICAS
celery-worker-hpa    Deployment/celery-worker   80/100       2         20        2
celery-worker-hpa    Deployment/celery-worker   250/100      2         20        5

Beat StatefulSet

# celery-beat.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: celery-beat
spec:
  serviceName: celery-beat
  replicas: 1
  selector:
    matchLabels:
      app: celery-beat
  template:
    metadata:
      labels:
        app: celery-beat
    spec:
      containers:
      - name: beat
        image: dodatech/celery-worker:latest
        command: ["celery", "-A", "tasks", "beat"]
        args: ["--loglevel=info", "--schedule=/var/run/celery/beat-schedule"]
        env:
        - name: CELERY_BROKER_URL
          value: "redis://redis-service:6379/0"
        volumeMounts:
        - name: beat-data
          mountPath: /var/run/celery
  volumeClaimTemplates:
  - metadata:
      name: beat-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 1Gi

kubectl apply -f celery-beat.yaml
kubectl get statefulset celery-beat

Expected output:

NAME          READY   AGE
celery-beat   1/1     30s

Common Mistakes

• No preStop hook for workers -- without preStop, Kubernetes kills workers immediately with SIGKILL, losing in-flight tasks. Add a preStop hook that sends graceful shutdown and set terminationGracePeriodSeconds to cover max task duration.
• HPA based on CPU/memory instead of queue depth -- CPU-based HPA does not reflect actual work demand. Use custom metrics (Prometheus + Adapter) with queue depth for accurate Celery worker scaling.
• Running Beat as a Deployment with replicas>1 -- multiple Beat instances all schedule the same tasks, causing duplicates. Use a StatefulSet with 1 replica or implement leader election for Beat.
• No resource limits -- workers without limits consume all node resources during spikes, starving other pods. Set CPU/memory requests and limits based on per-task profiling.
• Not configuring terminationGracePeriodSeconds -- default is 30 seconds. Long-running tasks need more. Calculate based on soft_time_limit + hard_time_limit + buffer. Set 300+ seconds for tasks with 5-minute timeouts.

Practice Questions

1. Why use StatefulSet for Beat instead of Deployment?
2. How does preStop hook enable graceful worker shutdown in K8s?
3. What metric should you use for HPA scaling of Celery workers?
4. How do you handle Celery configuration across environments in K8s?
5. Why is terminationGracePeriodSeconds important for Celery workers?

Challenge

Build a complete K8s deployment for Celery: (1) Helm chart with configurable worker count, concurrency, and queue names, (2) HPA with custom Prometheus metric for queue depth, (3) Beat StatefulSet with persistent volume for schedule, (4) rolling update Strategy with maxSurge=1 and maxUnavailable=0 (zero-downtime), (5) PodDisruptionBudget that ensures at least 50% of workers are available, (6) NetworkPolicy limiting worker access to only Redis and the result backend.

FAQ

Mini Project

Build a production Helm chart for Celery on K8s: (1) configurable worker deployment with resource limits, concurrency, and queues, (2) Beat StatefulSet with PVC and leader election via Kubernetes-native locking, (3) HPA with queue-depth-based autoscaling using Prometheus metrics, (4) PodDisruptionBudget for worker availability, (5) NetworkPolicy for pod-to-pod communication rules, (6) ServiceMonitor for Prometheus operator integration, and (7) a values.yaml with sensible defaults and documentation for each parameter.

What's Next

Continue with Supervisor Management to learn Process management with Supervisor. Then explore Systemd Integration for running Celery as a system service.