Celery Worker Pools: Prefork, Gevent, Thread, and Solo Compared

In this tutorial, you will learn about Celery Worker Pools: Prefork, Gevent, Thread, and Solo Compared. We cover key concepts, practical examples, and best practices to help you master this topic.

Celery supports four worker pool implementations -- prefork, gevent, thread, and solo -- each optimized for different workload patterns, concurrency models, and resource profiles in distributed task execution.

flowchart TD
    P[Pool Type Decision] --> CPU{CPU-bound?}
    CPU -->|Yes| Prefork[Prefork Pool
Multi-process]
    CPU -->|No| IO{I/O-bound?}
    IO -->|Yes| Gevent[Gevent Pool
Green threads]
    IO -->|No| Thread[Thread Pool
OS threads]
    P --> Debug{Debugging?}
    Debug --> Solo[Solo Pool
Single process]
    Prefork --> R1[Heavy computation]
    Gevent --> R2[Network calls]
    Thread --> R3[File I/O]
    Solo --> R4[Development]

What You'll Learn

• Prefork pool for CPU-intensive tasks
• Gevent pool for high-concurrency I/O
• Thread pool for lightweight parallelism
• Solo pool for debugging
• Pool selection criteria and benchmarks

Why It Matters

Choosing the wrong pool type causes poor resource utilization, high latency, or crashes. CPU-bound workloads need multiple processes. I/O-bound workloads benefit from green threads. The right pool type doubles throughput without changing infrastructure.

Real-World Use

DodaZIP uses the prefork pool for compression tasks (CPU-bound), gevent pool for network file transfers (I/O-bound), and solo pool during development. Each worker instance uses the pool matched to its workload, maximizing cluster-wide throughput.

Prefork Pool

from celery import Celery

app = Celery('prefork', broker='redis://localhost:6379/0')

app.conf.worker_concurrency = 8
app.conf.worker_pool = 'prefork'
app.conf.worker_prefetch_multiplier = 4

@app.task
def compress_video(input_path, output_path):
    import time
    time.sleep(2)
    result = f"Compressed {input_path} to {output_path}"
    print(result)
    return result

result = compress_video.delay("/videos/input.mp4", "/videos/output.mp4")
print(f"Task ID: {result.id}")

Start with prefork:

celery -A prefork worker --pool=prefork --concurrency=8 --loglevel=info

Expected output:

[2026-06-28 10:00:00: INFO] celery@host ready (prefork:8)
[2026-06-28 10:00:05: INFO] Task compress_video succeeded

Gevent Pool

from celery import Celery
import gevent

app = Celery('gevent_pool', broker='redis://localhost:6379/0')

app.conf.worker_pool = 'gevent'
app.conf.worker_concurrency = 100
app.conf.worker_pool_patch = True

@app.task
def fetch_url(url):
    import requests
    response = requests.get(url, timeout=5)
    length = len(response.text)
    result = f"Fetched {url}: {length} bytes"
    print(result)
    return length

urls = ["https://example.com"] * 50
results = [fetch_url.delay(url) for url in urls]
print(f"Submitted {len(results)} fetch tasks")

Start with gevent:

celery -A gevent_pool worker --pool=gevent --concurrency=100 --loglevel=info

Expected output:

[2026-06-28 10:00:00: INFO] celery@host ready (gevent:100)
Submitted 50 fetch tasks
[2026-06-28 10:00:03: INFO] Task fetch_url succeeded

Thread Pool

from celery import Celery
import time

app = Celery('thread_pool', broker='redis://localhost:6379/0')

app.conf.worker_pool = 'threads'
app.conf.worker_concurrency = 16

@app.task
def process_file(file_path):
    time.sleep(0.5)
    result = f"Processed {file_path}"
    print(result)
    return result

files = [f"file_{i}.txt" for i in range(20)]
start = time.time()
results = [process_file.delay(f) for f in files]
elapsed = time.time() - start
print(f"Submitted {len(results)} tasks in {elapsed:.2f}s")

Start with thread pool:

celery -A thread_pool worker --pool=threads --concurrency=16 --loglevel=info

Expected output:

Submitted 20 tasks in 0.01s
[2026-06-28 10:00:05: INFO] Task process_file succeeded (20 tasks in ~0.5s total)

Solo Pool

from celery import Celery

app = Celery('solo', broker='redis://localhost:6379/0')

app.conf.worker_pool = 'solo'

@app.task
def debug_task(x):
    result = x * 2
    print(f"Debug: {x} * 2 = {result}")
    return result

print("Running in solo mode (single process, no concurrency)")
result = debug_task(21)
print(f"Result: {result}")

Start with solo:

celery -A solo worker --pool=solo --loglevel=debug

Expected output:

Running in solo mode (single process, no concurrency)
Debug: 21 * 2 = 42
Result: 42
[2026-06-28 10:00:00: DEBUG] Task processed inline

Common Mistakes

• Using prefork for I/O-bound tasks -- each Process blocks on I/O, wasting memory. Use gevent for hundreds of concurrent network calls.
• Using gevent without monkey-patching -- gevent requires monkey-patching the standard library. Set worker_pool_patch = True or call gevent.monkey.patch_all() before imports.
• Setting thread concurrency too high -- Python's GIL limits CPU-bound thread performance. Threads excel at I/O but Prefork is better for CPU work.
• Using solo pool in production -- solo processes tasks synchronously. One slow task blocks all others. Solo is for development and debugging only.
• Mixing pool types in one worker -- a single worker instance uses one pool type. Run separate worker instances for different pool types.

Practice Questions

1. Which pool type is best for CPU-intensive video encoding tasks?
2. Why does gevent achieve higher concurrency than threads for I/O work?
3. When would you choose the thread pool over gevent?
4. What is the main limitation of the solo pool?
5. How does the prefork pool handle memory compared to threads?

Challenge

Benchmark all four pool types with 1000 tasks: 500 CPU-bound (prime number calculation) and 500 I/O-bound (HTTP requests). Measure total execution time, peak memory, and CPU utilization for each pool. Determine which pool type gives the best performance for each workload.

FAQ

Mini Project

Build a pool selection benchmark tool that: (1) generates a mixed workload of CPU tasks (prime factorization) and I/O tasks (parallel HTTP fetches), (2) runs the workload against prefork, gevent, and thread pools, (3) measures throughput, latency p50/p99, and peak memory, and (4) produces a recommendation based on the workload ratio. Test with 10/90, 50/50, and 90/10 CPU/I/O splits.

What's Next

Continue with Autoscaling Workers to learn dynamic pool resizing. Then explore Task Coordination for advanced multi-worker task patterns.