Introduction to Background Jobs

In this tutorial, you will learn about Introduction to Background Jobs. We cover key concepts, practical examples, and best practices to help you master this topic.

Background jobs move time-consuming tasks out of the request-response cycle into separate worker processes for better application performance and user experience.

What You Learn

You will learn what background jobs are, the difference between synchronous and asynchronous processing, common use cases, and how background job systems work.

Why It Matters

Web applications must respond to HTTP requests in milliseconds. Operations like sending emails, processing images, or generating reports take seconds or minutes. Without background jobs, these operations block the user's request, causing timeouts and poor user experience.

Real-World Use

Doda Browser uses background jobs for every non-critical operation. When you upload a file, the response is immediate. A background job scans the file, generates thumbnails, and updates the status. You never wait for these operations to finish.

What Are Background Jobs?

flowchart LR
    U[User] -->|Request| W[Web Server]
    W -->|Immediate Response| U
    W -->|Enqueue Job| Q[Job Queue]
    Q -->|Process| WK[Worker 1]
    Q -->|Process| WK2[Worker 2]
    Q -->|Process| WK3[Worker 3]
    WK -->|Update| DB[Database]
    style Q fill:#f90,color:#fff

Background jobs are tasks that execute outside the main application flow. The web server queues a job and returns a response immediately. A worker process picks up the job and executes it asynchronously.

Key Concepts

Synchronous vs Asynchronous

import time

# Synchronous - blocks until complete
def send_email_sync(to, subject):
    print(f"Sending email to {to}...")
    time.sleep(2)  # Simulate network call
    print(f"Email sent to {to}")
    return "sent"

# The caller waits 2 seconds
result = send_email_sync("user@example.com", "Welcome")
print(f"Result: {result}")  # User waited 2 seconds for this

Expected output:

Sending email to user@example.com...
Email sent to user@example.com
Result: sent

# Asynchronous with background job
import redis
import json
import time

r = redis.Redis()
job = {'to': 'user@example.com', 'subject': 'Welcome'}
r.lpush('email_queue', json.dumps(job))
print("Email queued, response immediate")

Expected output:

Email queued, response immediate

Common Use Cases

Job Processing Flow

import redis
import json
import time
import threading

r = redis.Redis()

# Producer - enqueues jobs
def produce_jobs():
    for i in range(5):
        job = {'task_id': i, 'data': f'item_{i}'}
        r.lpush('work_queue', json.dumps(job))
        print(f"Enqueued job {i}")
        time.sleep(0.1)

# Worker - processes jobs
def worker(name):
    while True:
        job_data = r.brpop('work_queue', timeout=5)
        if job_data:
            _, data = job_data
            job = json.loads(data)
            print(f"[{name}] Processing job {job['task_id']}")
            time.sleep(0.5)
            print(f"[{name}] Job {job['task_id']} done")
        else:
            break

# Run
producer_thread = threading.Thread(target=produce_jobs, daemon=True)
worker1 = threading.Thread(target=worker, args=('W1',), daemon=True)
worker2 = threading.Thread(target=worker, args=('W2',), daemon=True)

producer_thread.start()
worker1.start()
worker2.start()
time.sleep(4)

Expected output:

Enqueued job 0
Enqueued job 1
Enqueued job 2
Enqueued job 3
Enqueued job 4
[W1] Processing job 0
[W2] Processing job 1
[W1] Job 0 done
[W2] Job 1 done
[W1] Processing job 2
[W2] Processing job 3
[W1] Job 2 done
[W2] Job 3 done
[W1] Processing job 4
[W1] Job 4 done

Background Job System Components

# Simple background job system
import redis
import json
from functools import wraps

r = redis.Redis()

class BackgroundJob:
    def __init__(self, queue='default'):
        self.queue = queue

    def enqueue(self, func, *args, **kwargs):
        job = {
            'func': f"{func.__module__}.{func.__name__}",
            'args': args,
            'kwargs': kwargs,
        }
        r.lpush(self.queue, json.dumps(job))
        return job

def background(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        job = BackgroundJob()
        return job.enqueue(func, *args, **kwargs)
    return wrapper

@background
def process_file(path):
    print(f"Processing {path}")
    return "done"

@background
def send_email(to, subject):
    print(f"Sending email to {to}: {subject}")
    return "sent"

# These execute in background
process_file("/data/file.txt")
send_email("user@example.com", "Welcome")

print("Main thread continues immediately")

Expected output:

Main thread continues immediately

Common Mistakes

1. Using Background Jobs for Everything

Not everything needs a background job. If an operation takes under 100ms, execute it synchronously. Background jobs add complexity and should only be used when the delay matters.

2. Not Handling Worker Failures

If a worker crashes, in-progress jobs are lost without proper acknowledgment mechanisms. Use job acknowledgements (acks) to ensure reliable processing.

3. Ignoring Job Queue Backpressure

A producer that enqueues jobs faster than workers process them causes queue growth. Monitor queue depth and set up backpressure mechanisms.

4. Passing Non-Serializable Data

Jobs must be serializable (JSON, msgpack). Do not pass database connections, file handles, or complex objects as job arguments.

5. Forgetting Job Deduplication

The same job may be enqueued multiple times (user double-clicks submit). Implement idempotency or deduplication at the job level.

Practice Questions

1. What is a background job?

A task executed outside the main request-response cycle by a separate worker process. The web server queues the job and returns immediately.

2. What is the difference between sync and async processing?

Synchronous: the caller waits for completion. Asynchronous: the caller queues the work and continues. Background jobs use asynchronous processing.

3. What components make up a background job system?

Producer (enqueues jobs), Queue (stores pending jobs), Worker (processes jobs), and optionally a Result Store (stores outcomes).

4. What types of data can be passed to a background job?

Serializable data: strings, numbers, lists, dictionaries. Pass IDs for database records rather than the records themselves.

Challenge

Design a background job system for a video sharing platform. Users upload videos, and the system must: validate format, transcode to multiple resolutions, generate thumbnails, analyze content, store metadata, and notify the user. Map each step with time estimates and choose appropriate job processing strategies.

FAQ

Mini Project: Simple Background Job System

# simple_bg.py
import redis
import json
import time
import threading
from functools import wraps

r = redis.Redis()

class SimpleWorker:
    def __init__(self, queue='default'):
        self.queue = queue
        self.handlers = {}
        self.running = False

    def register(self, name):
        def decorator(func):
            self.handlers[name] = func
            return func
        return decorator

    def enqueue(self, task_name, *args, **kwargs):
        job = {
            'task': task_name,
            'args': args,
            'kwargs': kwargs,
            'enqueued_at': time.time(),
        }
        r.lpush(self.queue, json.dumps(job))
        print(f"Enqueued: {task_name}")

    def process_job(self, job_data):
        try:
            job = json.loads(job_data)
            task_name = job['task']
            handler = self.handlers.get(task_name)
            if handler:
                print(f"Processing: {task_name}")
                result = handler(*job['args'], **job['kwargs'])
                print(f"Completed: {task_name} -> {result}")
            else:
                print(f"No handler for: {task_name}")
        except Exception as e:
            print(f"Failed: {e}")

    def start(self):
        self.running = True
        while self.running:
            job_data = r.brpop(self.queue, timeout=1)
            if job_data:
                _, data = job_data
                self.process_job(data)

    def stop(self):
        self.running = False

# Usage
worker = SimpleWorker()

@worker.register('send_email')
def send_email(to, subject):
    time.sleep(0.5)
    return f"Email sent to {to}"

@worker.register('process_image')
def process_image(path):
    time.sleep(1)
    return f"Image processed: {path}"

# Enqueue jobs
worker.enqueue('send_email', 'user@example.com', 'Welcome')
worker.enqueue('process_image', '/uploads/photo.jpg')

# Start worker in background thread
t = threading.Thread(target=worker.start, daemon=True)
t.start()

time.sleep(3)
print("Done")

Expected output:

Enqueued: send_email
Enqueued: process_image
Processing: send_email
Completed: send_email -> Email sent to user@example.com
Processing: process_image
Completed: process_image -> Image processed: /uploads/photo.jpg
Done

What's Next

Now that you understand background jobs, learn about synchronous vs asynchronous processing in detail, then explore common use cases for background jobs.