Sidekiq for Ruby Background Jobs
In this tutorial, you will learn about Sidekiq for Ruby Background Jobs. We cover key concepts, practical examples, and best practices to help you master this topic.
Sidekiq is a high-performance background job processor for Ruby using Redis as the backend, supporting concurrency, scheduling, retries, and monitoring.
What You Learn
You will learn how to set up Sidekiq, define workers, configure sidekiq.yml, use Active Job integration, handle retries, and monitor jobs with the Sidekiq web UI.
Why It Matters
Sidekiq is the standard background job processor for Ruby on Rails. It processes millions of jobs per day in production at companies like GitHub, Shopify, and GitLab.
Real-World Use
DodaTech uses Sidekiq for all Ruby-based background operations. When a user signs up for Durga Antivirus Pro, Sidekiq processes the subscription, sends the welcome email, provisions the license, and logs the activity.
Basic Setup
# Gemfile
gem 'sidekiq'
gem 'redis'
# app/workers/email_worker.rb
class EmailWorker
include Sidekiq::Worker
sidekiq_options queue: 'email', retry: 3, backtrace: true
def perform(to, subject, body)
puts "Sending email to #{to}: #{subject}"
sleep 1
puts "Email sent to #{to}"
end
end
# Enqueue job
EmailWorker.perform_async('user@example.com', 'Welcome', 'Thanks for joining!')
EmailWorker.perform_in(10.minutes, 'user@example.com', 'Reminder', 'Check your inbox')
EmailWorker.perform_at(1.day.from_now, 'user@example.com', 'Follow-up', 'How is it going?')
Worker Definition
class ImageWorker
include Sidekiq::Worker
sidekiq_options(
queue: 'images',
retry: 5,
backoff: true,
timeout: 30,
lock: :until_executed,
)
def perform(image_path, operations)
puts "Processing image: #{image_path}"
operations.each do |op|
puts " Running #{op}..."
sleep 1
puts " #{op} complete"
end
puts "Image processing complete"
end
end
Sidekiq Configuration
# config/sidekiq.yml
:concurrency: 5
:queues:
- [critical, 5]
- [default, 3]
- [email, 2]
- [batch, 1]
:timeout: 30
:max_retries: 3
:pidfile: tmp/pids/sidekiq.pid
:logfile: log/sidekiq.log
:daemon: false
# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = {
url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0'),
network_timeout: 5,
pool_size: 10,
}
end
Sidekiq.configure_client do |config|
config.redis = {
url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0'),
pool_size: 5,
}
end
Active Job Integration
# config/application.rb
config.active_job.queue_adapter = :sidekiq
# app/jobs/process_upload_job.rb
class ProcessUploadJob < ApplicationJob
queue_as :default
retry_on ActiveRecord::Deadlocked, wait: :exponentially_longer
discard_on ActiveJob::DeserializationError
def perform(upload)
puts "Processing upload: #{upload.id}"
upload.process!
puts "Upload processed: #{upload.status}"
end
end
# Enqueue
ProcessUploadJob.perform_later(upload)
ProcessUploadJob.set(wait: 1.hour).perform_later(upload)
ProcessUploadJob.set(priority: 10).perform_later(upload)
Retry and Error Handling
class PaymentWorker
include Sidekiq::Worker
sidekiq_options retry: {
max: 5,
base: 2,
max_elapsed_time: 3600,
strategies: [
{ type: :exponential, factor: 2 },
{ type: :custom, ->(count) { count * 60 } },
]
}
def perform(order_id, amount)
puts "Processing payment for order #{order_id}: $#{amount}"
raise "Payment gateway timeout" if rand < 0.3
puts "Payment successful for #{order_id}"
rescue StandardError => e
if retries_exhausted?
AdminNotifier.failed_payment(order_id, amount)
raise
end
raise
end
end
Sidekiq Web UI
# config/routes.rb
require 'sidekiq/web'
Rails.application.routes.draw do
mount Sidekiq::Web => '/sidekiq'
end
# Start Sidekiq
bundle exec sidekiq -C config/sidekiq.yml
# Start Rails server
rails server
Access http://localhost:3000/sidekiq for the dashboard.
Batch Processing
class BatchWorker
include Sidekiq::Worker
def perform(items)
batch = Sidekiq::Batch.new
batch.description = "Process #{items.size} items"
batch.callback_queue = :default
batch.on(:success, BatchCallback, type: 'complete')
batch.on(:complete, BatchCallback, type: 'finish')
items.each do |item|
batch.jobs do
ItemWorker.perform_async(item['id'])
end
end
end
end
class BatchCallback
def on_success(status, options)
puts "Batch #{options['type']}: #{status.total} jobs"
end
end
Common Mistakes
1. Not Configuring Redis Pool Size
Sidekiq uses a Redis Connection Pool. If pool size is too low, workers block waiting for connections. Set pool_size to concurrency + 2.
2. Passing Complex Objects
Sidekiq serializes arguments to JSON. Pass record IDs, not model instances. Re-fetch from database inside perform.
3. Not Setting Queue Priorities
Without queue weights, all queues are equal. Set queue weights in sidekiq.yml to prioritize critical queues.
4. Ignoring Job Timeouts
A job that hangs forever blocks a worker slot. Set timeout option in sidekiq_options to kill stuck jobs.
5. Not Monitoring Sidekiq
Run the Sidekiq web UI or use a monitoring tool. Set alerts for queue depth, retry count, and dead jobs.
Practice Questions
1. How do you define a Sidekiq worker?
Include Sidekiq::Worker and define perform method. Options go in sidekiq_options. Call perform_async to enqueue.
2. How does Sidekiq handle retries?
Sidekiq retries failed jobs automatically with exponential backoff. Default is 25 retries. Configure with retry option.
3. What is Active Job integration?
Rails Active Job provides a unified interface for background job processors. Set config.active_job.queue_<a href="/design-patterns/adapter/">Adapter</a> = :sidekiq to use Sidekiq.
4. How do you schedule a delayed job?
Use MyWorker.perform_in(delay, args) or MyWorker.perform_at(timestamp, args) for absolute scheduling.
Challenge
Build a Sidekiq-based order processing system: validation worker, payment worker (3 retries, exponential backoff), inventory worker, notification worker, and a batch callback for order completion. Include monitoring with the Sidekiq web UI.
FAQ
Mini Project: Sidekiq Pipeline
# app/workers/order_worker.rb
class OrderWorker
include Sidekiq::Worker
sidekiq_options queue: 'orders', retry: 3
def perform(order_id, items)
puts "Processing order #{order_id} with #{items.size} items"
sleep 1
PaymentWorker.perform_async(order_id, calculate_total(items))
InventoryWorker.perform_async(order_id, items)
NotificationWorker.perform_in(10, order_id, 'Order confirmed')
puts "Order #{order_id} processing started"
end
private
def calculate_total(items)
items.sum { |i| i['price'] * i['quantity'] }
end
end
# Test
OrderWorker.perform_async('ORD-001', [
{ 'product' => 'Antivirus', 'price' => 49.99, 'quantity' => 1 },
{ 'product' => 'VPN', 'price' => 29.99, 'quantity' => 2 },
])
Expected output:
Processing order ORD-001 with 2 items
Order ORD-001 processing started
What's Next
Now that you understand Sidekiq, explore Huey for Python as a lightweight alternative, then learn about job scheduling for recurring tasks.
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro