RabbitMQ Clustering — Complete Guide

In this tutorial, you will learn about RabbitMQ Clustering. We cover key concepts, practical examples, and best practices to help you master this topic.

RabbitMQ clustering connects multiple broker nodes into a single logical broker, enabling horizontal scaling, high availability, and fault tolerance.

What You Learn

You will learn how RabbitMQ clustering works, how to set up a multi-node cluster, how nodes communicate, and the trade-offs of different cluster topologies.

Why It Matters

A single RabbitMQ node is a single point of failure. If it crashes, all queues, exchanges, and bindings are unavailable. Clustering distributes load across nodes and provides continuity when individual nodes fail.

Real-World Use

Doda Browser runs a 3-node RabbitMQ cluster across availability zones. If one zone goes down, the other two continue processing messages. No scan requests are lost during infrastructure failures.

Cluster Architecture

flowchart TB
    subgraph "RabbitMQ Cluster"
        N1[Node 1
rabbit@node1] --- N2[Node 2
rabbit@node2]
        N2 --- N3[Node 3
rabbit@node3]
        N3 --- N1
    end
    P[Producer] -->|connect to any node| N1
    C[Consumer] -->|connect to any node| N2
    style N1 fill:#f90,color:#fff
    style N2 fill:#f90,color:#fff
    style N3 fill:#f90,color:#fff

All nodes in a cluster share: exchanges, bindings, users, virtual hosts, and permissions. Queues live on specific nodes but are visible from any node.

Setting Up a 3-Node Cluster

On each node, set the Erlang cookie and hostname:

# Node 1: rabbit@node1
sudo tee /etc/rabbitmq/rabbitmq.conf << EOF
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = rabbit@node1
cluster_formation.classic_config.nodes.2 = rabbit@node2
cluster_formation.classic_config.nodes.3 = rabbit@node3
EOF

# Copy Erlang cookie to all nodes
# /var/lib/rabbitmq/.erlang.cookie must be identical
sudo systemctl start rabbitmq-server

Verify cluster status:

sudo rabbitmqctl cluster_status

Expected output:

Cluster status of node rabbit@node1 ...
Basics
Cluster name: rabbit@node1
Disk Nodes: [rabbit@node1, rabbit@node2, rabbit@node3]
Running Nodes: [rabbit@node1, rabbit@node2, rabbit@node3]

Adding Nodes to an Existing Cluster

# On the new node, stop and join
sudo rabbitmqctl stop_app
sudo rabbitmqctl join_cluster rabbit@node1
sudo rabbitmqctl start_app

# Verify
sudo rabbitmqctl cluster_status

Client Connection to a Cluster

Clients connect to any node. The node forwards operations to the correct owning node:

import pika

# Connect to any node in the cluster
params = pika.ConnectionParameters(
    host='rabbitmq-cluster.example.com',
    port=5672,
    credentials=pika.PlainCredentials('guest', 'guest')
)
connection = pika.BlockingConnection(params)
channel = connection.channel()

channel.queue_declare(queue='cluster_queue', durable=True)
print(f"Connected to cluster node: {connection.server_properties.get('platform')}")

channel.close()
connection.close()

Cluster Node Types

Network Partitions

# Configure partition handling
sudo tee -a /etc/rabbitmq/rabbitmq.conf << EOF
cluster_partition_handling = pause_minority
EOF

sudo systemctl restart rabbitmq-server

Partition handling strategies:

• pause_minority: Smallest side of the partition pauses (recommended)
• pause_if_all_down: Pause if cannot reach all listed nodes
• autoheal: Automatically recover after partition
• ignore: Do nothing (risks split-brain)

Common Mistakes

1. Mismatched Erlang Cookies

Each node must have the exact same .erlang.cookie file. Mismatched cookies prevent nodes from joining the cluster.

2. Using RAM Nodes for Critical Data

RAM nodes lose metadata on restart. Use disk nodes for production. RAM nodes only for performance-critical, non-durable metadata.

3. Not Configuring Partition Handling

Without partition handling, a network split creates split-brain where each side operates independently. Queues and messages diverge permanently.

4. Connecting to a Single Node

Clients should connect through a load balancer or use a connection library that handles failover. Connecting to one node directly is a single point of failure.

5. Ignoring Network Latency

Cluster nodes should have low-latency connections (under 5ms RTT). High latency causes timeouts and false partition detection.

Practice Questions

1. What is the minimum recommended cluster size?

3 nodes. This provides fault tolerance (survive 1 node failure) and a majority for partition handling decisions.

2. What is an Erlang cookie used for?

Authentication between cluster nodes. All nodes must share the same cookie to communicate securely.

3. What is the difference between a disk node and a RAM node?

Disk nodes store metadata on disk and survive restarts. RAM nodes store metadata in memory and must rejoin after restart. Queue data is always on disk.

4. How does a cluster handle a Network Partition?

Based on cluster_partition_handling setting: pause_minority pauses the smaller side, autoheal tries to recover, ignore does nothing.

Challenge

Design a RabbitMQ cluster topology for a global application with nodes in US, EU, and Asia. Consider network latency between regions, partition handling, and whether a single cluster is appropriate or multiple federated clusters.

FAQ

Mini Project: 3-Node Cluster Setup Script

#!/bin/bash
# Setup a 3-node RabbitMQ cluster on Ubuntu
# Run on each node with the node's hostname

set -e

NODE_NAME=$(hostname)
COOKIE="DODATECH_SECRET_COOKIE_VALUE"
NODES=("node1" "node2" "node3")

# Install RabbitMQ
sudo apt-get update
sudo apt-get install -y rabbitmq-server

# Stop and configure
sudo systemctl stop rabbitmq-server

# Set identical Erlang cookie on all nodes
echo "$COOKIE" | sudo tee /var/lib/rabbitmq/.erlang.cookie
sudo chmod 400 /var/lib/rabbitmq/.erlang.cookie
sudo chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie

# Configure cluster
sudo tee /etc/rabbitmq/rabbitmq.conf << EOF
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_partition_handling = pause_minority
EOF

# Add node config
for i in "${!NODES[@]}"; do
    echo "cluster_formation.classic_config.nodes.$((i+1)) = rabbit@${NODES[$i]}" | \
        sudo tee -a /etc/rabbitmq/rabbitmq.conf
done

# Enable management UI
sudo rabbitmq-plugins enable rabbitmq_management

# Start
sudo systemctl start rabbitmq-server

# Join cluster (run on node2 and node3 only)
if [ "$NODE_NAME" != "node1" ]; then
    sudo rabbitmqctl stop_app
    sudo rabbitmqctl join_cluster rabbit@node1
    sudo rabbitmqctl start_app
fi

# Verify
sudo rabbitmqctl cluster_status

echo "Node $NODE_NAME cluster setup complete"

Expected output:

Node node1 cluster setup complete

What's Next

Now that you understand RabbitMQ clustering, explore queue mirroring for replicating queue contents, then learn about the management UI for cluster monitoring.