Elasticsearch Cluster: Nodes, Shards & Replication

In this tutorial, you'll learn about Elasticsearch Cluster: Nodes, Shards & Replication. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

An Elasticsearch cluster is a group of nodes that collectively store data, provide search capabilities, and distribute workload through Sharding and Replication for scalability and fault tolerance.

What You'll Learn

In this tutorial, you will learn about node roles (master, data, ingest, coordinating), how primary and replica shards work, how data is routed to shards, and how to configure a multi-node cluster for high availability.

Why It Matters

A single-node Elasticsearch instance is a single point of failure. If it goes down, all log data and search capabilities are lost. A properly configured cluster distributes data across nodes, replicates it for redundancy, and scales horizontally as data grows. Understanding cluster architecture is essential for running ELK in production.

Real-World Use

Durga Antivirus Pro runs a 5-node Elasticsearch cluster across two availability zones. Each data node holds 10 primary shards and 2 replicas. When one availability zone experiences a network issue, the cluster continues serving queries from the replica shards in the other zone. The operations team can add nodes to increase capacity without any downtime.

Node Roles

Elasticsearch nodes can have multiple roles:

• Master -- manages cluster state, coordinates nodes
• Data -- stores indexed data, handles CRUD and search
• Ingest -- pre-processes documents before indexing
• Coordinating -- routes requests, aggregates results

For production, dedicate at least 3 master-eligible nodes for quorum and separate data nodes for storage.

Shards and Replicas

An index in Elasticsearch is divided into shards. Each shard is a full Lucene index.

• Primary shards -- hold the original data. The number is set at index creation and cannot be changed.
• Replica shards -- copies of primary shards for redundancy and read throughput.

Create an index with explicit shard settings:

curl -X PUT "localhost:9200/logs" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}'

With 5 primary shards and 1 replica, the index uses 10 total shards across your cluster.

Data Distribution

When you index a document, Elasticsearch determines the target shard:

shard = hash(document_id) % number_of_primary_shards

This ensures even distribution across shards. The primary shard then replicates the document to all replica shards on different nodes.

Cluster Health

Monitor cluster health via the API:

curl "localhost:9200/_cluster/health"

Expected output:

{
  "cluster_name": "elasticsearch",
  "status": "green",
  "number_of_nodes": 5,
  "number_of_data_nodes": 3,
  "active_primary_shards": 50,
  "active_shards": 100,
  "unassigned_shards": 0
}

Status values: green (all shards assigned), yellow (replicas unassigned), red (primary shards unassigned).

Configuring a Multi-Node Cluster

Configure each node with a shared cluster.name and unique node.name:

# elasticsearch.yml on node-1
cluster.name: logs-cluster
node.name: node-1
node.roles: [ master, data ]
discovery.seed_hosts: [ "node-1:9300", "node-2:9300", "node-3:9300" ]
cluster.initial_master_nodes: [ "node-1", "node-2", "node-3" ]

# elasticsearch.yml on node-2
cluster.name: logs-cluster
node.name: node-2
node.roles: [ data ]
discovery.seed_hosts: [ "node-1:9300", "node-2:9300", "node-3:9300" ]

Common Mistakes

1. Too Many Primary Shards

More shards is not better. Each shard has overhead. Aim for 20-40GB per shard. An index with 100 shards for 2GB of data wastes resources.

2. Uneven Shard Distribution

If nodes have different capacities, shards may land unevenly. Use index.routing.allocation.total_shards_per_node to control distribution.

3. Split-Brain Without Enough Masters

A cluster with only 2 master-eligible nodes can suffer split-brain. Always run 3 master-eligible nodes for proper quorum.

4. Not Setting Watermark Thresholds

Disk watermark thresholds prevent nodes from running out of space. Configure cluster.routing.allocation.disk.watermark.low and high.

5. Ignoring Shard Rebalancing

When adding or removing nodes, Elasticsearch rebalances shards. This can cause performance impact. Use _cluster/reroute or set cluster.routing.rebalance.enable to control timing.

Practice Questions

1. What are the main node roles in Elasticsearch? Master, data, ingest, and coordinating. Some nodes can serve multiple roles.

2. What is the difference between a primary shard and a replica shard? A primary shard holds the original data. A replica shard is a copy that provides redundancy and serves read requests.

3. What does a green cluster health status mean? All primary and replica shards are assigned to nodes. The cluster is fully operational.

4. How does Elasticsearch determine which shard a document goes to? It computes hash(document_id) % number_of_primary_shards for routing.

5. Challenge: Design a cluster for 10TB of log data with 30-day retention, specifying node count, roles, shard count, and replica factor, ensuring zero data loss if one node fails.

What's Next

Master Logstash grok patterns to parse complex, unstructured log formats into structured data.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.