Elasticsearch Unassigned Shard Fix

In this tutorial, you'll learn about Elasticsearch Unassigned Shard Fix. We cover key concepts, practical examples, and best practices.

Elasticsearch shows unassigned_shards: 5 in cluster health — shards are stuck in UNASSIGNED state and will not allocate to any node. The cluster is degraded, and data may be at risk.

The Problem

GET /_cat/shards?v&state=UNASSIGNED

index      shard prirep state      node
my-index   0     p      UNASSIGNED
my-index   2     r      UNASSIGNED
logs-2024  1     p      UNASSIGNED

Three shards are unassigned. The primary shard for my-index shard 0 has no copy available — the node that held it went down and never came back.

Step-by-Step Fix

1. Analyze why shards are unassigned

GET /_cluster/allocation/explain
{
  "index": "my-index",
  "shard": 0,
  "primary": true
}

The response tells you the reason: CLUSTER_RECOVERED (node left), INDEX_CREATED (new index), REPLICA_ADDED (more replicas configured), NODE_LEFT (node crashed), EXISTING_INDEX_RESTORED (from snapshot).

2. Fix disk watermarks

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
  }
}

3. Reroute a stale primary

POST /_cluster/reroute
{
  "commands": [
    {
      "allocate_stale_primary": {
        "index": "my-index",
        "shard": 0,
        "node": "data-node-3",
        "accept_data_loss": true
      }
    }
  ]
}

4. Increase node count or rebalance

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.node_concurrent_recoveries": 5,
    "cluster.routing.allocation.cluster_concurrent_rebalance": 5
  }
}

5. Delete a stuck index as last resort

DELETE /my-index

Only if the data is recoverable from elsewhere (snapshot, reindexed copy).

Expected output:

GET /_cluster/health
{
  "status": "green",
  "unassigned_shards": 0
}

Prevention Tips

• Use index.number_of_replicas: 2 for critical indices
• Monitor disk usage and set watermarks before flood_stage hits
• Enable shard allocation awareness for rack/zone distribution
• Schedule regular snapshots for disaster recovery
• Add enough data nodes for replica distribution

Common Mistakes with shard alloc

1. Non-exhaustive pattern matches that compile with warnings then crash at runtime
2. Misunderstanding that String is [Char] with poor performance for large text operations
3. Using foldl instead of foldl' causing stack overflow on large lists

These mistakes appear frequently in real-world ELASTICSEARCH code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ