ELK Production: Performance Tuning, Index Lifecycle & ILM

In this tutorial, you'll learn about ELK Production: Performance Tuning, Index Lifecycle & ILM. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Running the ELK Stack in production requires performance tuning for Elasticsearch, implementing Index Lifecycle Management (ILM) for automated index rollover and retention, and monitoring cluster health to ensure reliability at scale.

What You'll Learn

In this tutorial, you will configure ILM policies for hot-warm-cold architecture, tune Elasticsearch for write-heavy workloads, monitor cluster performance with Elasticsearch monitoring APIs, and plan capacity for log growth.

Why It Matters

A production ELK Stack ingests terabytes of log data daily. Without performance tuning, indexing slows down, queries become slow, and the cluster falls behind. Without ILM, indices grow unbounded until disks fill up. These problems cause data loss and outages. Proper production configuration ensures your logging platform keeps up with data volume and stays healthy.

Real-World Use

Durga Antivirus Pro ingests 500GB of log data per day from scanner nodes. The Elasticsearch cluster uses a hot-warm-cold ILM policy: hot nodes index new data on SSDs with high IOPS, warm nodes store 30 days of data on standard SSDs, and cold nodes store 90 days on spinning disks with reduced replicas. This architecture reduced storage costs by 60% while maintaining fast indexing for recent logs.

Step 1: Create an ILM Policy

Index Lifecycle Management automates index transitions through phases:

PUT _ilm/policy/logs-ilm-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate": { "require": { "data": "warm" } }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "allocate": { "require": { "data": "cold" } },
          "freeze": {}
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Step 2: Apply ILM to an Index Template

Create an index template that applies the ILM policy to all log indices:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-ilm-policy",
      "index.lifecycle.rollover_alias": "logs"
    }
  }
}

Step 3: Performance Tuning for Elasticsearch

Optimize Elasticsearch for write-heavy log ingestion:

# elasticsearch.yml
indices.memory.index_buffer_size: 10%
indices.fielddata.cache.size: 20%
thread_pool.write.queue_size: 10000
thread_pool.search.queue_size: 5000
refresh_interval: 30s
translog.durability: async
translog.sync_interval: 5s

Key changes:

• refresh_interval: 30s -- reduces refresh frequency (default 1s), improving write throughput
• translog.durability: async -- makes translog writes asynchronous, speeding up indexing at the cost of some durability

Step 4: Monitor Cluster Performance

Use Elasticsearch monitoring APIs to track performance:

# Node stats
curl "localhost:9200/_nodes/stats/fs,os,process,jvm,indices"

# Indexing throughput
curl "localhost:9200/_cat/indices?v&s=docs.count:desc"

# Hot threads (CPU bottlenecks)
curl "localhost:9200/_nodes/hot_threads"

For real-time monitoring, enable Metricbeat to collect Elasticsearch metrics, or use Kibana Stack Monitoring to view cluster health, indexing rates, and search latency.

Step 5: Capacity Planning

Estimate storage requirements:

Daily log volume: 100GB
Replication factor: 2 (1 primary + 1 replica)
Total daily storage: 200GB
Retention: 30 days
Total storage needed: 6TB
Overhead (merging, overhead): ~20%
Total: 7.2TB

Shard count guideline: aim for 20-40GB per shard. For 200GB daily volume, use 5-10 primary shards per day.

Common Mistakes

1. Refresh Interval Too Low

Default 1-second refresh interval causes excessive I/O on write-heavy clusters. Increase to 30 seconds or more for log ingestion.

2. Not Using Rollover

Without rollover, indices grow indefinitely. Large indices are slow to search and hard to manage. Always configure rollover in ILM.

3. Hot-Warm Without Allocation Rules

Hot-warm architecture requires node attributes and allocation rules. Without them, data is not distributed to the correct tier.

4. Insufficient Heap Size

Elasticsearch heap should be 50% of system RAM (max 32GB). Too little heap causes frequent GC pauses. Too much wastes memory on OS cache.

5. Ignoring Segment Merging

Without force-merging in the warm phase, segments accumulate and degrade search performance. Use forcemerge in the warm phase ILM action.

Practice Questions

1. What are the four phases of Elasticsearch ILM? Hot (active indexing), warm (less frequent queries), cold (rarely accessed), and delete (removal after retention period).

2. How does the rollover action work in ILM? It creates a new index when the current index exceeds a configurable size or age, keeping each index at a manageable size.

3. What refresh interval is recommended for log ingestion? 30 seconds. This reduces indexing overhead while keeping data searchable within a reasonable delay.

4. What is the recommended shard size for Elasticsearch? 20-40GB per shard. This balances search performance with management overhead.

5. Challenge: Design a production ELK architecture for 1TB daily log ingestion with 60-day hot retention, 180-day warm retention, and 2-year cold retention on inexpensive object storage, specifying node counts, storage tiers, and ILM policies.

What's Next

Your ELK Stack knowledge is complete. Apply these patterns to build a production-grade logging platform for any infrastructure.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.