Skip to content

Elasticsearch & Kibana — Log Analysis & Visualization Guide

DodaTech Updated 2026-06-24 6 min read

In this tutorial, you'll learn about Elasticsearch & Kibana. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Elasticsearch is a distributed search and analytics engine that stores, searches, and analyzes large volumes of data in near real time. Kibana is its visualization layer for exploring data, building dashboards, and managing the Elastic Stack.

What You'll Learn

Why It Matters

Logs grow exponentially with infrastructure scale. Without a centralized search platform, finding a specific error across 500 servers requires SSH-ing into each machine. Elasticsearch indexes logs so you can search terabytes of data in milliseconds. DodaTech's Elasticsearch cluster processes 5TB of logs daily, enabling the SRE team to find and fix issues in minutes rather than hours.

Real-World Use

When DodaZIP's API error rate spiked, the SRE team used Kibana Discover to search status:500 AND service:user-service across the last hour, filtered by pod name, and identified a new deployment with a missing environment variable — all in under 30 seconds.

flowchart TD
    A[Log Sources] --> B[Fluentd / Filebeat]
    B --> C[Elasticsearch Cluster]
    C --> D[Node 1: Master]
    C --> E[Node 2: Data]
    C --> F[Node 3: Data]
    C --> G[Node 4: Ingest]
    D --> H[Kibana]
    H --> I[Dashboards]
    H --> J[Discover]
    H --> K[Alerting]
    C --> L[Index Lifecycle Management]
    L --> M[HOT -> WARM -> COLD -> DELETE]
    style C fill:#005571,color:#fff
    style H fill:#005571,color:#fff
ℹī¸ Info

Prerequisites: Basic Linux administration. Understanding of Fluentd or Logstash for log shipping.

Installation

# Import Elastic GPG key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

# Install Elasticsearch
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update && sudo apt install elasticsearch

# Configure
sudo sed -i 's/#cluster.name: my-application/cluster.name: dodatech-logs/' /etc/elasticsearch/elasticsearch.yml
sudo sed -i 's/#node.name: node-1/node.name: es-01/' /etc/elasticsearch/elasticsearch.yml
sudo sed -i 's/#network.host: 192.168.0.1/network.host: 0.0.0.0/' /etc/elasticsearch/elasticsearch.yml

# Start Elasticsearch
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

# Verify
curl http://localhost:9200

# Expected output:
# {
#   "name" : "es-01",
#   "cluster_name" : "dodatech-logs",
#   "cluster_uuid" : "abc123",
#   "version" : {
#     "number" : "8.14.0",
#     "build_flavor" : "default",
#     "build_type" : "deb",
#     "lucene_version" : "9.10.0"
#   },
#   "tagline" : "You Know, for Search"
# }

Cluster Configuration

# /etc/elasticsearch/elasticsearch.yml
cluster.name: dodatech-logs
node.name: es-01
node.roles: [master, data, ingest]

path.data: /data/elasticsearch
path.logs: /var/log/elasticsearch

network.host: 0.0.0.0
http.port: 9200
transport.port: 9300

discovery.seed_hosts: ["es-01.dodatech.com", "es-02.dodatech.com", "es-03.dodatech.com"]
cluster.initial_master_nodes: ["es-01", "es-02", "es-03"]

gateway.recover_after_nodes: 2

indices.lifecycle.poll_interval: 10m

indices.fielddata.cache.size: 20%

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/elastic-certificates.p12
xpack.monitoring.collection.enabled: true

Index Management

// Create an index template for application logs
PUT _index_template/dodazip-logs
{
  "index_patterns": ["dodazip-logs-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "dodatech-log-policy",
      "index.lifecycle.rollover_alias": "dodazip-logs"
    },
    "mappings": {
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "namespace": { "type": "keyword" },
        "pod": { "type": "keyword" },
        "hostname": { "type": "keyword" },
        "status": { "type": "integer" },
        "duration_ms": { "type": "integer" },
        "error_stack": { "type": "text", "index": false }
      }
    }
  }
}

// Create an index lifecycle policy
PUT _ilm/policy/dodatech-log-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "allocate": {
            "number_of_replicas": 1,
            "require": { "data_tier": "warm" }
          },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "60d",
        "actions": {
          "allocate": {
            "number_of_replicas": 0,
            "require": { "data_tier": "cold" }
          },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Kibana Installation

# Install Kibana
sudo apt install kibana

# Configure
sudo tee -a /etc/kibana/kibana.yml << 'EOF'
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
kibana.index: ".kibana"
logging.dest: /var/log/kibana/kibana.log
EOF

# Start Kibana
sudo systemctl enable kibana
sudo systemctl start kibana

# Verify
curl http://localhost:5601/api/status

# Expected output:
# {"version":{"number":"8.14.0"},"status":{"overall":{"level":"available"}}

Searching Logs

// Kibana Query Language (KQL) examples
// In Kibana Discover:

// Basic search
status:500

// Phrase search
message: "OutOfMemoryError"

// Boolean
level: error AND service: user-service

// Range
duration_ms > 1000 AND duration_ms < 5000

// Wildcard
hostname: web-*

// Existence
_exists_: error_stack

// Negation
NOT level: debug

// Nested field
kubernetes.pod.name: dodazip-*

// Lucene syntax
// Regex
message: /error\s.*timeout/i

// Fuzzy
hostname: web-01~

// Proximity
"database connection"~5

Kibana Dashboard

// Sample saved dashboard structure (partial)
{
  "attributes": {
    "title": "DodaTech API Overview",
    "panels": [
      {
        "panelIndex": "1",
        "panelReference": {
          "type": "visualization",
          "id": "error-rate-over-time]
        },
        "gridData": { "x": 0, "y": 0, "w": 24, "h": 8 }
      },
      {
        "panelIndex": "2",
        "panelReference": {
          "type": "visualization",
          "id": "top-error-endpoints"
        },
        "gridData": { "x": 0, "y": 8, "w": 12, "h": 8 }
      }
    ]
  }
}

Alerting

// Create a Kibana alert rule
POST api/alerting/rule
{
  "name": "High Error Rate",
  "ruleTypeId": ".es-query",
  "consumer": "alerts",
  "schedule": { "interval": "5m" },
  "actions": [
    {
      "group": "query matched",
      "id": "pagerduty-connector",
      "params": {
        "routingKey": "YOUR_PAGERDUTY_KEY",
        "eventAction": "trigger",
        "payload": {
          "summary": "Error rate exceeded 5% in the last 5 minutes",
          "severity": "critical]
        }
      }
    },
    {
      "group": "query matched",
      "id": "slack-connector",
      "params": {
        "message": "⚠ī¸ High error rate alert: {{context.value}}% errors in last 5 minutes"
      }
    }
  ],
  "params": {
    "index": ["dodazip-logs-*"],
    "timeField": "@timestamp",
    "timeWindowSize": 5,
    "timeWindowUnit": "m",
    "threshold": [5],
    "thresholdComparator": ">=",
    "aggType": "avg",
    "aggField": "error_rate",
    "termField": "service",
    "termSize": 10,
    "esQuery": "{\"query\":{\"bool\":{\"filter\":[{\"range\":{\"@timestamp\":{\"gte\":\"now-5m\"}},{\"term\":{\"level\":\"error\"}}]}}"
  }
}

Common Configuration Mistakes

  1. Not configuring index lifecycle management: Without ILM, indices grow unbounded, consuming all disk space. Configure a policy that rolls over, moves to warm/cold tiers, and deletes old data.

  2. Over-Sharding indices: Too many shards per node degrades cluster performance. Aim for 20-40GB per shard and limit shards to <1,000 per node.

  3. Running Kibana on the same node as Elasticsearch: Kibana's background tasks (reporting, alerting) compete with Elasticsearch for resources. Deploy Kibana on separate nodes.

  4. Ignoring refresh_interval for bulk indexing: Default refresh (1s) creates small segments during bulk loads. Set refresh_interval: -1 during large log backfills and restore to 10s after.

  5. Not setting up snapshot repositories: Without snapshots, accidental index deletion or cluster failure causes permanent data loss. Configure an S3 or filesystem snapshot Repository with daily snapshots.

Practice Questions

  1. What is an inverted index in Elasticsearch? Answer: Elasticsearch stores terms mapped to document IDs, enabling instant full-text search. Each field is indexed according to its mapping type (keyword, text, integer, date).

  2. How does index lifecycle management (ILM) work? Answer: ILM automatically transitions indices through hot (writable, high-performance), warm (read-only, reduced replicas), cold (searchable, no replicas), and delete phases based on age or size.

  3. What is the difference between keyword and text field types? Answer: keyword fields are exact-match, used for filtering and aggregations. text fields are analyzed for full-text search. A field can have both via multi-fields.

  4. How does Kibana alerting integrate with external services? Answer: Kibana connectors support PagerDuty, Slack, email, Jira, Webhook, and other services. Alert rules trigger actions through connectors when query conditions are met.

Challenge

Deploy a production Elasticsearch and Kibana stack: set up a 3-node Elasticsearch cluster with dedicated master, data, and ingest node roles, configure TLS for transport and HTTP, create an ILM policy with hot/warm/cold/delete phases, create index templates for structured log data, set up Filebeat or Fluentd to ship application logs, configure Kibana data views and alerts for error rate thresholds, and automate snapshot backups to S3.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.

← Previous Fluentd — Log Collection & Aggregation Guide Next → Jaeger — Distributed Tracing Guide

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Devops Tools