Elasticsearch: Indexing, Mapping & Query DSL

In this tutorial, you'll learn about Elasticsearch: Indexing, Mapping & Query DSL. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Elasticsearch is a distributed RESTful search engine that stores data as JSON documents in indexes, uses mappings to define field types, and provides a powerful Query DSL for searching and aggregating data at scale.

What You'll Learn

In this tutorial, you will create and manage indexes, define mappings with explicit field types, write queries using the Elasticsearch Query DSL, and use aggregations to analyze log data.

Why It Matters

Elasticsearch is the foundation of the ELK Stack. Understanding how it indexes data, how mappings control field analysis, and how queries retrieve data is essential for building fast, accurate log search. Poor mapping choices lead to incorrect search results. Bad queries cause slow dashboards. Mastering these fundamentals ensures your logging platform performs well.

Real-World Use

Durga Antivirus Pro indexes scan events in Elasticsearch. Each event includes fields like scan_duration_ms, threat_name, file_path, and verdict. Explicit mappings ensure scan_duration_ms is indexed as a long (not text) so numerical aggregations work correctly. The operations team uses range queries to find scans that took longer than 30 seconds.

Indexing Documents

Index a document by sending a PUT or POST request to the Elasticsearch REST API:

curl -X POST "localhost:9200/my-index/_doc/" -H 'Content-Type: application/json' -d'
{
  "@timestamp": "2026-06-21T10:00:00Z",
  "message": "User login successful",
  "user": "alice",
  "status": 200,
  "duration_ms": 45
}'

Expected response:

{
  "_index": "my-index",
  "_id": "abc123",
  "_version": 1,
  "result": "created"
}

Dynamic vs Explicit Mapping

By default, Elasticsearch uses dynamic mapping -- it detects field types automatically:

• Strings become text (full-text search) with a keyword sub-field
• Numbers become long or double
• Dates are detected from common date formats

For production use, define explicit mappings to control field types and analysis:

curl -X PUT "localhost:9200/logs" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "message": {
        "type": "text",
        "analyzer": "standard"
      },
      "status": { "type": "integer" },
      "duration_ms": { "type": "long" },
      "user": {
        "type": "keyword"
      }
    }
  }
}'

Query DSL

The Query DSL is a JSON-based query language. Here is a match query that searches the message field:

curl -X GET "localhost:9200/logs/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "message": "login failed"
    }
  }
}'

For structured filters, use a term query (exact match on keyword fields):

curl -X GET "localhost:9200/logs/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "term": {
      "status": 403
    }
  }
}'

Aggregations

Aggregations analyze data across documents. This query calculates average scan duration by user:

curl -X GET "localhost:9200/logs/_search" -H 'Content-Type: application/json' -d'
{
  "size": 0,
  "aggs": {
    "by_user": {
      "terms": { "field": "user.keyword" },
      "aggs": {
        "avg_duration": {
          "avg": { "field": "duration_ms" }
        }
      }
    }
  }
}'

Common Mistakes

1. Using text Instead of keyword for Exact Fields

Fields like status codes, hostnames, and log levels should be keyword for exact-match filtering and aggregations.

2. No Index Template

Index templates automatically apply mappings to new indices matching a pattern. Without them, every new index uses dynamic mapping, causing inconsistent field types.

3. Overly Broad Queries

Using match on large text fields searches the entire inverted index. Add filters to narrow the document set before the query runs.

4. Not Setting shards and replicas

Default settings may not match your workload. Use 1 shard per 20GB of data and at least 1 replica for production.

5. Ignoring the _source Field

By default Elasticsearch stores the original JSON document in _source. Disabling it saves disk space but removes the ability to retrieve original log data.

Practice Questions

1. What is the difference between text and keyword field types? text is analyzed for full-text search. keyword is stored as-is for exact matches, aggregations, and sorting.

2. How do you define an explicit mapping in Elasticsearch? Use the PUT API with a mappings.properties block that defines each field name and its type.

3. What is the difference between a match query and a term query? match analyzes the input text and searches the inverted index. term searches for an exact value in a keyword field.

4. What are aggregations used for? Aggregations compute metrics (avg, sum, max) or group documents (terms, date_histogram) across search results for analytics.

5. Challenge: Create an index with explicit mappings for an application log that includes timestamp, log level, service name, request path, response status, and duration. Write a query that finds the top 5 slowest endpoints for 500 errors.

What's Next

Build Logstash pipelines to Process and transform your log data before it reaches Elasticsearch.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.