Filebeat: Lightweight Log Shipper for ELK Stack

In this tutorial, you'll learn about Filebeat: Lightweight Log Shipper for ELK Stack. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Filebeat is a lightweight shipper that installs as an agent on servers, tails log files, and forwards them to Logstash or Elasticsearch, using minimal CPU and memory while providing reliable delivery with backpressure handling.

What You'll Learn

In this tutorial, you will install Filebeat, configure inputs to read log files, set up multiline handling for stack traces, enable modules for common applications, and ship logs to Logstash.

Why It Matters

Logstash is powerful but heavy. Running it on every log-producing server would consume significant resources. Filebeat uses less than 10MB of memory per instance, making it safe to run on every server. It handles log rotation, backpressure, and reliable delivery, ensuring no logs are lost even when the destination is temporarily unavailable.

Real-World Use

Doda Browser runs Filebeat on 200 application servers. Each Filebeat instance tails application logs, Nginx access logs, and system auth logs. Filebeat uses multiline configuration to merge Java stack traces into single events before forwarding to Logstash for Parsing. During a recent network outage, Filebeat queued logs locally and delivered them when connectivity was restored, with zero data loss.

Step 1: Install Filebeat

Download and install Filebeat:

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.14.0-linux-x86_64.tar.gz
tar xvf filebeat-8.14.0-linux-x86_64.tar.gz
cd filebeat-8.14.0-linux-x86_64

Step 2: Configure Filebeat Inputs

Edit filebeat.yml to define log file inputs:

filebeat.inputs:
  - type: filestream
    id: app-logs
    enabled: true
    paths:
      - /var/log/myapp/*.log
    fields:
      service: myapp
      env: production

  - type: filestream
    id: syslog
    enabled: true
    paths:
      - /var/log/syslog

Step 3: Configure Multiline Handling

Stack traces and exception logs span multiple lines. Configure multiline to merge them:

filebeat.inputs:
  - type: filestream
    id: java-logs
    paths:
      - /var/log/myapp/error.log
    multiline:
      type: pattern
      pattern: '^\d{4}-\d{2}-\d{2}'
      negate: true
      match: after

This merges lines that do not start with a date (exception stack traces) with the previous line (the error message).

Step 4: Configure Output to Logstash

Send logs to Logstash for processing:

output.logstash:
  hosts: ["logstash.example.com:5044"]
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/filebeat/ca.crt"]

Alternatively, send directly to Elasticsearch:

output.elasticsearch:
  hosts: ["https://elasticsearch.example.com:9200"]
  username: "filebeat"
  password: "${ES_PASSWORD}"

Step 5: Enable Filebeat Modules

Filebeat includes modules for common applications that auto-configure inputs and processing:

filebeat modules enable nginx system auditd

Modules set up the correct paths, multiline handling, and field mappings for the target application automatically.

Start Filebeat

Run Filebeat as a service:

sudo systemctl start filebeat
sudo systemctl enable filebeat

Verify Filebeat is running and connected:

filebeat status

Common Mistakes

1. Not Configuring Multiline for Stack Traces

Without multiline, Java or Python exception stack traces are sent as individual events, breaking the relationship between the error message and trace lines.

2. Using Wrong Registry Path

Filebeat tracks read state in a registry file. If the registry is lost or corrupted, Filebeat re-reads all files. Persist the registry to a stable location.

3. Too Many Harvesters

Each file input creates a harvester. With thousands of log files, Filebeat may use significant resources. Use wildcard patterns carefully.

4. No SSL Between Filebeat and Logstash

Without TLS, log data (including sensitive information) travels in plain text. Always enable SSL in production.

5. Ignoring Backpressure Settings

When Logstash is slow, Filebeat can overwhelm it or drop events. Configure queue.mem.events and output.logstash.pipeline for proper backpressure handling.

Practice Questions

1. What is Filebeat and why is it used? A lightweight log shipper that collects log files from servers and forwards them to Logstash or Elasticsearch with minimal resource usage.

2. How does Filebeat handle multiline log entries like stack traces? Using the multiline configuration that specifies a pattern, whether to negate the match, and whether to merge lines before or after the matched line.

3. What is a Filebeat module? A preconfigured input and processing setup for common applications like Nginx, Apache, and system logs that simplifies deployment.

4. How does Filebeat ensure reliable log delivery? It tracks read state in a registry file and queues events in memory, retrying delivery on failure without dropping data.

5. Challenge: Configure Filebeat to collect logs from a Java application with multiline stack trace handling, tag them with environment and service fields, and forward to Logstash over SSL.

What's Next

Learn how Elasticsearch clusters distribute data across nodes using shards and Replication for scalability and fault tolerance.

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro.