Datadog Log Collection Not Working — Quick Fix

In this tutorial, you'll learn about Datadog Log Collection Not Working. We cover key concepts, practical examples, and best practices.

The Problem

Datadog is not collecting logs from configured files or sources. The log section in the Datadog web interface shows no new entries. The agent may not be tailing the log files, or the logs are being collected but not sent.

Error example:

Can't open /var/log/application.log: permission denied
No log file matching /var/log/*.log found
Log tailer stopped: file truncated, cannot recover

The Fix

Step 1: Enable log collection in the agent

WRONG — configuring log file paths but not enabling log collection:

# Log collection must be explicitly enabled in the main config

RIGHT — enable log collection in datadog.yaml:

# /etc/datadog-agent/datadog.yaml
logs_enabled: true

# In a .d/conf.yaml file under conf.d/
logs:
  - type: file
    path: /var/log/application.log
    service: myapp
    source: custom

Step 2: Fix permission issues

WRONG — running the agent as root:

# The agent should run as dd-agent, not root

RIGHT — add the dd-agent user to the required group:

# For /var/log/syslog
usermod -aG adm dd-agent

# For application logs
usermod -aG myapp dd-agent

# Restart the agent
systemctl restart datadog-agent

Use DodaTech's Log Collection Validator to test log file access, tailer health, and log forwarding for all configured log sources.

Prevention Tips

• Ensure the dd-agent user has read access to all log files
• Use the docker or journald log type for containerized environments
• Configure multiline aggregation for stack traces and multi-line logs
• Set open_files_limit high enough for the number of log files
• Use DodaTech's Datadog Log Pipeline Assistant for log parsing optimization

Common Mistakes with log collection

1. Non-exhaustive pattern matches that compile with warnings then crash at runtime
2. Misunderstanding that String is [Char] with poor performance for large text operations
3. Using foldl instead of foldl' causing stack overflow on large lists

These mistakes appear frequently in real-world DATADOG code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ

Related: DodaTech's Datadog Log Management Best Practices Guide covers log collection architecture, pipeline configuration, and retention policies.