journalctl — Querying Systemd Logs Like a Pro

Q: Can journalctl show logs from multiple machines?

By default, no. Use `journalctl --merge` to merge logs from `/var/log/journal/` across machines if you copy journal files. For centralized logging, use journal-gatewayd or rsyslog.

Q: How do I search for a string in journalctl?

Pipe to grep: `journalctl -u nginx | grep "connection refused"`. Or use `journalctl | grep` for simple patterns.

Q: What is the difference between `-p err` and `-p 0..3`?

Both show priority 0 (emerg) through 3 (err). `-p err` is the shorthand. `-p 0..3` is the explicit range syntax, useful for custom ranges like `-p 2..4`.

Q: Can I forward journald to Elasticsearch?

Yes — install `journald-elasticsearch` or use `journalctl -o json` and pipe to Logstash or Filebeat with a JSON input.

DodaTech Updated 2026-06-24 8 min read

In this tutorial, you'll learn about journalctl. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

journalctl is the command-line tool for querying logs collected by systemd-journald, giving administrators structured, filterable access to all system and service logs in one place.

What You'll Learn

How to use journalctl filters (unit, priority, time range, boot, fields), export logs in JSON and other formats, configure journald for persistent storage with Rate Limiting and size caps, and integrate journald with external syslog.

Why journalctl Matters

Traditional syslog scatters logs across /var/log/ files. journald collects all logs in a structured binary format with consistent metadata — PID, UID, GID, executable path, kernel facility, and priority. This structure lets you query across all services at once: "show all ERROR-level logs from the last hour" or "show everything that happened between two timestamps." Durga Antivirus Pro uses journald fields to tag scan events with request IDs for traceability across Microservices.

Learning Path

flowchart LR
  A[System Rescue] --> B[journalctl Guide
You are here]
  B --> C[Server Hardening CIS]
  B --> D[Monitoring & Logging]
  style B fill:#f90,color:#fff

Basic journalctl Usage

# View all logs (oldest first)
journalctl

# Follow new logs in real time
journalctl -f

# Show last 50 lines
journalctl -n 50

# Show last 10 minutes of logs
journalctl --since "10 minutes ago"

# Show logs without pager
journalctl --no-pager -n 100

Expected output:

Jun 24 10:00:01 server kernel: Linux version 6.2.0-26-generic ...
Jun 24 10:00:02 server systemd[1]: Starting Journal Service...
Jun 24 10:00:02 server systemd-journald[345]: Runtime Journal is using 4.0M
Jun 24 10:00:05 server sshd[789]: Server listening on 0.0.0.0 port 22.
Jun 24 10:00:06 server nginx[1234]: Starting nginx.

Filtering by Service Unit

# All logs for a specific service
journalctl -u nginx

# Multiple units
journalctl -u nginx -u postgresql

# Follow a specific service
journalctl -u myapp -f

# Combine unit and priority
journalctl -u sshd -p err -n 20

Filtering by Priority

# Emergency (0) — system unusable
journalctl -p emerg

# Alert (1) — immediate action needed
journalctl -p alert

# Critical (2), Error (3), Warning (4)
journalctl -p err          # Shows 0-3 (emerg, alert, crit, err)
journalctl -p warning      # Shows 0-4

# Notice (5), Info (6), Debug (7)
journalctl -p info         # All info and above

# Custom range (e.g., err to emerg)
journalctl -p 0..3

Filtering by Time

# Specific time
journalctl --since "2026-06-24 10:00:00"

# Relative time
journalctl --since "1 hour ago"
journalctl --since "yesterday" --until "today"

# Time range
journalctl --since "2026-06-24 00:00:00" --until "2026-06-24 23:59:59"

# From last boot only
journalctl -b

# From specific boot
journalctl --list-boots
journalctl -b -1           # Previous boot
journalctl -b -2           # Two boots ago

# Since last boot
journalctl --since "30 minutes ago" -b

Expected journalctl --list-boots:

-2 abc123... Wed 2026-06-22 08:00:00 — Wed 2026-06-23 18:30:00
-1 def456... Wed 2026-06-23 18:30:00 — Thu 2026-06-24 06:00:00
 0 ghi789... Thu 2026-06-24 06:00:00 — still running

Filtering by Fields

journald logs have structured fields that you can query directly:

# Show all available fields
journalctl --fields

# Filter by specific field
journalctl _PID=1234
journalctl _UID=1000
journalctl _SYSTEMD_UNIT=nginx.service
journalctl _COMM=sshd

# Combine fields
journalctl _PID=1234 + _PID=5678

# Exclude a field
journalctl -u nginx _PID=!1

Common Fields

Field	Description
`_PID`	Process ID
`_UID`	User ID
`_GID`	Group ID
`_COMM`	Command name
`_EXE`	Executable path
`_SYSTEMD_UNIT`	Systemd unit name
`_BOOT_ID`	Unique boot identifier
`_MACHINE_ID`	Machine identifier
`_HOSTNAME`	Hostname
`PRIORITY`	0 (emerg) through 7 (debug)
`SYSLOG_FACILITY`	Syslog facility code

Output Formats

# Verbose (shows all fields)
journalctl -u nginx -o verbose

# JSON (for programmatic access)
journalctl -u nginx -o json

# JSON with metadata
journalctl -u nginx -o json-pretty

# Short (default, syslog-compatible)
journalctl -u nginx -o short

# Short with monotonic timestamp
journalctl -u nginx -o short-monotonic

# Export (binary, for transfer to another system)
journalctl -u nginx -o export > nginx-logs.journal

Expected JSON output:

{
  "__CURSOR" : "s=abc...;",
  "__REALTIME_TIMESTAMP" : "1719237600000000",
  "_BOOT_ID" : "ghi789...",
  "_MACHINE_ID" : "abc123...",
  "_HOSTNAME" : "server01",
  "PRIORITY" : "6",
  "_UID" : "0",
  "_GID" : "0",
  "_COMM" : "nginx",
  "_EXE" : "/usr/sbin/nginx",
  "_SYSTEMD_UNIT" : "nginx.service",
  "MESSAGE" : "started"
}

Advanced Queries

# Kernel messages only
journalctl -k

# Show messages matching a pattern
journalctl -u nginx | grep "error"

# Use jq for JSON querying
journalctl -u nginx -o json | jq '.MESSAGE'

# Show log message count per service
journalctl --output=short-unix --no-pager | \
    awk '{print $5}' | sort | uniq -c | sort -rn | head -10

# Show disk usage by unit
journalctl --disk-usage

Log Size Management

# Current disk usage
journalctl --disk-usage

# Vacuum by size (keep only 500MB)
sudo journalctl --vacuum-size=500M

# Vacuum by time (keep last 2 weeks)
sudo journalctl --vacuum-time=2weeks

# Vacuum by file count (keep 5 most recent)
sudo journalctl --vacuum-files=5

# Rotate journal files immediately
sudo journalctl --rotate

Journald Configuration

# /etc/systemd/journald.conf
[Journal]
# Storage options: auto (default), persistent, volatile, none
Storage=persistent

# Compression
Compress=yes
Seal=yes

# Size limits
SystemMaxUse=4G
SystemKeepFree=1G
MaxFileSec=1month

# Rate limiting (prevent log floods)
RateLimitIntervalSec=30s
RateLimitBurst=10000

# Sync interval
SyncIntervalSec=5m

# Forward to syslog
ForwardToSyslog=no
ForwardToWall=yes

# Max log entry size (default: 10K, increase for debug logs)
MaxLevelStore=debug

After changing the config:

sudo systemctl restart systemd-journald

Integration with External Logging

# Forward journald to syslog-ng or rsyslog
sudo apt install rsyslog

# journald sends to syslog by default when ForwardToSyslog=yes
# rsyslog can then forward to remote servers
sudo tee /etc/rsyslog.d/remote.conf << 'EOF'
*.* @logserver.example.com:514
EOF

sudo systemctl restart rsyslog

Journal Gateway (HTTP API)

For remote log access without SSH:

# Install journal-gateway
sudo apt install systemd-journal-remote

# Enable the gateway service
sudo systemctl enable --now systemd-journal-gatewayd

# Access via HTTP
# http://server:19531/browse

Common Errors

1. "No entries" When Logs Should Exist

The service may use syslog instead of journald. Check the service unit: ensure StandardOutput=journal and StandardError=journal. Alternatively, the service wrote logs before journald started — check journalctl -b for the correct boot.

2. Journal Files Corrupted

If journal files become corrupted (power loss, disk full), journald recreates them. You can manually remove corrupt files: sudo rm /var/log/journal/*/system@*.journal then sudo systemctl restart systemd-journald.

3. Logs Missing After Reboot (Volatile Storage)

If Storage=auto and /var/log/journal does not exist, logs are stored in /run/log/journal (volatile). Create the directory: sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald.

4. journalctl --vacuum-size Not Freeing Space

Vacuum only removes archived (inactive) journals, not the currently active one. Use --rotate first to archive the active journal, then vacuum.

5. Rate Limiting Drops Legitimate Logs

If a service logs heavily, Rate Limiting drops entries. Increase RateLimitIntervalSec and RateLimitBurst in journald.conf, or set RateLimitInterval=0 to disable (not recommended).

6. journalctl Performance Slow on Large Systems

Querying months of logs across many services is slow. Use precise filters (--since, -u, -p) to narrow the search. Consider setting SystemMaxUse to a reasonable size.

7. JSON Output Truncates Long Lines

The --no-pager flag prevents truncation. Use journalctl -o json --no-pager | jq for complete JSON records.

Practice Questions

1. How do you show all ERROR-level messages from the last boot? journalctl -b -p err — shows priority 0 (emerg) through 3 (err) for the current boot.

2. What command streams new Nginx error logs in real time? journalctl -u nginx -f -p err — follows new entries at error level and above.

3. How do you save logs from a specific time range to a file? journalctl --since "2026-06-24 10:00" --until "2026-06-24 12:00" > logs.txt

4. What is the difference between Storage=persistent and Storage=volatile? persistent stores logs in /var/log/journal/ (survives reboots). volatile stores in /run/log/journal/ (lost on reboot).

5. How do you check disk usage of journal logs? journalctl --disk-usage shows total disk space consumed by journal files.

Challenge: Write a script that runs daily via a systemd timer. The script should: (1) export all ERROR-level logs from the past 24 hours to a JSON file, (2) parse the JSON with jq to count errors per service, (3) rotate journal files, (4) vacuum logs older than 30 days, (5) log the report to journald with a custom MY_REPORT field. Test the script works correctly.

Can journalctl show logs from multiple machines?

By default, no. Use journalctl --merge to merge logs from /var/log/journal/ across machines if you copy journal files. For centralized logging, use journal-gatewayd or rsyslog.

How do I search for a string in journalctl?

Pipe to grep: journalctl -u nginx | grep "connection refused". Or use journalctl | grep for simple patterns.

What is the difference between `-p err` and `-p 0..3`?

Both show priority 0 (emerg) through 3 (err). -p err is the shorthand. -p 0..3 is the explicit range syntax, useful for custom ranges like -p 2..4.

Does journalctl slow down the system?

journald is optimized for performance. Writes use memory-mapped files and are extremely fast. Queries can be slow on millions of entries, but proper filtering keeps them efficient.

Can I forward journald to Elasticsearch?

Yes — install journald-elasticsearch or use journalctl -o json and pipe to Logstash or Filebeat with a JSON input.

What's Next

Linux Server Hardening — CIS Benchmarks

Linux Containers — LXC & LXD Guide

Monitoring & Logging

Built by the developers of Doda Browser, DodaZIP, and Durga Antivirus Pro. Updated 2026-06-24.

← Previous Linux System Rescue & Recovery — Boot Repair, Data Recovery & Emergency Procedures Next → Linux Server Hardening — CIS Benchmarks & Security Best Practices

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro

Home Browse Linux Administration