Log Rotation with Cron — Automated Log Management and Retention

In this tutorial, you will learn about Log Rotation with Cron. We cover key concepts, practical examples, and best practices to help you master this topic.

Learn log rotation with cron: schedule log rotation for application servers, database logs, and system logs using custom cron scripts that handle size-based rotation, compression, retention policies, and growth monitoring.

What You Learn

You will learn to implement log rotation with cron: size-based and time-based rotation policies, gzip compression of rotated logs, retention-based cleanup, log growth monitoring, and alerting on abnormal log volume.

Why It Matters

Logs are the most common cause of disk-full incidents. A single application that logs 1 GB per day fills a 100 GB partition in 100 days. Without rotation, logs consume all available disk space and cause application failures.

Real-World Use

DodaTech runs a log rotation cron job every hour. It rotates any log file larger than 100 MB, compresses rotated files with gzip, deletes logs older than 30 days, and alerts if any application logs more than 500 MB per day.

Log Rotation Script

#!/bin/bash
# log-rotate.sh — Custom log rotation with cron
LOG_DIRS=("/var/log/app" "/var/log/nginx" "/var/log/postgresql")
MAX_SIZE_MB=100
RETENTION_DAYS=30
COMPRESS=true

for dir in "${LOG_DIRS[@]}"; do
    if [ ! -d "$dir" ]; then
        continue
    fi

    find "$dir" -name "*.log" -type f | while read logfile; do
        size_mb=$(du -m "$logfile" | cut -f1)
        if [ "$size_mb" -gt "$MAX_SIZE_MB" ]; then
            timestamp=$(date +%Y%m%d_%H%M%S)
            rotated="${logfile}.${timestamp}"
            mv "$logfile" "$rotated"
            echo "[$(date)] Rotated: $logfile ($size_mb MB -> $rotated)"

            if [ "$COMPRESS" = true ]; then
                gzip "$rotated"
                echo "[$(date)] Compressed: ${rotated}.gz"
            fi

            touch "$logfile"
        fi
    done

    # Remove old rotated logs
    find "$dir" -name "*.log.*" -type f -mtime +$RETENTION_DAYS -delete 2>/dev/null
    find "$dir" -name "*.log.*.gz" -type f -mtime +$RETENTION_DAYS -delete 2>/dev/null
done

import os
import gzip
import time
import shutil
from datetime import datetime

class LogRotator:
    def __init__(self, max_size_mb=100, retention_days=30, compress=True):
        self.max_size_bytes = max_size_mb * 1024 * 1024
        self.retention_seconds = retention_days * 86400
        self.compress = compress

    def rotate_log(self, log_path):
        if not os.path.exists(log_path):
            return False

        size = os.path.getsize(log_path)
        if size < self.max_size_bytes:
            return False

        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
        rotated_path = f"{log_path}.{timestamp}"

        os.rename(log_path, rotated_path)
        size_mb = size / (1024 * 1024)
        print(f"Rotated: {log_path} ({size_mb:.0f} MB -> {rotated_path})")

        if self.compress:
            with open(rotated_path, 'rb') as f_in:
                with gzip.open(f"{rotated_path}.gz", 'wb') as f_out:
                    shutil.copyfileobj(f_in, f_out)
            os.remove(rotated_path)
            print(f"Compressed: {rotated_path}.gz")

        open(log_path, 'w').close()
        return True

    def cleanup_old_logs(self, directory):
        now = time.time()
        removed = 0
        for f in os.listdir(directory):
            fpath = os.path.join(directory, f)
            if os.path.isfile(fpath) and (f.endswith('.gz') or '.log.' in f):
                if now - os.path.getmtime(fpath) > self.retention_seconds:
                    os.remove(fpath)
                    removed += 1
        print(f"Cleaned {removed} old log files from {directory}")

        # Simulate rotation
rotator = LogRotator(max_size_mb=1, retention_days=30)

test_log = "/tmp/test_app.log"
with open(test_log, 'w') as f:
    f.write("x" * (2 * 1024 * 1024))  # 2 MB

rotator.rotate_log(test_log)
rotator.cleanup_old_logs("/tmp")

Expected output:

Rotated: /tmp/test_app.log (2 MB -> /tmp/test_app.log.20260628_000000)
Compressed: /tmp/test_app.log.20260628_000000.gz
Cleaned 1 old log files from /tmp

Log Growth Monitoring

import os
import time
from collections import defaultdict

class LogGrowthMonitor:
    def __init__(self, log_dir, daily_limit_mb=500):
        self.log_dir = log_dir
        self.daily_limit_bytes = daily_limit_mb * 1024 * 1024
        self.previous_sizes = {}

    def measure_growth(self):
        current_sizes = {}
        total_growth = 0

        for root, dirs, files in os.walk(self.log_dir):
            for f in files:
                fpath = os.path.join(root, f)
                try:
                    current_sizes[fpath] = os.path.getsize(fpath)
                except OSError:
                    continue

        for path, size in current_sizes.items():
            prev = self.previous_sizes.get(path, 0)
            growth = size - prev
            if growth > 0:
                total_growth += growth

        self.previous_sizes = current_sizes
        growth_mb = total_growth / (1024 * 1024)

        if growth_mb > (self.daily_limit_bytes / (1024 * 1024)):
            print(f"ALERT: Log growth {growth_mb:.0f} MB exceeds daily limit of {self.daily_limit_bytes / (1024*1024):.0f} MB")

        return growth_mb

monitor = LogGrowthMonitor("/var/log/app", daily_limit_mb=500)
growth = monitor.measure_growth()
print(f"Log growth since last check: {growth:.1f} MB")

Expected output:

Log growth since last check: 0.0 MB

Common Mistakes

1. Not Rotating Frequently Enough

A log rotation cron that runs daily may allow logs to grow to 10+ GB between rotations. Run rotation every hour for high-volume logs. Size-based rotation (rotate when > 100 MB) is more reliable than time-based.

2. Deleting Logs Without Reading Them

A retention policy of 7 days means you lose forensic data after a week. Archive important logs to long-term storage before deletion. Use tiered retention: 7 days on disk, 30 days in warm storage, 1 year in cold storage.

3. Not Compressing Rotated Logs

Text logs compress 80-90% with gzip. Uncompressed rotated logs consume 10x more disk than necessary. Always compress rotated logs with gzip. Use pigz for parallel compression on multi-core systems.

4. Rotating Open Files Without Signal

Rotating a log file while a process has it open does not free disk space until the process closes the file handle. Send the appropriate signal (SIGHUP for most daemons, USR1 for nginx) after rotation to trigger file handle reopening.

5. No Alerting on Log Volume Spikes

A sudden increase in log volume may indicate an attack, a bug, or misconfiguration. Monitor log growth rate per application and alert on abnormal increases (e.g., 5x normal volume).

Practice Questions

1. What is the difference between size-based and time-based log rotation?

Size-based rotation triggers when a log file exceeds a threshold (e.g., 100 MB). Time-based rotation triggers at a fixed interval (e.g., daily). Size-based is more reliable for preventing disk-full incidents.

2. How do you handle open log files during rotation?

Move the log file, create a new empty file, then send SIGHUP to the logging process. The process closes the old file handle and opens the new file. Common signals: SIGHUP for syslog/USR1 for nginx, logrotate uses copytruncate as fallback.

3. What compression should you use for rotated logs?

gzip provides the best balance of compression ratio and speed. For very large logs, use zstd (faster compression/decompression) or xz (better compression ratio, slower). Test with your actual log data.

4. How long should you retain logs?

Depends on requirements: security/audit: 1-7 years, troubleshooting: 30-90 days, operational monitoring: 7-14 days. Use tiered retention: fast storage for recent logs, slow storage for archives.

Challenge

Build a log management system with cron: (1) rotation script that runs hourly, rotates logs > 100 MB, handles open files with SIGHUP, compresses with gzip, (2) tiered retention: 7 days on local disk, 30 days in S3, 1 year in Glacier, (3) growth monitor that tracks per-application log volume and alerts on 5x normal increase, (4) structured log Parsing: extract and index error rates, request counts, and latency percentiles from rotated logs, (5) dashboard showing log volume by application, growth trends, and retention usage.

FAQ

Mini Project: Log Rotation System

Build a cron-based log rotation system: (1) rotation script: rotates logs > 100 MB (configurable), handles open files with SIGHUP, compresses with gzip level 6, (2) tiered retention: 7 days on disk (fast access), 7-30 days in S3 (warm), 30-365 days in Glacier (cold), (3) growth monitor: tracks daily log volume per application, alerts on 5x normal increase, (4) rotation monitoring: tracks success/failure per rotation, file size before/after, compression ratio, (5) Prometheus metrics: log file size, rotated count, bytes saved by compression, retention Compliance.

What's Next

Now that you understand log rotation with cron, explore cache warming strategies with cron, then learn about automated report generation.