AWS CloudWatch Log Group Retention Policy Fix

In this tutorial, you'll learn about AWS CloudWatch Log Group Retention Policy Fix. We cover key concepts, practical examples, and best practices to help you understand and apply this topic effectively.

Your CloudWatch log group accumulates logs indefinitely and never expires — no retention policy is set, causing unlimited storage costs and making it hard to find relevant logs.

Step-by-Step Fix

1. Check current retention policy

aws logs describe-log-groups --query 'logGroups[].{name:logGroupName,retention:retentionInDays}'

Expected output:

[
    {"name": "/aws/lambda/my-function", "retention": null},
    {"name": "/aws/ec2/my-instance", "retention": 30}
]

A null retention means logs are stored forever.

2. Set a retention policy

# Wrong: no retention set, logs accumulate forever
# Right: set retention to 30 days for the log group

aws logs put-retention-policy --log-group-name /aws/lambda/my-function --retention-in-days 30

Expected output (no output on success).

3. Bulk set retention for multiple log groups

# Set retention on all log groups without a policy
for group in $(aws logs describe-log-groups --query 'logGroups[?retentionInDays==null].[logGroupName]' --output text); do
    echo "Setting retention for $group"
    aws logs put-retention-policy --log-group-name "$group" --retention-in-days 30
done

4. Delete old log streams

# List log streams older than 90 days
aws logs describe-log-streams \
  --log-group-name /aws/lambda/my-function \
  --order-by LastEventTime \
  --descending \
  --query 'logStreams[?starts_with(logStreamName,`2023`)].logStreamName' \
  --max-items 10

# Delete a specific log stream
aws logs delete-log-stream --log-group-name /aws/lambda/my-function --log-stream-name "2023/01/15/[$LATEST]abc123"

5. Export logs to S3 before deletion

# Export logs to S3 for archival before setting retention
aws logs create-export-task \
  --log-group-name /aws/lambda/my-function \
  --from 1700000000 \
  --to 1700100000 \
  --destination my-log-archive-bucket \
  --destination-prefix lambda-logs/2024-01

6. Automate retention with Terraform

resource "aws_cloudwatch_log_group" "lambda_logs" {
  name              = "/aws/lambda/my-function"
  retention_in_days = 30

  tags = {
    Environment = "production"
    Purpose     = "lambda execution logs"
  }
}

Prevention

• Always set a retention policy when creating a log group (30 days for development, 90 days for production).
• Use AWS Config rules to detect log groups without retention policies.
• Create a monthly cron job to review and clean up old log groups.
• Use S3 lifecycle policies for long-term archival of exported logs.
• Monitor CloudWatch costs in the AWS Cost Explorer dashboard.

Common Mistakes with cloudwatch log group

1. Misunderstanding that String is [Char] with poor performance for large text operations
2. Using foldl instead of foldl' causing stack overflow on large lists
3. Forgetting deriving (Show, Eq) on custom data types needed for debugging

These mistakes appear frequently in real-world AWS code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ