Kubernetes Pod Evicted Due to Resource Pressure Fix

In this tutorial, you'll learn about Kubernetes Pod Evicted Due to Resource Pressure Fix. We cover key concepts, practical examples, and best practices.

The Problem

Your pod shows Evicted status and disappears from the cluster:

NAME                     READY   STATUS    RESTARTS   AGE
my-app-6b4c9f8d7-abc12   0/1     Evicted   0          10s

Running kubectl get pods shows the old evicted pod alongside a new one that may also get evicted. The kubelet evicts pods when a node runs out of memory, disk space, or PID resources.

Quick Fix

Step 1: Identify why the pod was evicted

kubectl describe pod my-app-6b4c9f8d7-abc12

Check the Status and Events sections:

Status Reason: Evicted
Message: The node was low on resource: memory.

Common eviction reasons:

• The node was low on resource: memory. -- memory pressure
• The node was low on resource: ephemeral-storage. -- disk pressure
• The node had condition: DiskPressure. -- node disk is full

Step 2: Check node conditions

kubectl describe nodes

Look for pressure conditions:

Conditions:
  Type              Status  LastHeartbeatTime
  MemoryPressure    True    ...
  DiskPressure      True    ...
  PIDPressure       False   ...

Any True pressure condition means the kubelet will evict more pods. Fix the node first, not just the pod.

Step 3: Free node resources

If DiskPressure is true, clean up unused resources on the node:

# SSH into the node
ssh user@node-ip

# Check disk usage
df -h /var/lib/docker
df -h /var/lib/kubelet

# Remove unused images
docker image prune -a -f

# Remove stopped containers
docker container prune -f

For memory pressure, reduce pod resource requests or add more nodes.

Step 4: Set resource limits and requests

Pods without limits are the first to be evicted:

WRONG -- no resource limits:

resources: {}

RIGHT -- set requests and limits:

resources:
  requests:
    memory: "128Mi"
    cpu: "100m"
  limits:
    memory: "256Mi"
    cpu: "200m"

Step 5: Use pod priority and preemption

Critical pods should have higher priority to survive eviction:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000
globalDefault: false
description: "High priority for critical pods"

Reference it in your pod spec:

priorityClassName: high-priority

Step 6: Add resource reservations for system pods

Configure kubelet reserve resources for system daemons:

# On the node, edit /var/lib/kubelet/config.yaml
kubeletReserved:
  memory: "256Mi"
  cpu: "200m"
  ephemeral-storage: "1Gi"

Then restart kubelet:

systemctl restart kubelet

Use DodaTech's Cluster Monitor to track node pressure trends and set up alerts before eviction happens.

Prevention

• Set resource requests and limits on every pod.
• Monitor node pressure metrics with kubectl top nodes.
• Use PriorityClass to protect critical workloads.
• Configure kubelet reserved resources for system daemons.
• Add autoscaling to add nodes under pressure.
• Clean up unused container images and volumes regularly.

Common Mistakes with pod evicted

1. Using return to exit a function early instead of wrapping a pure value in the monad
2. Mixing let bindings with <- bindings in do notation, producing type errors
3. Overlapping type class instances that cause GHC to reject the program with ambiguous dispatch errors

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ