Kubernetes Node Not Ready Fix

In this tutorial, you'll learn about Kubernetes Node Not Ready Fix. We cover key concepts, practical examples, and best practices.

The Problem

One or more nodes show NotReady status:

NAME       STATUS     ROLES    AGE   VERSION
node-1     NotReady   <none>   5d    v1.28.0
node-2     Ready      <none>   5d    v1.28.0

Pods on the NotReady node are rescheduled to other nodes, and the node cannot accept new workloads. The control plane marks it NotReady when the node fails to report its status for a configured period (node-monitor-grace-period, default 40 seconds).

Quick Fix

Step 1: SSH into the node and check kubelet

ssh user@node-ip
systemctl status kubelet

If the kubelet is not running:

sudo systemctl restart kubelet
sudo systemctl enable kubelet

Check kubelet logs for errors:

sudo journalctl -u kubelet --no-pager -n 50

Common errors:

• Failed to connect to apiserver -- network issue or API server down
• Node not found -- node object was deleted from the cluster
• Out of disk or MemoryPressure -- resource exhaustion

Step 2: Check node resources

# On the node
free -h
df -h
top

WRONG -- disk is 100% full:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   50G     0 100% /

RIGHT -- free up disk space:

# Remove unused Docker images
docker image prune -a -f

# Remove stopped containers
docker container prune -f

# Clean kubelet logs
sudo journalctl --vacuum-size=500M

# Clean apt cache
sudo apt-get clean

Step 3: Check network connectivity from the node

# From the node, test API server connectivity
curl -k https://<api-server-address>:6443/healthz

# Check DNS resolution
nslookup kubernetes.default.svc.cluster.local <cluster-dns-ip>

If the node cannot reach the API server, check:

• Firewall rules blocking port 6443
• CNI plugin (Calico, Flannel, Cilium) status
• iptables rules on the node
• Route table entries

Step 4: Restart the container runtime

# For containerd
sudo systemctl restart containerd

# For Docker
sudo systemctl restart docker

# Then restart kubelet
sudo systemctl restart kubelet

Step 5: Check for full pod CIDR

If the node cannot allocate pod IPs, check the CNI:

# For Calico
calicoctl node status

# For Flannel
cat /run/flannel/subnet.env

WRONG -- exhausted pod CIDR range:

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24   # 254 IPs may not be enough

RIGHT -- use a larger CIDR or reduce pods per node:

# In kubelet config
maxPods: 50   # reduce from default 110

Step 6: Remove and rejoin the node

As a last resort, remove the node and rejoin:

# On the control plane
kubectl drain node-1 --ignore-daemonsets
kubectl delete node node-1

# On the node
sudo kubeadm reset
sudo kubeadm join <api-server>:6443 --token <token> --discovery-token-ca-cert-hash <hash>

Use DodaTech's Cluster Health tool to monitor node status and receive alerts when nodes become NotReady.

Prevention

• Set up node monitoring with Prometheus and alerting.
• Reserve resources for system daemons using kubeletReserved.
• Regularly clean up disk space on nodes.
• Run kubectl top nodes in your daily routine.
• Configure the CNI plugin with sufficient pod CIDR range.
• Use PodDisruptionBudgets for critical workloads during node maintenance.

Common Mistakes with node not ready

1. Misunderstanding that String is [Char] with poor performance for large text operations
2. Using foldl instead of foldl' causing stack overflow on large lists
3. Forgetting deriving (Show, Eq) on custom data types needed for debugging

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ