Kubernetes Taint Toleration Error Fix

In this tutorial, you'll learn about Kubernetes Taint Toleration Error Fix. We cover key concepts, practical examples, and best practices.

The Problem

Your pod is stuck in Pending and describing it shows:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  10s   default-scheduler  0/3 nodes are available: 3 node(s) had taints that the pod didn't tolerate.

At least one node has a taint that repels the pod. The scheduler will not place the pod on a node unless the pod has a matching toleration for all taints with effect NoSchedule or NoExecute.

Quick Fix

Step 1: List taints on all nodes

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

NAME       TAINTS
master     node-role.kubernetes.io/master:NoSchedule
node-1     <none>
node-2     <none>
node-3     dedicated=gpu:NoSchedule

The output shows each node's taints. Your pod must tolerate every NoSchedule and NoExecute taint on the node where it needs to run.

Step 2: Describe the pod to confirm

kubectl describe pod my-pod

Check the Tolerations section:

Tolerations:     <none>

If the pod has no tolerations and the node has taints, the pod cannot be scheduled there.

Step 3: Add tolerations to the pod spec

WRONG -- no tolerations for a tainted node:

spec:
  containers:
  - name: app
    image: nginx
  tolerations: []

RIGHT -- match the toleration to the taint:

spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "gpu"
    effect: "NoSchedule"
  containers:
  - name: app
    image: nginx

For the master node taint:

spec:
  tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Exists"
    effect: "NoSchedule"

Step 4: Use Exists operator for broad tolerations

When you do not care about the taint value, use Exists:

spec:
  tolerations:
  - key: "dedicated"
    operator: "Exists"
    effect: "NoSchedule"

To tolerate all taints (use with caution):

spec:
  tolerations:
  - operator: "Exists"

This allows the pod to be scheduled on any node, including control-plane nodes.

Step 5: Handle NoExecute taints

NoExecute evicts existing pods that do not tolerate it:

spec:
  tolerations:
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"
    effect: "NoExecute"
    tolerationSeconds: 60   # stay for 60 seconds after the taint is added

Step 6: Remove taints from nodes (when appropriate)

If you control the cluster and want to allow all pods on a node:

kubectl taint nodes node-3 dedicated-

The trailing - removes the taint. Only do this if the taint is no longer needed (e.g., you decommissioned the GPU workload).

Use DodaTech's K8s Debugger to view a taint-toleration matching matrix across all nodes and pods.

Prevention

• Document all node taints in your cluster inventory.
• Add tolerations to deployment templates before applying.
• Use PreferNoSchedule instead of NoSchedule where possible (it is a soft preference).
• Set default tolerations in your organization's pod template standards.
• Test scheduling with kubectl run and --dry-run=client before deploying.

Common Mistakes with taint toleration

1. Using return to exit a function early instead of wrapping a pure value in the monad
2. Mixing let bindings with <- bindings in do notation, producing type errors
3. Overlapping type class instances that cause GHC to reject the program with ambiguous dispatch errors

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ