How to Debug Kubernetes Mutating Webhooks

In this tutorial, you'll learn about How to Debug Kubernetes Mutating Webhooks. We cover key concepts, practical examples, and best practices.

The Problem

Your mutating admission webhook is not modifying resources as expected, or it is blocking all resource creation with Internal error occurred: failed calling <a href="/backend/webhooks/">webhook</a>. Certificates may be misconfigured, the webhook service may be unreachable, or the rules may not match.

Quick Fix

Fix 1: Webhook Service Not Reachable

WRONG — webhook service running but using the wrong port:

apiVersion: v1
kind: Service
metadata:
  name: webhook-service
spec:
  ports:
  - port: 443
    targetPort: 8080  # container listens on 9443

RIGHT — match targetPort to the container's listening port:

spec:
  ports:
  - port: 443
    targetPort: 9443   # match the container port

Fix 2: TLS Certificate Issues

kubectl describe mutatingwebhookconfiguration webhook-config
# Webhook "webhook.example.com":
#   ClientConfig:
#     Service:  namespace/webhook-service
#     CaBundle: (truncated)
#     (CABundle must match the CA that signed the webhook's serving cert)

WRONG — the CA bundle does not match the webhook certificate:

# Webhook returns: "x509: certificate signed by unknown authority"

RIGHT — use cert-manager to manage certificates:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: webhook-cert
spec:
  dnsNames:
  - webhook-service.namespace.svc
  - webhook-service.namespace.svc.cluster.local
  issuerRef:
    name: selfsigned-issuer
  secretName: webhook-tls

Reference the secret in the webhook configuration:

clientConfig:
  service:
    name: webhook-service
    namespace: my-namespace
    path: /mutate
  caBundle: <base64-encoded-CA-cert>

Fix 3: Webhook Timeout

# Default timeout is 10 seconds — your webhook takes longer
# Error: "context deadline exceeded"

RIGHT — increase the timeout:

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: webhook-config
webhooks:
- name: webhook.example.com
  clientConfig:
    service: ...
  timeoutSeconds: 30  # increased from default 10

Fix 4: Rules Not Matching Resources

WRONG — webhook does not fire because the rules do not match:

rules:
- operations: ["CREATE"]
  apiGroups: [""]
  apiVersions: ["v1"]
  resources: ["pods"]
  scope: "Namespaced"
# Check that the operation is CREATE, resource is pods, and scope matches

RIGHT — use * wildcard for testing:

rules:
- operations: ["*"]
  apiGroups: ["*"]
  apiVersions: ["*"]
  resources: ["*"]
  scope: "*"

Fix 5: Failure Policy Blocks Resources

failurePolicy: Fail   # if the webhook is down, all creates fail

RIGHT — use Ignore for non-critical webhooks:

failurePolicy: Ignore  # if webhook fails, resource is still created

Fix 6: Check Webhook Logs

kubectl logs -n webhook-ns deployment/webhook -f
# 2024/03/15 10:30:00 "msg"="admit called" "resource"="pods"
# 2024/03/15 10:30:00 "msg"="mutation applied" "patch"="..."

Use DodaTech's Webhook Debugger to simulate admission requests, validate TLS configuration, and trace webhook execution end-to-end.

Prevention

• Use cert-manager for automatic certificate rotation.
• Set failurePolicy: Ignore during development.
• Monitor webhook latency with Prometheus metrics.
• Log all admission requests and responses for debugging.
• Test webhooks in a non-production cluster first.

Common Mistakes with mutating webhook

1. Mixing let bindings with <- bindings in do notation, producing type errors
2. Overlapping type class instances that cause GHC to reject the program with ambiguous dispatch errors
3. Non-exhaustive pattern matches that compile with warnings then crash at runtime

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ