Skip to content

How to Debug Kubernetes Validating Webhooks

DodaTech Updated 2026-06-24 3 min read

In this tutorial, you'll learn about How to Debug Kubernetes Validating Webhooks. We cover key concepts, practical examples, and best practices.

The Problem

Your validating admission webhook is rejecting valid resources or allowing invalid ones. The error message says admission <a href="/backend/webhooks/">webhook</a> "<a href="/backend/webhooks/">webhook</a>.example.com" denied the request but the reason is unclear. Validating webhooks can block all resource operations if misconfigured.

Quick Fix

Fix 1: Webhook Rejects Everything

WRONG — returning allowed: false for all requests:

func validate(w http.ResponseWriter, r *http.Request) {
    // Always denies everything:
    resp := admissionv1.AdmissionReview{
        Response: &admissionv1.AdmissionResponse{
            Allowed: false,
            Result: &metav1.Status{
                Message: "Rejected",
            },
        },
    }
    // (all creates/updates are blocked)
}

RIGHT — implement proper validation logic:

func validate(w http.ResponseWriter, r *http.Request) {
    body, _ := io.ReadAll(r.Body)
    var review admissionv1.AdmissionReview
    json.Unmarshal(body, &review)

    allowed := true
    var message string
    // implement validation logic here
    // if validation fails:
    //   allowed = false
    //   message = "Validation failed: reason"

    resp := admissionv1.AdmissionReview{
        TypeMeta: review.TypeMeta,
        Response: &admissionv1.AdmissionResponse{
            Allowed: allowed,
            UID:     review.Request.UID,
            Result: &metav1.Status{
                Message: message,
            },
        },
    }
    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(resp)
}

Fix 2: AdmissionReview Version Mismatch

WRONGwebhook sends v1 response but API server sent v1beta1:

// The request is v1beta1.AdmissionReview, handler returns v1.AdmissionReview

RIGHT — use the same version as the request:

func validate(w http.ResponseWriter, r *http.Request) {
    body, _ := io.ReadAll(r.Body)
    var review admissionv1.AdmissionReview
    json.Unmarshal(body, &review)
    // Use the same apiVersion in the response:
    resp := admissionv1.AdmissionReview{
        TypeMeta: metav1.TypeMeta{
            APIVersion: review.APIVersion,  // match the request version
            Kind:       "AdmissionReview",
        },
        Response: &admissionv1.AdmissionResponse{
            UID:     review.Request.UID,
            Allowed: true,
        },
    }
}

Fix 3: matchConditions Not Met

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: webhook.example.com
  matchConditions:
  - name: skip-default-namespace
    expression: "request.namespace != 'default'"
  # The webhook does not fire for resources in the 'default' namespace

If validation is not triggering, the matchConditions expression may be filtering it out.

Fix 4: objectSelector Filters Resources

objectSelector:
  matchLabels:
    validate: "true"
# Only resources with label validate=true trigger the webhook

RIGHT — remove or adjust the selector to match your resources:

# Remove objectSelector entirely for all-resources validation

Fix 5: Dry-Run Mode

# For testing without blocking resources:
sideEffects: None
timeoutSeconds: 5

Use dry-run mode during development:

# Replace the webhook config's failurePolicy:
failurePolicy: Ignore

Fix 6: Multiple Webhooks Conflicting

# Two validating webhooks — one rejects, the other allows
# The rejected resource is denied regardless of the second webhook

RIGHT — order webhooks by <a href="/backend/webhooks/">webhooks</a>[].name (lexicographic) and use reinvocationPolicy:

reinvocationPolicy: IfNeeded  # for mutating webhooks

Use DodaTech's Policy Simulator to test validating webhook rules against sample resources before deploying to production.

Prevention

  • Log admission requests and responses for debugging.
  • Use failurePolicy: Ignore during initial development.
  • Start with permissive rules and tighten gradually.
  • Test webhooks with kubectl apply --validate=false to bypass validation.
  • Monitor webhook latency and error rates with Prometheus.

Common Mistakes with validating webhook

  1. Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable
  2. Using head and tail instead of pattern matching, causing runtime errors on empty lists
  3. Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks

These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.

Practice Exercise

Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.

This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.

FAQ

### What happens when a validating webhook is down?

If failurePolicy: Fail, all matching API operations fail. If failurePolicy: Ignore, operations proceed without validation. For production, use Fail for security-critical validations and Ignore for advisory ones.

Can a validating webhook modify the resource?

No, validating webhooks can only accept or reject. Use a mutating webhook if you need to modify resources. Validating webhooks that attempt to return a patch will cause an error.

How do I get raw AdmissionReview payloads for debugging?

Set kubectl --v=8 to see the admission review requests and responses. Or add a webhook logging middleware that dumps the request body. Use tools like admission-<a href="/backend/webhooks/">webhook</a>-bootstrapper for testing.

Built by the developers of DodaTech

Doda Browser, DodaZIP & Durga Antivirus Pro