How to Debug Kubernetes Validating Webhooks
In this tutorial, you'll learn about How to Debug Kubernetes Validating Webhooks. We cover key concepts, practical examples, and best practices.
The Problem
Your validating admission webhook is rejecting valid resources or allowing invalid ones. The error message says admission <a href="/backend/webhooks/">webhook</a> "<a href="/backend/webhooks/">webhook</a>.example.com" denied the request but the reason is unclear. Validating webhooks can block all resource operations if misconfigured.
Quick Fix
Fix 1: Webhook Rejects Everything
WRONG — returning allowed: false for all requests:
func validate(w http.ResponseWriter, r *http.Request) {
// Always denies everything:
resp := admissionv1.AdmissionReview{
Response: &admissionv1.AdmissionResponse{
Allowed: false,
Result: &metav1.Status{
Message: "Rejected",
},
},
}
// (all creates/updates are blocked)
}
RIGHT — implement proper validation logic:
func validate(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
var review admissionv1.AdmissionReview
json.Unmarshal(body, &review)
allowed := true
var message string
// implement validation logic here
// if validation fails:
// allowed = false
// message = "Validation failed: reason"
resp := admissionv1.AdmissionReview{
TypeMeta: review.TypeMeta,
Response: &admissionv1.AdmissionResponse{
Allowed: allowed,
UID: review.Request.UID,
Result: &metav1.Status{
Message: message,
},
},
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(resp)
}
Fix 2: AdmissionReview Version Mismatch
WRONG — webhook sends v1 response but API server sent v1beta1:
// The request is v1beta1.AdmissionReview, handler returns v1.AdmissionReview
RIGHT — use the same version as the request:
func validate(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
var review admissionv1.AdmissionReview
json.Unmarshal(body, &review)
// Use the same apiVersion in the response:
resp := admissionv1.AdmissionReview{
TypeMeta: metav1.TypeMeta{
APIVersion: review.APIVersion, // match the request version
Kind: "AdmissionReview",
},
Response: &admissionv1.AdmissionResponse{
UID: review.Request.UID,
Allowed: true,
},
}
}
Fix 3: matchConditions Not Met
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
webhooks:
- name: webhook.example.com
matchConditions:
- name: skip-default-namespace
expression: "request.namespace != 'default'"
# The webhook does not fire for resources in the 'default' namespace
If validation is not triggering, the matchConditions expression may be filtering it out.
Fix 4: objectSelector Filters Resources
objectSelector:
matchLabels:
validate: "true"
# Only resources with label validate=true trigger the webhook
RIGHT — remove or adjust the selector to match your resources:
# Remove objectSelector entirely for all-resources validation
Fix 5: Dry-Run Mode
# For testing without blocking resources:
sideEffects: None
timeoutSeconds: 5
Use dry-run mode during development:
# Replace the webhook config's failurePolicy:
failurePolicy: Ignore
Fix 6: Multiple Webhooks Conflicting
# Two validating webhooks — one rejects, the other allows
# The rejected resource is denied regardless of the second webhook
RIGHT — order webhooks by <a href="/backend/webhooks/">webhooks</a>[].name (lexicographic) and use reinvocationPolicy:
reinvocationPolicy: IfNeeded # for mutating webhooks
Use DodaTech's Policy Simulator to test validating webhook rules against sample resources before deploying to production.
Prevention
- Log admission requests and responses for debugging.
- Use
failurePolicy: Ignoreduring initial development. - Start with permissive rules and tighten gradually.
- Test webhooks with
kubectl apply --validate=falseto bypass validation. - Monitor webhook latency and error rates with Prometheus.
Common Mistakes with validating webhook
- Placing the wildcard pattern first in case expressions, making all subsequent patterns unreachable
- Using
headandtailinstead of pattern matching, causing runtime errors on empty lists - Forgetting that lazy evaluation defers computation until the value is forced, causing space leaks with unevaluated thunks
These mistakes appear frequently in real-world K8S code. DodaTech's contributors have identified these patterns through analysis of open-source projects and production systems.
Practice Exercise
Write a pure function that safely divides two integers using Maybe, then test it with edge cases like division by zero and negative numbers.
This exercise reinforces the concepts covered in this guide. Try implementing it before checking online solutions.
FAQ
Built by the developers of DodaTech
Doda Browser, DodaZIP & Durga Antivirus Pro