Advanced Topics

Admission Webhooks

● Advanced ⏱ 20 min read

Every create, update, and delete request to the Kubernetes API passes through an admission chain before being written to etcd. Admission webhooks are HTTP endpoints you register into that chain. They can reject requests (validating) or modify them in-flight (mutating). Kyverno, OPA Gatekeeper, the OTel Operator, and cert-manager all work this way. This guide covers building your own.

Admission Chain

API server admission chain — request flow

kubectl

API request

→

Authn

who are you?

→

Authz

RBAC check

→

Mutating

inject defaults
add labels

→

Validating

accept or
reject

→

etcd

persisted

Mutating runs before Validating. Object shape is final by the time Validating sees it.

Admission chain: Authn → Authz → Mutating webhooks → schema validation → Validating webhooks → etcd. A rejection at any stage returns 4xx to the client.

Validating vs Mutating

Type	Can modify object?	Use for
MutatingWebhookConfiguration	Yes — returns a JSON patch	Injecting sidecars, adding default labels/annotations, setting missing resource requests
ValidatingWebhookConfiguration	No — accept or reject only	Enforcing policy: block images without digest, require labels, reject dangerous configurations

AdmissionReview Format

The API server sends an AdmissionReview object to your webhook endpoint and expects an AdmissionReview response with the admission decision.

AdmissionReview request (sent by API server)

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "request": {
    "uid": "abc-123",
    "kind": {"group": "apps", "version": "v1", "kind": "Deployment"},
    "resource": {"group": "apps", "version": "v1", "resource": "deployments"},
    "namespace": "production",
    "operation": "CREATE",          // CREATE, UPDATE, DELETE, CONNECT
    "userInfo": {"username": "alice"},
    "object": { ... }               // the Deployment being created (JSON)
  }
}

AdmissionReview response — allow with mutation

{
  "apiVersion": "admission.k8s.io/v1",
  "kind": "AdmissionReview",
  "response": {
    "uid": "abc-123",              // must echo the request UID
    "allowed": true,
    "patchType": "JSONPatch",
    "patch": "W3sib3AiOiJhZGQiLCJwYXRoIjoiL21ldGFkYXRhL2xhYmVscy9pbmplY3RlZCIsInZhbHVlIjoidHJ1ZSJ9XQ=="
    // base64([{"op":"add","path":"/metadata/labels/injected","value":"true"}])
  }
}

// Reject response:
// "allowed": false,
// "status": {"code": 403, "message": "image must be pinned to a digest"}

Webhook Server

minimal webhook server — Go

package main

import (
    "encoding/json"
    "net/http"
    admissionv1 "k8s.io/api/admission/v1"
)

func admitHandler(w http.ResponseWriter, r *http.Request) {
    var review admissionv1.AdmissionReview
    if err := json.NewDecoder(r.Body).Decode(&review); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    // Your validation/mutation logic here
    allowed, message := validate(review.Request)

    review.Response = &admissionv1.AdmissionResponse{
        UID:     review.Request.UID,
        Allowed: allowed,
    }
    if !allowed {
        review.Response.Result = &metav1.Status{
            Code:    403,
            Message: message,
        }
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(review)
}

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/validate", admitHandler)
    // TLS required — API server only calls HTTPS endpoints
    http.ListenAndServeTLS(":8443", "/certs/tls.crt", "/certs/tls.key", mux)
}

TLS & cert-manager

Admission webhooks must use HTTPS. The API server verifies the webhook server's certificate using the CA bundle specified in the webhook configuration. Use cert-manager to issue and rotate the certificate automatically.

cert-manager + webhook registration

# cert-manager issues a certificate for the webhook Service
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-webhook-cert
  namespace: my-webhook-system
spec:
  secretName: my-webhook-tls
  dnsNames:
  - my-webhook.my-webhook-system.svc
  - my-webhook.my-webhook-system.svc.cluster.local
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: my-validator
  annotations:
    cert-manager.io/inject-ca-from: my-webhook-system/my-webhook-cert
spec:
  webhooks:
  - name: validate.myapp.io
    admissionReviewVersions: ["v1"]
    clientConfig:
      service:
        name: my-webhook
        namespace: my-webhook-system
        path: /validate
    rules:
    - operations: ["CREATE", "UPDATE"]
      apiGroups: ["apps"]
      apiVersions: ["v1"]
      resources: ["deployments"]
    sideEffects: None
    failurePolicy: Fail

Failure Policy

Policy	Behaviour when webhook is unreachable	Use
`Fail`	Request is rejected. Cluster operations are blocked if webhook is down.	Security-critical enforcement. Webhook must be HA.
`Ignore`	Request proceeds as if webhook approved it. Webhook failure is silent.	Non-critical defaults/labels. Never for security gates.

⚠️

failurePolicy: Fail + unavailable webhook = cluster lockout

If a validating webhook with failurePolicy: Fail goes down and you have no namespaceSelector exclusion for kube-system, even system pods can't be created. Always exclude kube-system and your own webhook namespace from the rules, and run webhooks with at least 2 replicas.

CEL ValidatingAdmissionPolicy

Since Kubernetes 1.30 (GA), ValidatingAdmissionPolicy lets you write admission policies in CEL (Common Expression Language) directly in the API server — no webhook server process needed. Simpler, faster, and no TLS to manage.

ValidatingAdmissionPolicy — require image digest

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-image-digest
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["pods"]
  validations:
  - expression: >
      object.spec.containers.all(c,
        c.image.contains("@sha256:"))
    message: "All container images must be pinned to a digest (@sha256:...)"

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-image-digest-binding
spec:
  policyName: require-image-digest
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchLabels:
        environment: production

Debugging Webhooks

# Check webhook configurations
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

# Describe to see rules and caBundle
kubectl describe validatingwebhookconfiguration my-validator

# Check webhook pod logs
kubectl logs -n my-webhook-system deploy/my-webhook -f

# Test webhook manually with curl (from inside cluster)
kubectl run test --rm -it --image=curlimages/curl -- \
  curl -k https://my-webhook.my-webhook-system.svc/validate \
  -H 'Content-Type: application/json' \
  -d '{"apiVersion":"admission.k8s.io/v1","kind":"AdmissionReview",...}'

# API server audit log shows webhook decisions
kubectl logs -n kube-system kube-apiserver-master | grep webhook