Identity and Admission Control

Every request that reaches the Kubernetes API server passes through three conceptually distinct stages, and confusing them makes troubleshooting much harder than it needs to be. Authentication answers "who is making this request" - proving an identity via a certificate, token or other credential. Authorization (commonly RBAC, covered in the security module) then answers "is this identity allowed to do this specific action on this specific resource." Only after both of those succeed does admission get a turn: admission controllers can validate a request and reject it even though it was technically authorized (for example, rejecting a Pod that tries to run as root under a strict policy), or they can mutate it, silently modifying the object before it is finally stored (for example, injecting a sidecar). Pods themselves authenticate to the API as a ServiceAccount - Kubernetes' built-in identity for workloads rather than humans - and the modern, safer pattern is a projected, audience-bound token that automatically expires and rotates, rather than a long-lived static one. If a workload never needs to call the Kubernetes API at all, explicitly disable automountServiceAccountToken so it is not handed credentials it has no use for.

Analogy: Kubernetes is a thermostat, not a remote control. You declare the temperature; independent controllers measure reality and keep acting. Debugging means finding which sensor, rule or actuator prevents convergence.

Read the object as evidence

kubectl get RESOURCE NAME -n NAMESPACE -o yaml
kubectl describe RESOURCE NAME -n NAMESPACE
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
kubectl explain RESOURCE.spec

Do not memorize these as a ritual. The first command exposes desired and observed state, the second connects conditions and events, the third supplies a timeline, and the fourth checks the server's schema. Compare what the controller was asked to do with what it reports doing. Then test the narrowest hypothesis.

Scenario: A validating admission webhook is deployed with failurePolicy: Fail, meaning any request is rejected outright if the webhook cannot be reached - a reasonable-sounding safety default. But the webhook's own backend runs as a Deployment inside the cluster, and when that Deployment's Pods all go down during an unrelated incident, the webhook becomes unreachable, and failurePolicy: Fail now blocks the creation of every single new Pod cluster-wide, including the webhook's own replacement Pods trying to come back up. Admission controls sit directly on the API's critical write path, so they need careful scoping (excluding their own namespace, for instance) and genuine high-availability operations, not just a reasonable-sounding default.

Follow an API request through gates

Authentication identifies the caller, RBAC authorizes verb and resource, mutating admission may alter the object, validating admission may reject it, and persistence records the accepted form. Reproduce authorization with kubectl auth can-i create pods -n team --as=system:serviceaccount:team:builder. Use server dry-run to exercise admission safely, and scope webhooks with namespace and object selectors.

Scenario: Pod creation times out only in namespaces selected by one webhook. Its backend Service has no endpoints. Excluding its own recovery namespace and defining an intentional failure policy avoids a circular outage.
Warning: failurePolicy: Ignore improves write availability but may bypass a security guarantee; Fail preserves the guarantee but puts webhook reliability on the API critical path. Document the tradeoff.

Production reasoning

Ask four questions: Who owns this object? What dependency must become ready next? Which controller reports the blocking condition? What evidence would disprove my current theory? This prevents symptom-driven changes. Record the context, namespace, object generation, image digest and recent rollout before mutation; a recreated Pod may erase the evidence you needed.

Warning: Running is not the same as ready, healthy, durable or correct. Kubernetes status is layered. Confirm the application-level outcome as well as the object state.
Goal: Put this model into practice in the Kubernetes lab rbac-least-privilege. Open /labs/kubernetes, choose rbac-least-privilege, predict the failure path before changing anything, then use the simulator's check command to validate the finished state.

Deliberate practice

Before the lab, write the expected object relationship and the first three commands you will run. Afterward, explain why the fix converged and name one tempting change that would only mask the symptom. Repeat using an explicit namespace and a structured output format. This prediction-observation-explanation loop is what turns command familiarity into production judgment.

45-minute investigation

  1. Map (5 min): draw the owner-to-child chain and mark every namespace, selector, identity and dependency involved.
  2. Predict (5 min): write one expected status condition, one likely event and one log or metric signal before opening the lab.
  3. Observe (10 min): collect YAML, describe output and ordered events. Do not mutate state. Record which observation disproves your first theory.
  4. Repair (15 min): make the smallest declarative correction, watch the responsible controller converge, and verify the user-facing path rather than stopping at Running.
  5. Stress (10 min): change one relevant constraint - replica count, label, readiness, resource value or placement rule - predict the outcome, observe it, then restore the known-good declaration.
Tip: Keep a short incident note with symptom, evidence, hypothesis, change, result. Across four sections this produces a reusable runbook instead of a pile of remembered commands.