Production Workload Review

Every earlier module in this track examined one concern in isolation - probes, resources, identity, storage, and so on - but reviewing a real production workload means holding all of them in your head at once, because they interact. A properly thorough review walks through ownership (which controller manages this, and how), image immutability (are you pinned to a specific, unchanging artifact or a moving tag), probes (do they represent real user-serving health, from the observability module), resource requests/limits, identity and RBAC, configuration and secret handling, storage durability, disruption budgets, security context, and observability - as one connected system, not a checklist to tick independently. The uncomfortable truth this reveals is that a manifest can be perfectly valid YAML, accepted without complaint by the API server, while the actual service it describes is deeply fragile in practice - validity and resilience are simply different properties. The most useful review technique is to explicitly ask, for each dependency and each layer: what happens if this specific node is lost right now? What happens if this specific downstream dependency becomes slow or unavailable? What happens during a routine rollout? What happens when a credential needs to rotate? What happens if this entire region becomes unavailable? A design that has genuine, considered answers to all five is meaningfully more production-ready than one that has simply never been asked.

Analogy: Kubernetes is a thermostat, not a remote control. You declare the temperature; independent controllers measure reality and keep acting. Debugging means finding which sensor, rule or actuator prevents convergence.

Read the object as evidence

kubectl get RESOURCE NAME -n NAMESPACE -o yaml
kubectl describe RESOURCE NAME -n NAMESPACE
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
kubectl explain RESOURCE.spec

Do not memorize these as a ritual. The first command exposes desired and observed state, the second connects conditions and events, the third supplies a timeline, and the fourth checks the server's schema. Compare what the controller was asked to do with what it reports doing. Then test the narrowest hypothesis.

Scenario: A service is deployed with three replicas, which sounds appropriately resilient on its surface - but all three happen to be scheduled onto the same physical node (no anti-affinity or topology spread was ever configured, from the scheduling module), all three run from one mutable image tag that could silently change underneath them, and all three depend on a single, unprotected database instance with no failover of its own. Losing that one node, that one image, or that one database independently takes down the entire "three-replica" service at once. Replica count alone was never a proxy for availability - true availability comes from independent failure domains at every layer a request actually passes through, not from a number in a YAML file.

Review systems, not isolated manifests

Trace image provenance, controller ownership, probes, resources, placement, disruption, Service endpoints, identity, policy, configuration and storage. For every dependency ask how it is discovered, authenticated, timed out and degraded. Simulate node loss and a routine rollout together. Three replicas are not highly available if one selector, zone, quota or downstream service remains singular.

Scenario: An API spans zones but its only NAT gateway and external credential provider are regional single points. Pod placement looks resilient while the request path is not. Draw the entire dependency graph.
Tip: Require evidence for operational claims: a restore result for backup, a failed-zone exercise for resilience, and latency/error measurements for a safe rollout. YAML presence is not proof.

Production reasoning

Ask four questions: Who owns this object? What dependency must become ready next? Which controller reports the blocking condition? What evidence would disprove my current theory? This prevents symptom-driven changes. Record the context, namespace, object generation, image digest and recent rollout before mutation; a recreated Pod may erase the evidence you needed.

Warning: Running is not the same as ready, healthy, durable or correct. Kubernetes status is layered. Confirm the application-level outcome as well as the object state.
Goal: Put this model into practice in the Kubernetes lab rolling-update-hpa-scaling. Open /labs/kubernetes, choose rolling-update-hpa-scaling, predict the failure path before changing anything, then use the simulator's check command to validate the finished state.

Deliberate practice

Before the lab, write the expected object relationship and the first three commands you will run. Afterward, explain why the fix converged and name one tempting change that would only mask the symptom. Repeat using an explicit namespace and a structured output format. This prediction-observation-explanation loop is what turns command familiarity into production judgment.

45-minute investigation

  1. Map (5 min): draw the owner-to-child chain and mark every namespace, selector, identity and dependency involved.
  2. Predict (5 min): write one expected status condition, one likely event and one log or metric signal before opening the lab.
  3. Observe (10 min): collect YAML, describe output and ordered events. Do not mutate state. Record which observation disproves your first theory.
  4. Repair (15 min): make the smallest declarative correction, watch the responsible controller converge, and verify the user-facing path rather than stopping at Running.
  5. Stress (10 min): change one relevant constraint - replica count, label, readiness, resource value or placement rule - predict the outcome, observe it, then restore the known-good declaration.
Tip: Keep a short incident note with symptom, evidence, hypothesis, change, result. Across four sections this produces a reusable runbook instead of a pile of remembered commands.