Lifecycle, Restart Policy and Termination

A Pod's overall phase (Pending, Running, Succeeded, Failed) is a coarse, high-level summary - useful for a quick glance, but not enough to actually debug anything. The real diagnostic detail lives one level down, in each container's individual state and its human-readable reason (like CrashLoopBackOff or OOMKilled). Here is the sequence the kubelet (the node-level agent introduced earlier) actually follows: it runs any init containers one at a time, in order, waiting for each to fully finish before starting the next; then it starts the main application containers; it can run optional lifecycle hooks like postStart and preStop at the right moments; and it applies your chosen restartPolicy (Always, OnFailure, or Never) whenever a container exits. Termination follows its own careful sequence too: kubelet first runs preStop (a chance to clean up), then sends the TERM signal (a polite "please shut down") to the process, waits up to terminationGracePeriodSeconds for it to exit on its own, and only if it has not exited by then does it send the far more forceful KILL signal, which cannot be caught or ignored. It also helps to clearly separate the three probe types (explored fully in the observability module): readiness failing simply removes the Pod from receiving new traffic without restarting anything; liveness failing causes an actual restart of the container; startup exists purely to give slow-booting applications extra time before the other two probes start counting against it.

Analogy: Kubernetes is a thermostat, not a remote control. You declare the temperature; independent controllers measure reality and keep acting. Debugging means finding which sensor, rule or actuator prevents convergence.

Read the object as evidence

kubectl get RESOURCE NAME -n NAMESPACE -o yaml
kubectl describe RESOURCE NAME -n NAMESPACE
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
kubectl explain RESOURCE.spec

Do not memorize these as a ritual. The first command exposes desired and observed state, the second connects conditions and events, the third supplies a timeline, and the fourth checks the server's schema. Compare what the controller was asked to do with what it reports doing. Then test the narrowest hypothesis.

Scenario: A process ignores the TERM signal entirely and takes 45 seconds to gracefully close its open connections and finish in-flight work, but the Pod's terminationGracePeriodSeconds only allows 30 seconds before kubelet escalates to KILL. During every rolling update, that mismatch kills live, in-progress user requests. Proper signal handling inside the application, a grace period generous enough to cover real drain time, and readiness being turned off before the process actually starts shutting down all have to be designed together - none of them alone is sufficient.

Read a restart precisely

Run kubectl get pod NAME -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.state}{"\tlast="}{.lastState}{"\n"}{end}' and pair it with kubectl logs NAME -c APP --previous. Exit code 137 can mean a cgroup OOM or forced SIGKILL; events and memory metrics distinguish them. Exit code 143 usually reflects SIGTERM. A CrashLoopBackOff is increasing retry delay, not the root cause.

Scenario: A rollout hangs for exactly the termination grace period. The process never handles TERM, so kubelet eventually kills it and live connections reset. Add signal handling, stop readiness before draining, and set a measured grace period.
Tip: Diagnose the previous container before another restart overwrites the evidence.

Production reasoning

Ask four questions: Who owns this object? What dependency must become ready next? Which controller reports the blocking condition? What evidence would disprove my current theory? This prevents symptom-driven changes. Record the context, namespace, object generation, image digest and recent rollout before mutation; a recreated Pod may erase the evidence you needed.

Warning: Running is not the same as ready, healthy, durable or correct. Kubernetes status is layered. Confirm the application-level outcome as well as the object state.
Goal: Put this model into practice in the Kubernetes lab pod-crashloop-debug. Open /labs/kubernetes, choose pod-crashloop-debug, predict the failure path before changing anything, then use the simulator's check command to validate the finished state.

Deliberate practice

Before the lab, write the expected object relationship and the first three commands you will run. Afterward, explain why the fix converged and name one tempting change that would only mask the symptom. Repeat using an explicit namespace and a structured output format. This prediction-observation-explanation loop is what turns command familiarity into production judgment.

45-minute investigation

  1. Map (5 min): draw the owner-to-child chain and mark every namespace, selector, identity and dependency involved.
  2. Predict (5 min): write one expected status condition, one likely event and one log or metric signal before opening the lab.
  3. Observe (10 min): collect YAML, describe output and ordered events. Do not mutate state. Record which observation disproves your first theory.
  4. Repair (15 min): make the smallest declarative correction, watch the responsible controller converge, and verify the user-facing path rather than stopping at Running.
  5. Stress (10 min): change one relevant constraint - replica count, label, readiness, resource value or placement rule - predict the outcome, observe it, then restore the known-good declaration.
Tip: Keep a short incident note with symptom, evidence, hypothesis, change, result. Across four sections this produces a reusable runbook instead of a pile of remembered commands.