Backup, Restore and Scaling

Scaling a StatefulSet down (say, from 5 replicas to 3) removes the higher-ordinal Pods, but by default it deliberately leaves their PersistentVolumeClaims behind, untouched - a safety-first choice that protects against permanently losing data to an accidental scale-down, at the cost of leaving storage costs and cleanup decisions as an explicit, manual follow-up rather than an automatic one. Building on the storage module's distinction between "persistent" and "backed up," StatefulSet backups need the same application-consistency care - a coordinated, application-aware snapshot rather than a raw block copy - and restoring one has an extra wrinkle beyond a stateless workload: the restore has to correctly map data back onto the right ordinal identity, not just "a PVC of the right size." Any real operational plan for a stateful, clustered workload should automate membership changes deliberately (using the application's own tooling to add/remove a member, not just changing the replica count and hoping) and should be tested, not assumed, against several distinct failure scenarios: losing one node, losing an entire zone, and recovering the whole cluster from backup with nothing left running at all.

Analogy: Kubernetes is a thermostat, not a remote control. You declare the temperature; independent controllers measure reality and keep acting. Debugging means finding which sensor, rule or actuator prevents convergence.

Read the object as evidence

kubectl get RESOURCE NAME -n NAMESPACE -o yaml
kubectl describe RESOURCE NAME -n NAMESPACE
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
kubectl explain RESOURCE.spec

Do not memorize these as a ritual. The first command exposes desired and observed state, the second connects conditions and events, the third supplies a timeline, and the fourth checks the server's schema. Compare what the controller was asked to do with what it reports doing. Then test the narrowest hypothesis.

Scenario: A team scales a clustered database's StatefulSet down from five replicas to three, and afterward, to "clean up," manually deletes the leftover PersistentVolumeClaims for the two removed ordinals - only to discover afterward that one of those ordinals had held the only copy of a particular data partition, because the application's own replication had not actually finished rebalancing that data elsewhere first. Kubernetes did exactly what was asked (reduced replica count, left the claims for manual review); the mistake was deleting those claims without first verifying, through the application's own tooling, whether any of them held data that existed nowhere else. Retention and deletion decisions after a scale-down need to be explicit and reviewed against the application's actual data placement, never treated as routine cleanup.

Rehearse scale and recovery

Scaling down usually leaves claims for removed ordinals. That protects data but creates decisions about cost, reuse and deletion. Before shrinking a clustered database, remove or rebalance members through its own API, verify data placement, then change replicas. Backups need an ordinal-independent restore procedure and a mapping from restored data to member identity.

Scenario: A team scales from five to three, manually deletes claims four and three, then discovers a partition replica existed only on those brokers. Kubernetes followed the replica count; application rebalancing was omitted.
Tip: Test replacement on the same node, a different node, a lost zone and an empty-cluster restore. Each exercise validates a different combination of scheduling, attachment, DNS and application recovery.

Production reasoning

Ask four questions: Who owns this object? What dependency must become ready next? Which controller reports the blocking condition? What evidence would disprove my current theory? This prevents symptom-driven changes. Record the context, namespace, object generation, image digest and recent rollout before mutation; a recreated Pod may erase the evidence you needed.

Warning: Running is not the same as ready, healthy, durable or correct. Kubernetes status is layered. Confirm the application-level outcome as well as the object state.
Goal: Put this model into practice in the Kubernetes lab statefulset-redis-identity. Open /labs/kubernetes, choose statefulset-redis-identity, predict the failure path before changing anything, then use the simulator's check command to validate the finished state.

Deliberate practice

Before the lab, write the expected object relationship and the first three commands you will run. Afterward, explain why the fix converged and name one tempting change that would only mask the symptom. Repeat using an explicit namespace and a structured output format. This prediction-observation-explanation loop is what turns command familiarity into production judgment.

45-minute investigation

  1. Map (5 min): draw the owner-to-child chain and mark every namespace, selector, identity and dependency involved.
  2. Predict (5 min): write one expected status condition, one likely event and one log or metric signal before opening the lab.
  3. Observe (10 min): collect YAML, describe output and ordered events. Do not mutate state. Record which observation disproves your first theory.
  4. Repair (15 min): make the smallest declarative correction, watch the responsible controller converge, and verify the user-facing path rather than stopping at Running.
  5. Stress (10 min): change one relevant constraint - replica count, label, readiness, resource value or placement rule - predict the outcome, observe it, then restore the known-good declaration.
Tip: Keep a short incident note with symptom, evidence, hypothesis, change, result. Across four sections this produces a reusable runbook instead of a pile of remembered commands.