Cluster Bootstrap and Configuration

Everything covered so far in this track assumes a cluster already exists - this module steps back to how one actually comes into being and stays healthy over time. kubeadm is the standard tool for bootstrapping a cluster from scratch: kubeadm init generates the control-plane certificates, runs the core control-plane components as static Pods (Pods defined directly by files on the node's disk, managed by kubelet itself rather than by the API server, which matters because they can start even before the API server is fully up), and produces a join configuration that worker nodes then use, along with a time-limited discovery token and credentials, to securely attach themselves to the cluster. Getting a cluster running at all, though, is only the first layer of decisions - a genuinely production-ready design also has to deliberately choose a Pod/Service network plan and CNI, a DNS setup, how the control-plane API endpoint itself stays reachable if any one control-plane node goes down, how certificates are rotated over their lifetime, and how far apart control-plane and node (kubelet) versions are allowed to drift (version skew, covered later in this module). Because so many of these choices interact, most real deployments favor declarative, repeatable cluster-provisioning tooling built around kubeadm rather than treating kubeadm init as a one-off manual command run once and never revisited.

Analogy: Kubernetes is a thermostat, not a remote control. You declare the temperature; independent controllers measure reality and keep acting. Debugging means finding which sensor, rule or actuator prevents convergence.

Read the object as evidence

kubectl get RESOURCE NAME -n NAMESPACE -o yaml
kubectl describe RESOURCE NAME -n NAMESPACE
kubectl get events -n NAMESPACE --sort-by=.metadata.creationTimestamp
kubectl explain RESOURCE.spec

Do not memorize these as a ritual. The first command exposes desired and observed state, the second connects conditions and events, the third supplies a timeline, and the fourth checks the server's schema. Compare what the controller was asked to do with what it reports doing. Then test the narrowest hypothesis.

Scenario: A cluster is built with three control-plane nodes for high availability, which sounds resilient on paper - but every client's kubeconfig, and every internal component's configuration, happens to point directly at one specific control-plane node's IP address rather than at a shared, load-balanced endpoint in front of all three. When that one specific node is lost, every client that was configured to reach it becomes unable to reach the API server at all, even though the other two control-plane replicas are perfectly healthy and could easily serve those same requests. True control-plane availability has to include the client-facing endpoint and load balancer in front of the replicas, not just the replica count behind it.

Bootstrap is more than kubeadm init

Before initialization, choose a stable control-plane endpoint, certificate names, Pod and Service CIDRs, runtime, cgroup configuration and supported CNI. Afterward verify system Pods, node readiness, DNS and a cross-node workload before joining production traffic. Preserve kubeadm configuration and automate node joins rather than depending on an expiring token copied from a terminal.

Scenario: Control-plane Pods are replicated across three nodes, but every kubeconfig points directly at node one. Losing that node makes the HA components unreachable. The load-balanced API endpoint is part of quorum availability.
Warning: Overlapping Pod, Service and corporate network CIDRs create selective unreachable paths that may appear long after bootstrap. Address planning is a cluster contract.

Production reasoning

Ask four questions: Who owns this object? What dependency must become ready next? Which controller reports the blocking condition? What evidence would disprove my current theory? This prevents symptom-driven changes. Record the context, namespace, object generation, image digest and recent rollout before mutation; a recreated Pod may erase the evidence you needed.

Warning: Running is not the same as ready, healthy, durable or correct. Kubernetes status is layered. Confirm the application-level outcome as well as the object state.
Goal: Put this model into practice in the Kubernetes lab namespace-basics. Open /labs/kubernetes, choose namespace-basics, predict the failure path before changing anything, then use the simulator's check command to validate the finished state.

Deliberate practice

Before the lab, write the expected object relationship and the first three commands you will run. Afterward, explain why the fix converged and name one tempting change that would only mask the symptom. Repeat using an explicit namespace and a structured output format. This prediction-observation-explanation loop is what turns command familiarity into production judgment.

45-minute investigation

  1. Map (5 min): draw the owner-to-child chain and mark every namespace, selector, identity and dependency involved.
  2. Predict (5 min): write one expected status condition, one likely event and one log or metric signal before opening the lab.
  3. Observe (10 min): collect YAML, describe output and ordered events. Do not mutate state. Record which observation disproves your first theory.
  4. Repair (15 min): make the smallest declarative correction, watch the responsible controller converge, and verify the user-facing path rather than stopping at Running.
  5. Stress (10 min): change one relevant constraint - replica count, label, readiness, resource value or placement rule - predict the outcome, observe it, then restore the known-good declaration.
Tip: Keep a short incident note with symptom, evidence, hypothesis, change, result. Across four sections this produces a reusable runbook instead of a pile of remembered commands.