kubectl Commands You Actually Use: A Production Troubleshooting Playbook

Start With Context, Not the Broken Pod

Many Kubernetes mistakes happen before troubleshooting begins: the engineer is looking at the wrong cluster or namespace. Establish both explicitly.

kubectl config current-context
kubectl config get-contexts
kubectl config set-context --current --namespace=payments
kubectl cluster-info

Avoid changing the current context during an incident if several terminals are open. Passing --context and -n is noisier but unambiguous:

kubectl --context=prod-eu -n payments get deploy,pods,svc

The first useful view is wide and label-aware:

kubectl -n payments get pods -o wide --show-labels
kubectl -n payments get deploy,rs,pods
kubectl -n payments get events --sort-by=.metadata.creationTimestamp

Events are namespaced, short-lived clues. Read the newest entries at the bottom. They often expose failed scheduling, image pulls, mount errors, and probe failures before application logs can.

---

Describe Explains State; Logs Explain the Process

kubectl describe combines desired configuration, current status, conditions, and events:

kubectl -n payments describe pod checkout-7f8b9d6c5-x2k4m
kubectl -n payments describe deploy checkout

Look at container state and last state, restart count, readiness, mounted volumes, service account, node, resource requests, and the final Events section. Then inspect logs:

kubectl -n payments logs checkout-7f8b9d6c5-x2k4m
kubectl -n payments logs checkout-7f8b9d6c5-x2k4m -c api --tail=100
kubectl -n payments logs checkout-7f8b9d6c5-x2k4m -c api --previous
kubectl -n payments logs deploy/checkout --all-containers --since=15m --prefix

--previous is essential for a restarting container: normal logs shows the current attempt, which may not have failed yet. Logs from a Deployment are convenient, but during an incident name the Pod when replica-specific behavior matters.

---

Query Structured Data Instead of Grepping Tables

Human-readable output is for scanning. JSONPath and custom columns are safer for repeated queries.

kubectl -n payments get pods \
  -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount'

kubectl -n payments get pod checkout-7f8b9d6c5-x2k4m \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{"\n"}{end}'

kubectl -n payments get pods --field-selector=status.phase!=Running
kubectl -n payments get pods -l app=checkout,environment=production

Labels express application identity; field selectors query a limited set of API fields. Do not parse aligned columns in automation.

---

Inspect the Desired Configuration and Rollout

kubectl -n payments get deploy checkout -o yaml
kubectl -n payments rollout status deploy/checkout --timeout=2m
kubectl -n payments rollout history deploy/checkout
kubectl -n payments rollout history deploy/checkout --revision=12
kubectl -n payments diff -f checkout.yaml
kubectl apply --server-side --dry-run=server -f checkout.yaml

The API object is the source of truth, but managed fields and status make raw YAML noisy. Use kubectl get for inspection and keep declarative manifests in version control. Before applying, diff shows the live change and server-side dry-run asks the API server to validate it without persisting it.

If a new revision is clearly bad:

kubectl -n payments rollout undo deploy/checkout --to-revision=11
kubectl -n payments rollout status deploy/checkout

Rollback restores the previous Pod template; it does not undo an external database migration or a changed Secret.

---

Test From Inside the Cluster

A laptop test crosses ingress, load balancers, and external DNS. To isolate Service discovery and pod networking, test from a Pod:

kubectl -n payments run netshoot --rm -it --restart=Never \
  --image=nicolaka/netshoot -- sh

dig checkout.payments.svc.cluster.local
curl -sv http://checkout:8080/ready

For an existing container:

kubectl -n payments exec deploy/frontend -c app -- \
  wget -qO- http://checkout:8080/ready

Minimal images may contain no shell, curl, or DNS tools. When supported by the cluster, attach an ephemeral debugging container rather than modifying the application image:

kubectl -n payments debug -it pod/checkout-7f8b9d6c5-x2k4m \
  --image=busybox:1.36 --target=api

---

Resources, Ownership, and Authorization

kubectl top pods -n payments --containers
kubectl top nodes
kubectl -n payments get pod checkout-7f8b9d6c5-x2k4m \
  -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'
kubectl auth can-i patch deployments -n payments
kubectl auth can-i --list -n payments

top requires the metrics API and is a recent sample, not historical monitoring. Ownership tells you which controller to change: editing a ReplicaSet-owned Pod is temporary because the controller will replace it.

The reliable order is: verify context, inspect the broad state, read events, describe the object, inspect current and previous logs, test the dependency from inside the cluster, then change the controller declaratively. Practice that loop in the kubectl warmup sandbox, then apply it to Debug a CrashLooping Pod.