CrashLoopBackOff and Pending Pods: A Systematic Kubernetes Debugging Guide

The Status Is a Symptom, Not a Root Cause

CrashLoopBackOff means a container starts, exits, and Kubernetes is delaying another restart with increasing backoff. Pending means the Pod has been accepted by the API but one or more containers are not running - often because it cannot schedule, mount storage, or pull an image.

Start with one snapshot:

kubectl -n shop get pod orders-api-6c7f76d889-8ps2q -o wide
kubectl -n shop describe pod orders-api-6c7f76d889-8ps2q

In describe, distinguish State from Last State. The current state may be Waiting with reason CrashLoopBackOff while the last state shows Terminated, exit code 1, and a useful reason.

---

CrashLoopBackOff Decision Tree

First retrieve the failed attempt:

kubectl -n shop logs orders-api-6c7f76d889-8ps2q -c api --previous
kubectl -n shop logs orders-api-6c7f76d889-8ps2q -c api --tail=200

An exit code narrows the search:

Signal	Likely direction
Exit 0 with Always restart policy	Main process completed when it should remain alive
Exit 1 or 2	Application/configuration/startup error
Exit 126 or 127	Command not executable or not found
Exit 137	SIGKILL, commonly memory limit exceeded
Exit 143	SIGTERM, often a controlled termination
Probe events	Kubelet killed a process after liveness failures

For an OOM:

kubectl -n shop get pod orders-api-6c7f76d889-8ps2q \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{" last="}{.lastState.terminated.reason}{" exit="}{.lastState.terminated.exitCode}{"\n"}{end}'
kubectl -n shop top pod orders-api-6c7f76d889-8ps2q --containers
kubectl -n shop get deploy orders-api -o jsonpath='{.spec.template.spec.containers[0].resources}'

Do not automatically raise the limit. Compare working set over time, concurrency, heap configuration, and the request/limit. A leak with a larger limit only fails later.

For command or configuration errors, inspect what the container actually received:

kubectl -n shop get pod orders-api-6c7f76d889-8ps2q -o yaml
kubectl -n shop get configmap orders-config -o yaml
kubectl -n shop get secret orders-secret -o jsonpath='{.data.DB_HOST}' | base64 -d

Base64 is encoding, not encryption. Avoid printing production secret values unless necessary and keep terminal history and incident transcripts in mind.

---

When Probes Cause the Loop

A readiness failure removes the Pod from Service endpoints but does not restart it. A liveness failure causes the kubelet to restart the container. A startup probe, when configured, delays liveness and readiness until startup succeeds.

kubectl -n shop describe pod orders-api-6c7f76d889-8ps2q | Select-String -Pattern 'Unhealthy' -Context 0,2

On a normal Linux workstation use grep -A2 Unhealthy; the important part is the event text: status code, refused connection, timeout, or missing path. Test the exact handler from within the Pod if tools exist:

kubectl -n shop exec orders-api-6c7f76d889-8ps2q -c api -- \
  wget -S -O- http://127.0.0.1:8080/health/live

Do not make liveness depend on a database or third-party API. An upstream outage would restart every healthy application replica and amplify the failure.

---

Pending: Read the Scheduler Event First

kubectl -n shop describe pod worker-7d4b6f7dbb-n9l4z

Typical scheduler messages are unusually precise:

0/6 nodes are available: 3 Insufficient cpu, 2 node(s) had untolerated taint,
1 node(s) didn't match Pod's node affinity/selector.

Check requests, not current usage, because scheduling is based primarily on requested resources:

kubectl -n shop get pod worker-7d4b6f7dbb-n9l4z \
  -o jsonpath='{.spec.containers[*].resources.requests}'
kubectl describe nodes
kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type

If affinity, selectors, or taints are involved:

kubectl -n shop get pod worker-7d4b6f7dbb-n9l4z \
  -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.affinity}{"\n"}{.spec.tolerations}'
kubectl describe node node-4 | grep -A3 Taints

Change constraints only after establishing their purpose. A zone constraint may protect data locality; a taint may reserve nodes for regulated workloads.

---

Pending After Scheduling: Images and Volumes

A Pod with a node assigned can still be Pending. Events may show ErrImagePull, ImagePullBackOff, FailedMount, or an unbound claim.

kubectl -n shop get pod worker-7d4b6f7dbb-n9l4z \
  -o jsonpath='{.spec.nodeName}{"\n"}{range .status.containerStatuses[*]}{.state.waiting.reason}{"\n"}{end}'
kubectl -n shop get pvc
kubectl -n shop describe pvc queue-data
kubectl get storageclass

For image failures, verify the full repository and tag, image pull secret, registry reachability, and node architecture. For storage, compare requested storage class, access mode, capacity, topology, and binding mode.

The disciplined approach prevents random YAML edits: identify the state transition that failed, collect its event or termination evidence, and change the controller that owns the Pod. The CrashLoop lab, resource requests and OOM lab, and PVC binding lab let you practice each branch separately.