Kubernetes Incident: A Readiness Probe Turned a Safe Rollout Into an Outage

14:07 - The Deployment Starts

The Storefront team deploys checkout-api:4.18.0 to the commerce namespace. The change adds a new health endpoint. Two minutes later, checkout requests return 503. CPU and memory are normal, and all four new Pods show Running.

That last detail sends the first responder in the wrong direction: Running describes the Pod phase, not whether it is Ready to receive Service traffic.

kubectl -n commerce get deploy,pods

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/checkout-api   0/4     4            0           211d

NAME                                READY   STATUS    RESTARTS   AGE
pod/checkout-api-78dfd8cf49-2md4v   0/1     Running   0          2m
pod/checkout-api-78dfd8cf49-7gptk   0/1     Running   0          2m
pod/checkout-api-78dfd8cf49-b6npx   0/1     Running   0          2m
pod/checkout-api-78dfd8cf49-qzg8d   0/1     Running   0          2m

The Deployment has zero available replicas.

---

14:10 - Follow the Service

kubectl -n commerce get svc checkout-api
kubectl -n commerce get endpointslice \
  -l kubernetes.io/service-name=checkout-api -o wide

NAME           ADDRESSTYPE   PORTS   ENDPOINTS   AGE
checkout-x8m2  IPv4          8080    <none>      211d

The Service exists and DNS resolves, but it has no Ready endpoints. The 503 is now explained; the next question is why readiness fails.

kubectl -n commerce describe pod checkout-api-78dfd8cf49-2md4v

Readiness:  http-get http://:8080/health delay=2s timeout=1s period=5s
Events:
  Warning  Unhealthy  Readiness probe failed: HTTP probe failed with statuscode: 404

Application logs show successful startup and no errors. A direct test confirms the mismatch:

kubectl -n commerce exec checkout-api-78dfd8cf49-2md4v -- \
  sh -c 'wget -qSO- http://127.0.0.1:8080/health 2>&1 | head -1'
# HTTP/1.1 404 Not Found

kubectl -n commerce exec checkout-api-78dfd8cf49-2md4v -- \
  sh -c 'wget -qSO- http://127.0.0.1:8080/health/ready 2>&1 | head -1'
# HTTP/1.1 200 OK

The image moved readiness from /health to /health/ready, but the manifest did not.

---

14:14 - Why Did the Old Replicas Disappear?

A normal rolling update should keep old Ready Pods while new Pods fail readiness. Rollout history exposes a second change:

kubectl -n commerce rollout history deploy/checkout-api
kubectl -n commerce get deploy checkout-api -o yaml

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 0
    maxUnavailable: 100%

Three weeks earlier, a developer changed the strategy to fit a temporary namespace quota. It allowed all old replicas to terminate before any new replica became Ready. The bad probe converted a rollout defect into a complete outage.

---

14:16 - Restore Service First

The incident commander rolls back rather than editing live YAML under pressure:

kubectl -n commerce rollout undo deploy/checkout-api --to-revision=41
kubectl -n commerce rollout status deploy/checkout-api --timeout=3m
kubectl -n commerce get endpointslice \
  -l kubernetes.io/service-name=checkout-api -o wide

At 14:18, four endpoints return and checkout recovers. The release is then corrected in Git:

strategy:
  rollingUpdate:
    maxUnavailable: 0
    maxSurge: 1
template:
  spec:
    containers:
      - name: api
        readinessProbe:
          httpGet:
            path: /health/ready
            port: http
          periodSeconds: 5
          timeoutSeconds: 2
          failureThreshold: 3

The team uses server-side dry-run and diff, then watches the rollout and endpoints:

kubectl apply --server-side --dry-run=server -f checkout.yaml
kubectl -n commerce diff -f checkout.yaml
kubectl -n commerce apply -f checkout.yaml
kubectl -n commerce rollout status deploy/checkout-api --timeout=3m

---

What Changed After the Incident

The root cause was an outdated readiness path. The unsafe rollout strategy removed the safety net. Monitoring also alerted on HTTP 503 rates but not Deployment available replicas or empty EndpointSlices, delaying the diagnosis.

The team added a pre-production smoke test that deploys the exact manifest and waits for availability, a policy rejecting maxUnavailable: 100% for customer-facing Deployments, an alert for desired replicas greater than available replicas, and a release check that curls the configured probe path.

They also documented the semantic contract: liveness means the process cannot recover without restart; readiness means it should receive traffic; startup protects slow initialization. Dependency failure belongs in readiness only when removing the Pod from traffic improves the situation.

Reproduce the probe mismatch against a real backing storefront service in Repair Storefront Probes, then practice the interaction between readiness, rollout capacity, and scaling in Rolling Update and HPA Scaling.