Kubernetes NetworkPolicy Incident: The Checkout API Timed Out After a Security Change

09:32 - Timeouts Begin

Northstar Retail is tightening network isolation in its checkout namespace. A platform engineer applies a default-deny policy during a planned morning change. Within a minute, the public gateway reports upstream timeouts for payments-api. Existing Pods remain Running and Ready.

The on-call engineer first proves that the application itself is alive:

kubectl -n checkout get pods -l app=payments-api -o wide
kubectl -n checkout exec payments-api-6dcbf78b97-pq4jh -- \
  wget -qO- http://127.0.0.1:8080/ready
# ok

The Service also has endpoints:

kubectl -n checkout get svc payments-api
kubectl -n checkout get endpointslice \
  -l kubernetes.io/service-name=payments-api -o wide

So this is not a selector, targetPort, or readiness failure.

---

09:36 - Compare Source Locations

A request from another Pod in checkout times out:

kubectl -n checkout run test --rm -it --restart=Never \
  --image=curlimages/curl -- curl -m 3 -sv http://payments-api:8080/ready
# curl: (28) Connection timed out after 3001 milliseconds

The same request from the Kong-compatible gateway in namespace edge also times out. A timeout rather than connection refused suggests dropped traffic, so the team inspects policies before touching the application.

kubectl -n checkout get networkpolicy
kubectl -n checkout describe networkpolicy default-deny
kubectl -n checkout get networkpolicy default-deny -o yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: checkout
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

The empty selector chooses every Pod in the namespace. Empty ingress and egress rule sets allow nothing. NetworkPolicies are additive, so the correct response is to add narrow allows, not weaken the deny policy.

---

09:41 - The First Fix Is Incomplete

The engineer adds ingress from gateway Pods:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-api-allow
  namespace: checkout
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: edge
          podSelector:
            matchLabels:
              app: gateway
      ports:
        - protocol: TCP
          port: 8080

Gateway traffic now reaches the API, but checkout still returns 500. Logs reveal database hostname resolution failures:

kubectl -n checkout logs deploy/payments-api --since=5m
# dial tcp: lookup postgres.data.svc.cluster.local: i/o timeout

The combined namespaceSelector and podSelector in one from item means gateway-labeled Pods in the edge namespace. If they were separate list items, it would mean any Pod in the selected namespace or matching Pods in any namespace - a much broader rule.

Ingress is fixed, but egress remains entirely denied, including DNS.

---

09:46 - Allow DNS and the Exact Dependency

The team confirms its DNS Pods carry k8s-app=kube-dns and then adds UDP and TCP 53 egress. TCP matters for large responses and fallback.

  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: data
          podSelector:
            matchLabels:
              app: postgres
      ports:
        - { protocol: TCP, port: 5432 }

After applying the policy, the test path succeeds from the actual gateway:

kubectl apply -f payments-api-policy.yaml
kubectl -n edge exec deploy/gateway -- \
  curl -fsS http://payments-api.checkout.svc.cluster.local:8080/ready
# ok

Recovery is declared at 09:49.

---

Post-Incident Findings

The direct cause was applying ingress and egress isolation without accompanying allows. The incomplete first repair missed DNS because the runbook tested only a local health endpoint. Staging did not enforce NetworkPolicy, so the manifest had never been exercised by a policy-capable network plugin.

The team moved policy rollout into GitOps, added connectivity tests from the real source namespaces, required a dependency matrix for every workload, and introduced default deny one namespace at a time. They also test negative requirements: the gateway may reach port 8080, but an unrelated namespace and port 9090 must fail.

NetworkPolicy behavior depends on a network plugin that enforces it; merely creating the API object is not proof that packets are filtered. Policy also does not replace application authentication or TLS.

Practice this exact debugging shape - Service, gateway route, ingress source, DNS, and dependency access - in NetworkPolicy and Kong Routing. For the preceding selector and port layer, use Repair Service Discovery.