Kubernetes Service and DNS Debugging: From Selector to EndpointSlice

Debug the Path One Layer at a Time

When frontend cannot call http://catalog:8080, do not start by restarting CoreDNS. The request crosses several independent layers:

client Pod -> DNS name -> Service virtual IP -> EndpointSlice address -> backend Pod port

Prove each transition.

---

1. Confirm the Name and Namespace

Inside the client Pod:

kubectl -n storefront exec deploy/frontend -- cat /etc/resolv.conf
kubectl -n storefront exec deploy/frontend -- nslookup catalog
kubectl -n storefront exec deploy/frontend -- nslookup catalog.storefront.svc.cluster.local

The short name catalog normally searches the client's namespace. A client in web will not find a Service in storefront by that short name; use catalog.storefront or the fully qualified name.

If the image lacks nslookup, start a temporary diagnostic Pod in the same namespace:

kubectl -n storefront run dns-test --rm -it --restart=Never \
  --image=busybox:1.36 -- nslookup catalog

If all names fail, inspect DNS components and their logs:

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system get svc kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100

If only one Service name fails, CoreDNS is probably not the root cause.

---

2. Inspect the Service Contract

kubectl -n storefront get svc catalog -o wide
kubectl -n storefront describe svc catalog
kubectl -n storefront get svc catalog -o yaml

A Service's port is what the client connects to. targetPort is the backend container port or a named port.

apiVersion: v1
kind: Service
metadata:
  name: catalog
spec:
  selector:
    app: catalog
    tier: api
  ports:
    - name: http
      port: 8080
      targetPort: http

Named target ports reduce drift when container port numbers change, but the name must match a port in the selected Pod.

---

3. Follow the Selector to EndpointSlices

kubectl -n storefront get pods -l app=catalog,tier=api --show-labels
kubectl -n storefront get endpointslice \
  -l kubernetes.io/service-name=catalog -o wide
kubectl -n storefront get endpointslice \
  -l kubernetes.io/service-name=catalog -o yaml

An empty EndpointSlice usually means the selector matches no Pods, the matched Pods are not Ready, or the Service has no selector by design. Compare labels exactly; app: catalog-api does not match app: catalog.

EndpointSlices are the current scalable endpoint API. The older Endpoints object may still be visible, but scripts and tooling should understand EndpointSlices.

Readiness matters because unready endpoints normally do not receive Service traffic:

kubectl -n storefront get pod catalog-6d9fc6b75b-j7r9m \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{"\n"}{end}'
kubectl -n storefront describe pod catalog-6d9fc6b75b-j7r9m

---

4. Bypass the Service

Get an endpoint IP and test it from the client namespace:

kubectl -n storefront get pods -l app=catalog -o wide
kubectl -n storefront exec deploy/frontend -- \
  curl -sv --connect-timeout 2 http://10.244.3.17:8080/ready
kubectl -n storefront exec deploy/frontend -- \
  curl -sv --connect-timeout 2 http://catalog:8080/ready

If Pod IP works but Service IP fails, investigate the Service definition and cluster service-routing implementation. If both fail, inspect the application listener and policy:

kubectl -n storefront exec pod/catalog-6d9fc6b75b-j7r9m -- \
  sh -c 'wget -S -O- http://127.0.0.1:8080/ready'
kubectl -n storefront get networkpolicy
kubectl -n storefront describe networkpolicy

An application listening only on 127.0.0.1 succeeds locally but rejects traffic sent to the Pod IP. It normally must bind 0.0.0.0 or the Pod address.

---

Interpret the Failure Mode

DNS success never proves application connectivity, and a ClusterIP is not normally reachable from outside the cluster. Keep external ingress debugging separate from this internal path.

The Repair Service Discovery lab deliberately combines a selector and targetPort failure, while NetworkPolicy and Kong Routing adds policy and gateway behavior after basic Service discovery works.