Ashutosh Patole _
kubernetes

Catalog Unreachable With Green Pods

Pods were Ready and endpoints were populated, yet the storefront got connection refused. A NetworkPolicy allowed the wrong pod label on the wrong port.

By Ashutosh PatoleUpdated 10 min read

Scenario

For the next round of hands-on Kubernetes practice, I asked my local LLM for a harder live scenario.

This one looked almost too easy. Two green catalog pods, a Service with populated endpoints, a client that simply could not connect. I had already fixed stranger things this month - a targetPort pointing at a port nobody was listening on in Payments API Down While Pods Are Ready, and a readiness probe aimed at a /healthz path that never existed in Pod Is Running but Not Ready.

I was expecting a port number. Instead, I ran into a misconfigured NetworkPolicy (if you need a primer on how ingress and egress isolation works by default, check out What is Network Policy in Kubernetes).

The Incident

The catalog API was unreachable from storefront in the shop namespace:

  • deploy/catalog — 2 replicas of nginx, serving HTTP on port 80
  • svc/catalog — ClusterIP on port 80
  • deploy/storefront — a client looping on http://catalog/
  • pod/tester — a generic debug pod

Requests to http://catalog/ were refused, even though the catalog pods were 1/1 Running and the Service had both pod IPs in its EndpointSlice.

Troubleshooting

Everything Green, Still Refused

Logging into the Lima VM, the first thing I did was the usual sweep.

bash
ashutosh@lima-k8s-lab:~$ k get nsNAME              STATUS   AGEdefault           Active   17dkube-node-lease   Active   17dkube-public       Active   17dkube-system       Active   17dlogistics         Active   10dpayments          Active   17dprod              Active   17dshipping          Active   17dshop              Active   2d2hashutosh@lima-k8s-lab:~$ alias k="kubectl -n shop"ashutosh@lima-k8s-lab:~$ k get podsNAME                          READY   STATUS    RESTARTS        AGEcatalog-754cffb478-4664l      1/1     Running   0               92scatalog-754cffb478-s4nc8      1/1     Running   0               95sstorefront-66bdcb66bf-jkbmw   1/1     Running   0               87stester                        1/1     Running   1 (2m42s ago)   2d2h

Every pod green. No restarts, no crashes, no Pending, nothing to read in the events.

bash
ashutosh@lima-k8s-lab:~$ k get pods -o wideNAME                          READY   STATUS    RESTARTS      AGE    IP            NODE           NOMINATED NODE   READINESS GATEScatalog-754cffb478-4664l      1/1     Running   0             10m    10.42.0.136   lima-k8s-lab   <none>           <none>catalog-754cffb478-s4nc8      1/1     Running   0             10m    10.42.0.135   lima-k8s-lab   <none>           <none>storefront-66bdcb66bf-jkbmw   1/1     Running   0             10m    10.42.0.137   lima-k8s-lab   <none>           <none>tester                        1/1     Running   1 (11m ago)   2d2h   10.42.0.116   lima-k8s-lab   <none>           <none>ashutosh@lima-k8s-lab:~$ k get svcNAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGEcatalog   ClusterIP   10.43.203.211   <none>        80/TCP    2d2hashutosh@lima-k8s-lab:~$ k get endpointslicesNAME            ADDRESSTYPE   PORTS   ENDPOINTS                 AGEcatalog-mtwrs   IPv4          80      10.42.0.135,10.42.0.136   2d2h

The EndpointSlice had both pod IPs on port 80, which is exactly where nginx listens. The Service was pointing at real, live backends. If this were a targetPort bug like the last one, the slice would have shown a different number.

So the routing table was right and the destination was right. Something was throwing traffic away before it arrived.

bash
ashutosh@lima-k8s-lab:~$ k logs  catalog-754cffb478-4664l --tail=1010.42.0.1 - - [05/Oct/2024:12:39:31 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:39:36 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:39:41 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:39:46 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:39:51 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:39:56 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:40:01 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:40:06 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:40:11 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-"10.42.0.1 - - [05/Oct/2024:12:40:16 +0000] "GET / HTTP/1.1" 200 615 "-" "kube-probe/1.36" "-" ashutosh@lima-k8s-lab:~$ k logs storefront-66bdcb66bf-jkbmw --tail=1012:38:30 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:38:42 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:38:53 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:39:04 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:39:15 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:39:26 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:39:37 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:39:48 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:39:59 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused12:40:10 storefront->catalog: wget: can't connect to remote host (10.43.203.211): Connection refused

Hold onto those two log blocks. I did not appreciate them at the time, and they turned out to be the most important evidence in the entire investigation.

The Detour: Chasing Labels

Having exhausted the obvious suspects, I started poking at labels. It did not make sense even as I typed it, but I had run out of other ideas.

bash
ashutosh@lima-k8s-lab:~$ k get pods --show-labelsNAME                          READY   STATUS    RESTARTS      AGE    LABELScatalog-754cffb478-4664l      1/1     Running   0             29m    app=catalog,pod-template-hash=754cffb478,tier=backendcatalog-754cffb478-s4nc8      1/1     Running   0             29m    app=catalog,pod-template-hash=754cffb478,tier=backendstorefront-66bdcb66bf-jkbmw   1/1     Running   0             29m    app=storefront,pod-template-hash=66bdcb66bftester                        1/1     Running   1 (30m ago)   2d2h   app=testerashutosh@lima-k8s-lab:~$ k get svc -o wide --show-labelsNAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE    SELECTOR      LABELScatalog   ClusterIP   10.43.203.211   <none>        80/TCP    2d2h   app=catalog   <none>

Service selector app=catalog, catalog pods labeled app=catalog. That is a clean match, and it was never the problem.

Then I made it worse. I tried to relabel storefront as app=catalog, on the half-formed theory that maybe the two were supposed to share a label:

bash
ashutosh@lima-k8s-lab:~$ k edit deploy storefronterror: deployments.apps "storefront" is invalidA copy of your changes has been stored to "/tmp/kubectl-edit-2035367222.yaml"error: Edit cancelled, no valid changes were saved.

The Deployment's spec.selector is immutable, the same wall I hit in The Pod That Refused to Be Edited, just one layer up. The controller uses that selector to identify which pods it owns, so letting it drift would orphan live pods.

bash
ashutosh@lima-k8s-lab:~$ k get allNAME                              READY   STATUS    RESTARTS      AGEpod/catalog-754cffb478-4664l      1/1     Running   0             32mpod/catalog-754cffb478-s4nc8      1/1     Running   0             32mpod/storefront-66bdcb66bf-jkbmw   1/1     Running   0             32mpod/tester                        1/1     Running   1 (33m ago)   2d3h NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGEservice/catalog   ClusterIP   10.43.203.211   <none>        80/TCP    2d3h NAME                         READY   UP-TO-DATE   AVAILABLE   AGEdeployment.apps/catalog      2/2     2            2           2d3hdeployment.apps/storefront   1/1     1            1           2d3h NAME                                    DESIRED   CURRENT   READY   AGEreplicaset.apps/catalog-754cffb478      2         2         2       2d3hreplicaset.apps/storefront-66bdcb66bf   1         1         1       2d3hreplicaset.apps/storefront-75c7846698   0         0         0       2d3h

That second storefront ReplicaSet, storefront-75c7846698 sitting at zero, is a fossil. It records a template edit that I made and then reverted, the rollout happened, and then it was rolled back. Nothing was broken by it, but it is the kind of detail that tells you a manifest has been through a few hands.

The Hint: NetworkPolicy

When the Service, the EndpointSlice, the labels, and the ports all line up and traffic still dies, the remaining explanation is that something is choosing to drop it. I asked a local LLM for a nudge and it pointed me at the one object I had not yet listed:

bash
ashutosh@lima-k8s-lab:~$ k get networkpolicy --show-labelsNAME              POD-SELECTOR   AGE    LABELScatalog-ingress   app=catalog    2d2h   <none>

There it is.

What the Policy Was Actually Doing

Before I edited anything, I want to be honest about how NetworkPolicy works, because the object is far more dangerous than it looks.

A NetworkPolicy is not a rule that adds a permission. It is a rule that subtracts from unlimited access. The moment any policy selects a pod for ingress, that pod becomes isolated for ingress and every single connection to it is denied unless some policy explicitly allows it. There is no implicit allow. A policy that permits nothing is not a weak policy; it is a total blackout.

Two details in this policy were enough to guarantee that blackout.

First, the peer selector matched nobody. The from block asked for pods carrying access: catalog-client:

yaml
spec:  ingress:  - from:    - podSelector:        matchLabels:          access: catalog-client    ports:    - port: 8080      protocol: TCP  podSelector:    matchLabels:      app: catalog  policyTypes:  - Ingress

No pod in the shop namespace had ever carried access: catalog-client. Not one. And a selector that matches zero pods is not an error k8s accepts it, stores it, and reports no warning anywhere. It simply produces an empty allowlist. The policy as written said: nobody, on any port, may reach `catalog`.

Second, the port was wrong, and wrong in a way that surprises everyone. The policy allowed 8080. The Service exposed 80. My first instinct was "those are different numbers, that's the bug," and it was but the reason is the part I had to think about properly.

NetworkPolicy ports are evaluated after kube-proxy has rewritten the packet. By the time a policy is consulted, the ClusterIP destination has already been DNAT'd to the pod's own address and port. So a policy's port is matched against the container's port on the pod, not the Service's front-door port.

When the Service port and targetPort differ, you must write the policy against targetPort. Here both happened to be 80, so the policy needed 80. It said 8080, which was a port nothing in this namespace had ever used.

So even after fixing the label, traffic would still have been dropped. Two independent faults, stacked which is exactly the shape of these scenarios, and the reason fixing one thing so rarely finishes the job.

Layer 1: The Wrong Label

I fixed the peer selector first.

bash
ashutosh@lima-k8s-lab:~$ k edit networkpolicy catalog-ingressnetworkpolicy.networking.k8s.io/catalog-ingress edited
yaml
spec:  ingress:  - from:    - podSelector:        matchLabels:          app: storefront    ports:    - port: 8080      protocol: TCP  podSelector:    matchLabels:      app: catalog  policyTypes:  - Ingress

And it still did not work:

bash
ashutosh@lima-k8s-lab:~$ kubectl -n shop exec -it tester -- sh -c 'wget -T3 -qO- http://catalog/; echo exit=$?'wget: can't connect to remote host (10.43.203.211): Connection refusedexit=1ashutosh@lima-k8s-lab:~$ kubectl -n shop exec -it deploy/storefront -- sh -c 'wget -T3 -qO- http://catalog/; echo exit=$?'wget: can't connect to remote host (10.43.203.211): Connection refusedexit=1

Both conditions had to be satisfied, right peer and right port. The peer was now correct, and the port was not.

Layer 2: The Wrong Port

Once the peer selector was matching, the port was the only thing left standing between storefront and a working request.

bash
ashutosh@lima-k8s-lab:~$ k edit networkpolicy catalog-ingressnetworkpolicy.networking.k8s.io/catalog-ingress edited
yaml
ashutosh@lima-k8s-lab:~$ k get networkpolicy catalog-ingress -o yamlapiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  annotations:    kubectl.kubernetes.io/last-applied-configuration: |      {"apiVersion":"networking.k8s.io/v1","kind":"NetworkPolicy","metadata":{"annotations":{},"name":"catalog-ingress","namespace":"shop"},"spec":{"ingress":[{"from":[{"podSelector":{"matchLabels":{"access":"catalog-client"}}}],"ports":[{"port":8080,"protocol":"TCP"}]}],"podSelector":{"matchLabels":{"app":"catalog"}},"policyTypes":["Ingress"]}}  creationTimestamp: "2024-10-03T09:59:59Z"  generation: 6  name: catalog-ingress  namespace: shop  resourceVersion: "20915"  uid: a24b3f84-d32c-4895-8d20-83363d101b36spec:  ingress:  - from:    - podSelector:        matchLabels:          app: storefront    ports:    - port: 80      protocol: TCP  podSelector:    matchLabels:      app: catalog  policyTypes:  - Ingress

Note the stale last-applied-configuration annotation still showing the original broken policy. That annotation is what kubectl apply uses for its three-way merge, which is a trap of its own and one I have been burned by elsewhere.

The Proof

Now the asymmetry showed up exactly as it should. storefront had the label, so it got through. tester did not, so it stayed locked out — which is the policy working correctly, and a far more convincing confirmation than a single successful request.

bash
ashutosh@lima-k8s-lab:~$ k exec -it deploy/storefront -- sh -c 'wget -T3 -qO- http://catalog/; echo exit=$?'<!DOCTYPE html><html><head><title>Welcome to nginx!</title><style>html { color-scheme: light dark; }body { width: 35em; margin: 0 auto;font-family: Tahoma, Verdana, Arial, sans-serif; }</style></head><body><h1>Welcome to nginx!</h1><p>If you see this page, the nginx web server is successfully installed andworking. Further configuration is required.</p> <p>For online documentation and support please refer to<a href="http://nginx.org/">nginx.org</a>.<br/>Commercial support is available at<a href="http://nginx.com/">nginx.com</a>.</p> <p><em>Thank you for using nginx.</em></p></body></html>exit=0ashutosh@lima-k8s-lab:~$ k exec -it tester -- sh -c 'wget -T3 -qO- http://catalog/; echo exit=$?'wget: can't connect to remote host (10.43.203.211): Connection refusedexit=1

The Learnings

I need to learn NetworkPolicy in depth this was the first time I had to reason about one, and I only found it because I ran out of more obvious things to blame. That is not a great discovery strategy, so here is what I am taking away. UNDERSTAND networkpolicy in depth.

For the next layer of this series, where a single broken selector strands an entire Service, see Waybill API Down: Four Broken Layers.

References

Table of contents

Related reading

kubernetes

Payments API Down While Pods Are Ready

Pods were 1/1 Ready, yet the API failed. Debugging two hidden Kubernetes faults: a mismatched Service targetPort and a Secret with a trailing newline.

- 7 min read - kubernetes, troubleshooting, services

kubernetes

What is Network Policy in Kubernetes

What Kubernetes Network Policy is, how pod selectors and ingress rules work, and how default-deny isolates traffic between your workloads.

- 6 min read - kubernetes, network-policy, security

kubernetes

Waybill API Down: Four Broken Layers

Four stacked Kubernetes faults: unschedulable pods, a missing Secret key, a failing sidecar readiness probe, and a Service selector matching nothing.

- 10 min read - kubernetes, troubleshooting, scheduling