THE ENGINEERING JOURNAL

FIELD NOTES / Kubernetes

The 200 that nobody received

The gateway logged success while clients timed out. The missing piece was a node holding connection state on the return path.

4 min readKarén Danielyan
Blue response packets stop at a missing section of a transparent bridge between two gateway blocks.

Clients of api.thalamus.am were occasionally timing out. The gateway access log showed successful requests.

The request had arrived. The upstream had answered. The log recorded a 200. There was no obvious pattern by endpoint or time of day, and the failures were rare enough to disappear into otherwise healthy dashboards.

I spent too long looking for an application error. The useful question turned out to be: which machines did the connection depend on after the application had done its work?

The timeouts lined up with node termination

Deploy events didn’t explain the failures. Traffic spikes didn’t either. Node termination times did.

The gateway pods weren’t on the terminating nodes. Neither were the upstream pods. We used Karpenter consolidation, so node replacement was a normal part of running the cluster. I had been following where the pods lived and had missed a node in the network path.

The distinction matters: moving pods off a node doesn’t move every connection’s kernel state off it. Karpenter’s disruption process handles node draining and pod eviction. An application’s traffic path can still depend on a machine that no longer hosts one of its pods.

A node without a gateway pod was still in the connection

The gateway was exposed through a Service with externalTrafficPolicy: Cluster. The path below applies to an NLB using instance targets, where traffic reaches a node’s NodePort before the gateway pod. NLBs can also use pod IP targets; AWS documents both target types. Don’t assume the same path for both.

With Cluster, kube-proxy can forward external traffic to a ready endpoint on another node. In the kube-proxy path involved here, source NAT made the receiving node part of the return path too. The Kubernetes source IP walkthrough shows how this happens.

Imagine the connection arriving at node A while the gateway pod runs on node B:

Request:  client -> NLB -> node A -> gateway pod on node B
Response: client <- NLB <- node A <- gateway pod on node B
                          ^
                          connection/NAT state lives here

Node A had no gateway pod to show up in the application’s deployment view. It still held state that this connection needed.

If node A disappears before the connection drains, that state disappears with it. The gateway on node B may finish handling the request while the response can no longer return through the original path.

This is a failure mode to investigate, not a claim that every consolidated node drops traffic. Load balancer deregistration, health checks and termination timing affect what happens. Kubernetes documents connection draining for deleting nodes; a healthy design needs enough time for that process to finish.

Local removed that extra node from our path

The change was to use externalTrafficPolicy: Local for the gateway Service. In this mode, kube-proxy keeps external Service traffic on node-local endpoints. It doesn’t forward a request to a gateway pod on another node.

With matching load balancer health checks, nodes without usable local gateway endpoints stop receiving new traffic once the load balancer observes the change. That removed the cross-node forwarding dependency from this setup.

It also avoids the source NAT used by that forwarding path. Whether the gateway sees the original client IP still depends on the NLB’s target type, protocol and client IP preservation settings. Local doesn’t override every other part of the network.

Check placement and draining together

Changing the policy was only part of the work. A node with no local gateway endpoint can’t forward the request elsewhere to make up for it.

I would review these settings together before making the same change in another cluster:

  • Spread gateway replicas across the nodes and zones where traffic must be served. Check that there is enough spare capacity during maintenance.
  • Use a PodDisruptionBudget for eviction-based disruption, and review the Deployment’s rollout settings separately. PDBs do not constrain a Deployment’s own rolling update.
  • Match the NLB health-check protocol, port and path to the controller and traffic policy. The AWS Load Balancer Controller has specific guidance for Local Services.
  • Allow time for health-check changes, target deregistration and application connection draining before termination.

Zone behavior also depends on cross-zone load balancing and DNS health handling. It isn’t as simple as every zone permanently receiving an equal share of traffic. AWS explains these details, including fail-open behavior when all targets are unhealthy, in its NLB health-check documentation.

Test the result from outside the cluster while a gateway rolls out and while a node drains. Keep requests open long enough to exercise the return path. A test from another pod may take a different route and miss the original problem.

A successful access log isn’t a delivery receipt

The meaning of a 200 entry depends on the proxy and its log format. It tells you what the proxy recorded at its point in the request lifecycle. It doesn’t prove that the client application received and processed the whole response.

When client observations and gateway logs disagree, follow the connection in both directions. In this case, the node that mattered wasn’t running a gateway pod. It was holding the connection state on the way back.

Original