Demystifying VKS Networking with VCF

A VKS cluster is built in layers, and different teams own different layers. A pod sits below anything kubectl will show you, and above anything vCenter will show you. That is true of Kubernetes on any virtual infrastructure, but it does mean very few people ever see the whole picture at once. This post walks the entire path, one layer at a time: what happens inside the node, what happens inside the virtual switch in the guest, and what happens when someone exposes a pod with a LoadBalancer Service and a user connects to it from outside.

I have drawn this on a whiteboard often enough that it seemed worth writing down properly. If you have ever had to answer one of these, this post is for you:

• Where should a pod-level firewall rule actually be written?
• Why does my application log every request as coming from the same address?
• Is AntreaProxy a process? What happens to my traffic if it restarts?
• How does a LoadBalancer Service actually get its address?
• Where does a packet really go when one pod talks to another?

Every one of those is the same question in a different costume: where does the packet go, and who rewrites it on the way?

One note before we start. Networking people say SNAT when they mean “rewrite the source address” and DNAT when they mean “rewrite the destination address.” Both appear a lot below, and almost everything confusing in this post comes down to one of them happening somewhere you did not expect.

The whole path, bottom up

Read this from the bottom up, in the direction an arriving packet travels. Everything here is the NSX overlay design. The other switching model is a post of its own.

Count the layers as you go. That number explains most “why is this so hard to troubleshoot” conversations on this platform.

What the diagram makes obvious is that the path crosses three ownership boundaries: the guest operating system and its CNI, the ESXi virtual switch, and the physical network. Each one is well instrumented on its own. None of them shows you the other two. That is the real reason a networking question here is hard to answer from a single console, and it is why this post walks the whole path instead of any one layer of it.

Inside the node: one switch, everything plugged into it

Now zoom in on the guest. The correction I make most often in a room is this: br-int is the OVS bridge. There is no OVS on one side and a bridge on the other. There is one bridge, it is called br-int, and everything plugs into it.

Every pod is connected by a veth pair, which is best thought of as a short virtual cable. One end is the pod’s eth0. The other end is a port on br-int. Two other ports matter: antrea-gw0 is the pod network’s gateway on this node, and antrea-tun0 is the tunnel port used only when a pod talks to a pod on a different node.

Three things live on that bridge, and it is worth being precise about which does what, because the names mislead:

Inside br-int: the tables a packet walks

This is the level almost nobody goes to, and it is the level where the rest of the post stops being mysterious. A packet enters at a port, walks through a series of tables in order, and leaves at a port. That is the whole model.

Two things are worth noticing about that sequence.

First, a Service is just a table. A ClusterIP is not an address that anything holds. No network card anywhere is configured with it. It is only something to match on. The packet arrives addressed to the ClusterIP, hits the table that handles Services, one backend pod is selected, and the destination is rewritten to that pod’s real address. Ping a ClusterIP and you may well get nothing back. That is not a fault. You are pinging something that was never meant to answer.

Second, NetworkPolicy is also just a table, and it sits in this same pipeline, right at the pod’s own interface. Where it sits matters more than it sounds, and it comes back later.

Three ways out, and that is all of pod networking

Once you can see the bridge, every packet a pod sends ends in one of exactly three ways. There is no fourth.

That table is worth reading twice, because it tells you where to write policy. A firewall at the vNIC — on any platform, with any CNI — can only act on traffic it can see, and two of those three paths either never reach a vNIC or arrive inside a tunnel. So pod-level policy belongs at the pod’s own interface, which is exactly where Antrea puts it.

You can write it with kubectl, or author and audit it in NSX and let the Antrea–NSX adapter push it down. Either way the bridge is what enforces it, so you get one policy plane for pods whichever door you came in through. The controls at the vNIC stay for what they are genuinely good at: keeping one cluster’s nodes away from another’s, and limiting both to the Supervisor and the registry.

The three address blocks

The packet walk below needs three names, so here they are in the shortest useful form. An NSX VPC draws addresses from three separate blocks:

How those get allocated, and what you commit to on the Workload Network page of the activation wizard, deserves its own post. For now: VIPs come from External, pods and nodes come from the private VPC CIDR, and the VPC gateway sits between the two.

Exposing a pod: what a LoadBalancer Service really does

There are two separate stories here, and merging them is what makes this hard to explain. First an address gets allocated, which is a chain of controllers across three clusters and never touches a packet. Then a client actually connects.

Part one: allocation

The guest cluster has no load balancer controller of its own. When someone applies a Service of type LoadBalancer:

1. The paravirtual cloud provider, running on the guest cluster’s control plane, notices it. It holds scoped credentials to the Supervisor.
2. It creates a VirtualMachineService in the Supervisor namespace, selecting that cluster’s node VMs.
3. VM Operator turns that into a Service plus an Endpoints object whose members are the node VM addresses.
4. The NSX reconciler watches Supervisor Services and calls the NSX API. Classic deployments run NCP; VPC-mode builds may run the NSX VPC Operator instead. Worth checking which one you have, although the NSX objects are named with an ncp- prefix either way, so the naming will not tell you.
5. NSX creates a virtual server and a server pool on the VPC gateway, and allocates a VIP from the External block.
6. The address travels back down the same chain into the guest Service’s status.loadBalancer.ingress.

This is why <pending> is almost never a guest-cluster problem. There is no controller in that cluster that could allocate an address in the first place. If it is stuck, the chain broke further up. Check the VirtualMachineService in the Supervisor namespace first, then the NSX reconciler, then whether the External block has any free addresses left.

Part two: the packet

Now a client connects to the VIP. The left column is where the packet is. The right column is what its addresses look like at that exact point, with anything rewritten highlighted.

The rewrite that matters happens earlier than most people expect. The load balancer changes the source address as well as the destination, so the client’s address is already gone at the VPC gateway. That is before the packet has reached a node, and long before Kubernetes has any say in it. It is also why your access logs are full of the same address over and over.

What the pod sees as the source address

The standard advice is that externalTrafficPolicy: Local preserves the client’s address. On this path it does not. It only changes which infrastructure address the pod sees, and I only know that because I went and measured it.

First, where this setting actually lives

externalTrafficPolicy is a field on the Kubernetes Service. You set it with kubectl, in YAML, next to type: LoadBalancer. You never set it in NSX Manager. It is plain upstream Kubernetes, not a VMware addition, and it works the same way on EKS, GKE, OpenShift, or a bare-metal cluster running MetalLB. If you already know this setting from another platform, you know it here.

What is platform-specific is how the setting gets carried out. Choosing Local makes Kubernetes allocate a healthCheckNodePort. Every node then answers a simple health check on that port: 200 if it is running a pod for this Service, 503 if it is not. The controller that reconciles the Service turns that into two things on the load balancer, its pool membership and its health monitor. Nodes that answer 503 fail the check and stop receiving traffic.

So NSX Manager is where you see the result, not where you configure it. If you edit that pool by hand, it will be put back. The Service object is the source of truth.

What each setting does

Here is the measurement, from an NSX VPC lab. Three replicas of traefik/whoami behind a LoadBalancer Service, called with curl from an admin host outside the cluster:

externalTrafficPolicy: Cluster
  RemoteAddr: 192.168.145.1     ← antrea-gw0, the rewrite inside the cluster

externalTrafficPolicy: Local
  RemoteAddr: 100.64.0.1        ← the VPC gateway load balancer

real client address
  never visible on this path at all

That node’s pod addresses came from 192.168.145.0/24, so .1 is its antrea-gw0. And 100.64.0.1 is not a guess. The NSX server pool for that Service shows SNAT Translation Mode: Automap, which replaces the source with the load balancer’s own address before handing the packet to a pool member. Every pool the reconciler creates has it set.

The virtual server is also plain TCP, on a Fast TCP profile, with no awareness of HTTP. So the source address the pod reads is the load balancer’s, not the caller’s. That is how any layer 4 load balancer doing source NAT behaves, on any platform, and it is exactly why terminating at layer 7 exists as a design pattern.

One clarification, because this is easy to overstate. The reconciler does create an HTTP profile on the same load balancer, alongside the TCP and UDP ones. The VPC load balancer is perfectly capable of layer 7. But that profile is used by virtual servers created from Ingress objects. A Service of type LoadBalancer always gets the layer 4 one. The limit is on the Service path, not on the load balancer.

This is worth stating plainly, because the usual Kubernetes advice about Local assumes a load balancer that is not rewriting the source. It is not a defect. It is the documented behaviour of a layer 4 virtual server whose pool is set to Automap. But it does mean Local buys you a shorter path and one less rewrite, not the client’s address.

Automap is a property of the pool, and NSX supports other modes, so in principle it is configurable. In practice these pools are created and maintained for you, so editing them by hand is not a supported workaround. Treat it as a property of the platform and design around it.

So when is Local worth setting?

Not for the client’s address. That is the whole point of the measurement above. But there are two honest reasons to use it. It removes a node-to-node hop, which is worth having on a latency-sensitive path and keeps that traffic off the tunnel between nodes. And it removes one layer of rewriting, which makes a packet capture much easier to read.

The cost is load distribution, and the arithmetic is worth doing. Take three nodes and four replicas: two on node B, one on node C, none on node A. The load balancer sees two healthy nodes and splits the traffic 50/50. Node B’s two pods share its half, so each gets a quarter. Node C’s single pod takes the other half on its own. That one pod is now handling twice the traffic of its peers, and nothing in Kubernetes will correct it. With Cluster, traffic spreads across all three nodes and then across all four pods, so each gets roughly a quarter, at the price of that extra hop.

So Local fits best where pod placement is already even by design: a workload that runs one pod per node, or one where you actually enforce a topology spread constraint. Set it on purpose, not by habit.

Verified two ways

What an application reports is suggestive, not proof. traefik/whoami tells you what its own socket says, which is one tool’s opinion. So I checked the same thing a completely different way, using Antrea Traceflow in liveTraffic mode. That captures the real frame inside the bridge instead of asking an application about it.

apiVersion: crd.antrea.io/v1beta1
kind: Traceflow
metadata:
  name: tf-live
spec:
  liveTraffic: true
  timeout: 60
  destination: { namespace: default, pod: whoami-… }
  packet:
    ipHeader: { protocol: 6 }
    transportHeader: { tcp: { dstPort: 8080 } }

Then call the VIP inside the timeout window and read status.capturedPacket. Both settings, same cluster, minutes apart:

externalTrafficPolicy: Local
  {"srcIP":"100.64.0.1",    "dstIP":"192.168.145.12",
   "tcp":{"dstPort":8080,"flags":2}, "ttl":123}

externalTrafficPolicy: Cluster
  {"srcIP":"192.168.145.1", "dstIP":"192.168.145.12",
   "tcp":{"dstPort":8080,"flags":2}, "ttl":123}

Two unrelated tools, the same two addresses. The load balancer’s Automap rewrite under Local, antrea-gw0 under Cluster, and in neither case the client.

The TTL is 123 in both captures. That is consistent with the rewrite inside the cluster being done by a rule in the bridge rather than being a routing hop. I used to present it as proof, and a reviewer was right to push back: an identical TTL does not prove no proxy was involved, because a proxy ends one connection and starts a new packet with its own TTL anyway. What actually settles it is the pipeline itself and the Traceflow capture, which show the packet being rewritten in the bridge with nothing in user space terminating anything. The TTL supports the conclusion; it does not carry it. (For the curious: starting from a Windows client’s 128, five hops were taken off on the way in, which matches the fabric, T0, transit gateway and VPC gateway.)

This also answers “can I see this flow with the product tooling?” Partly. Antrea Traceflow covers the half inside the guest and will capture live traffic. NSX Traceflow draws the overlay half, but cannot follow a VIP through the load balancer, because it injects a packet between two ports and a virtual server has no port to inject at. Neither tool does the whole path on its own. With the Antrea–NSX adapter registered, Antrea’s results appear in the NSX UI, which is the closest thing to one view across both.

The direction nobody asks about: outbound

Everything above is traffic arriving. Traffic leaving behaves differently, and it is routinely overlooked right up until a partner’s firewall gets involved.

A pod calling something outside leaves through antrea-gw0, out the node’s eth0, and through the VPC gateway, which rewrites the source again. So the mirror image of the client-address story applies: coming in, your pod cannot see who called it; going out, the far end cannot see which pod called them. Same mechanism, opposite direction. Neither is a fault. Both are what a rewrite does.

Which means that when a partner asks you to allow-list “the payments pod” at their firewall, by default you cannot. With the standard VPC gateway rewrite, the finest identity available at that boundary is the namespace, and only where Antrea Egress is in use. Without it, traffic leaves as the node’s address and even that distinction disappears.

Worth knowing that this ceiling has moved. VCF 9.1 supports Antrea Egress with the EgressSeparateSubnet feature gate, which takes egress addresses from a child segment instead of letting the VPC gateway rewrite them again. A chosen set of pods can then keep a dedicated, routable identity that NSX can match policy on. I have not tested this yet, so treat it as documented capability rather than a lab result. But if you have previously told a customer that namespace granularity is a hard ceiling, that answer now needs an asterisk.

One more thing worth saying plainly, because people assume the opposite: outbound traffic is not blocked by default. Nothing stops a pod reaching the internet unless you write policy that does.

Seeing it for yourself

Reading about address rewriting is far less convincing than watching it happen. Two demos, neither of which needs a packet capture.

Watch the allocation chain fire. Three terminals, then apply the Service in the first:

SUPERVISOR_NS=my-namespace

# T1 — guest cluster
kubectl get svc my-app -w

# T2 — Supervisor namespace
kubectl -n $SUPERVISOR_NS get virtualmachineservice,svc,endpoints -w

# T3 — the NSX reconciler. Find it first, the pod name varies by build:
kubectl -n vmware-system-nsx get pods
kubectl -n vmware-system-nsx logs -f <that-pod> --tail=50

T1 shows <pending>. T2 shows the VirtualMachineService appear, then the Service and Endpoints. T3 shows the reconciler working. Then T1 fills in. About fifteen seconds of wall clock, and the whole chain becomes obvious.

Prove the rewrite. This is the two-minute version that makes the entire packet diagram land:

kubectl create deploy whoami --image=traefik/whoami --replicas=3
kubectl expose deploy whoami --type=LoadBalancer --port=80
kubectl get svc whoami -w          # wait for EXTERNAL-IP

VIP=$(kubectl get svc whoami -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# Cluster (default): the rewrite inside the cluster
curl http://$VIP/                  # RemoteAddr = the pod network gateway
                                   # repeat — Hostname rotates across replicas

kubectl patch svc whoami -p '{"spec":{"externalTrafficPolicy":"Local"}}'

# Local: no rewrite inside the cluster — and still not you
curl http://$VIP/                  # RemoteAddr = the VPC gateway LB address
                                   # repeat — Hostname sticks to local backends

traefik/whoami prints both RemoteAddr and Hostname. Watch RemoteAddr change between the two runs. It moves from the pod network gateway to the load balancer’s address, and never to yours. That is a more interesting demo than the one I expected to be giving, because it shows the limit rather than a workaround.

If you want to go deeper, the tools split by layer: pktcap-uw on the ESXi host (use net-stats -l to find the switchport ID), plain tcpdump inside the node VM, and antctl inside the antrea-agent pod for the bridge itself. That last one is the most underused tool on the platform. It will show you the actual tables from the figures above, on your own cluster.

What I would actually do with this

Four design positions, ranked, with the one I would defend hardest first.

1. Size the External block against every consumer, not one. Count one VIP per guest cluster endpoint, one per LoadBalancer Service, and one outbound address per namespace, then add the growth rate of each. Everything else on this list can be fixed later. Running out of routable addresses mid-project cannot.
2. Put Services behind one layer 7 proxy instead of exposing each one. Ten microservices exposed as ten LoadBalancer Services is ten routable addresses. Behind one ingress it is one. Note what this does not buy you: an ingress inside the cluster still sits downstream of the VPC gateway rewrite, so it cannot see your client either. Consolidate for the address economics and the routing, not for the client’s address.
3. Be generous with private VPC CIDRs. They are free, they can repeat across VPCs, and a tight range is what stops a cluster growing eighteen months from now, when nobody remembers why the number was chosen.
4. Decide the pod-level security story explicitly, and early. Pod-to-pod traffic is enforced at the pod’s own interface, which is what the three-ways-out table means in practice. Either you accept NetworkPolicy inside the cluster as the enforcement point, or you deploy the Antrea–NSX adapter so those same policies are authored and audited from NSX. Both are defensible. Finding out which one you have during a security review is not.

None of this is exotic. It follows from a layered platform where each layer does its own job well and none of them can see the others. Once you can hold the whole path in your head at once, the policy question, the source-address question and the address-planning question all turn out to be the same picture from different angles.

Written against VCF 9.1, with the data-plane addresses measured in a live lab rather than assumed. The TTL argument in an earlier version of this post was overstated and has been corrected here. Defaults and object names move between releases, so verify against the version in front of you before designing to them. Corrections and disagreements welcome. I would rather fix this than have it be quietly wrong.