This is a lab build log, written as I went. Everything here was done once, on one set of versions, in my own lab. Where I am reporting what the documentation says instead of what I saw, I say so.
I wanted to answer one question: can a tenant on VMware Cloud Foundation get its own Argo CD, sign in with its VCF Automation identity, and then build a virtual machine, pods and a Kubernetes cluster from a single Git repository? Part 1 gets the platform ready and ends with a working Argo CD instance. Part 2 uses it.
Why bother with GitOps on VCF
Most platform teams I talk to have the same three people in the room. An application developer who raised a ticket for a VM and a cluster days ago. A platform engineer who found that production no longer matches the runbook because somebody changed it by hand. And someone from security asking who changed what, when, and who approved it.
GitOps answers all three with one idea: Git holds the desired state, and a controller keeps the platform matching it. A request is a pull request. Drift is detected and corrected. The audit trail is the commit history. Argo CD is the engine most teams pick for this: it is a CNCF graduated project, it has a UI that application teams actually use, and it takes Helm charts, Kustomize and plain YAML without repackaging.
The problem GitOps solves, and why Argo CD.
You can run upstream Argo CD on any Kubernetes cluster. What VCF adds is everything around it:
Nothing to build first. The instance runs as vSphere Pods on the Supervisor. There is no management cluster to stand up.
An operator runs it. You pick from VMware-built Argo CD versions; in my lab 3.4.4 and 3.3.12 were the recommended ones.
Sign-in is already there. The create wizard sets VCF Automation as the identity provider.
One instance per tenant. Each tenant creates its own Argo CD in its own namespace from the org portal.
Targets are attached, not hand-registered. You attach namespaces and VKS clusters from a Scope tab, or with a manifest in Git.
A VM is a manifest too. The same API that takes a Deployment takes a VirtualMachine and a Cluster. That is the subject of Part 2.
What you would otherwise build and keep working yourself.
Two routes to Argo CD on VCF 9.1.1
Supervisor Service, from vCenter
VCF Service, through VCF Automation
Status
Supported for production
Tech Preview in 9.1.1
Install
vCenter, Supervisor Services
Upload in Partner Services, install from VCF Services
Create an instance
Apply an ArgoCD resource in a namespace
Self-service from the org portal
Sign-in
Local admin, or OIDC you configure
VCF Automation, configured for you
Attach targets
ManagedEntity resources
Scope tab, or a ManagedEntity in Git
I followed the right-hand column. I did not exercise the left-hand one in this lab; its rows come from the VCF blog announcing the 9.1.1 GitOps service and the product documentation. Underneath, the VCF Service installed an ordinary Supervisor package on my Supervisor, argocd-service.vsphere.vmware.com.
Before anything else, check whether your Supervisor has the depot service:
$ kubectl -n kube-system get svc depot
Error from server (NotFound): services "depot" not found
That was my starting point. Regional Harbor is what creates this service, and Supervisor Services pull their images through it. Broadcom KB 447329 describes the failure you get if you install a service without it. Harbor is installed from the provider portal under Service Management → VCF Services.
Harbor under VCF Services, not yet installed.
The wizard has three pages: accept the certificates, choose the region, and keep the default properties. The review page is worth reading because it explains the dependency: besides Harbor itself, the install creates overlays that publish Harbor to every Supervisor in the region and create the depot endpoint.
Eleven resources, including the overlays that wire each Supervisor to this Harbor.
In my lab it took about ten minutes to reach Healthy. One pod, harbor-jobservice, crash-looped twice before settling; it starts before the components it depends on. Afterwards the depot resolves:
$ kubectl -n kube-system get endpoints depot
depot 10.150.4.10:443
Harbor Healthy.
Step 2: get the Argo CD Service package
I expected the Argo CD Service to appear in the catalog once Harbor was in, because my lab uses an online depot. It did not.
Harbor was listed. Argo CD was not.
The documented flow is to download the package with the VCF Download Tool and upload it yourself:
Generate a Software Depot ID with vcf-download-tool.
Register it on the Broadcom portal to get an activation code. The activation code replaced the old download token in 9.1 (KB 443647).
Download the component VCF_SERVICE_ARGOCD for version 9.1.1.
In the provider portal, go to Service Management → Partner Services → Upload.
The download tool talks to the Broadcom depot directly with the activation code. It does not go through the VCF Operations depot, which confused me at first.
Uploading the package. It is about 1 GB, so keep the tab open.After the upload, the card appears under VCF Services.
Step 3: install it, and think about two defaults
The install wizard is the same shape as Harbor’s. Two defaults deserve a decision instead of a click:
Scope. The default deploys the service to every region, including ones you add later.
keepCRDs: false. An uninstall removes the custom resource definitions, and with them every Argo CD instance your tenants created. Outside a lab I would set this to true.
Eight resources: the operator on the Supervisor, and the pieces that wire it into VCF Automation.
Those eight resources fall into three groups. The operator runs on the Supervisor. A UI plugin and a global role add the GitOps pages to tenant org portals. And a ConfigurationHandler bridges the two by handing VCF Automation connection details to the Supervisor side. Remember that last one.
The build that failed
My first attempt used build 25568464. Seven resources went Healthy. The ConfigurationHandler did not:
The operator itself was fine. What was broken was the integration: no org-portal self-service and no single sign-on. Worse, the uninstall then stalled, because the delete path tried to call an entity that had never been created.
What I did, and why you should not copy it blindly. I cleared the stuck uninstall by removing orphaned finalizers on three VCF Automation objects, after backing each one up. These are VCF Automation internals and this is not a documented procedure. In a lab that is a reasonable risk. Anywhere else, open a support case first. The exact steps follow so you can recognise the situation, and so support can see what was done.
Why the uninstall was stuck
The ConfigurationHandler had a deletionTimestamp and one finalizer, services.vcfa.broadcom.com/vcfa-service. Before the controller releases that finalizer it calls a delete behaviour in VCF Automation. But the entity was never created, because the original create had failed, so the delete call went out with an empty ID:
.../cloudapi/1.0.0/entities//behaviors/... 400 Bad Request
behaviorFailureCount: 101
delete backoff cooldown active, requeue after 10m0s
Note the entities//. The controller was trying to delete something that did not exist, and it would have retried forever.
The cleanup, step by step
Everything below was run as root on the VCF Automation node, against its own Kubernetes API, in the prelude namespace. The names are UUIDs from my lab; yours will differ. I took the handler’s name from its YAML in the provider portal. That YAML also names the service account it uses, in callbackServiceAccountName.
CH=de35349b-a9c1-5ead-85a1-e40fa7fba23f # the stuck ConfigurationHandler
SA=8df98ef4-ea51-56be-96f1-a515ac6f75d5 # its callback service account (and Secret of the same name)
1. The ConfigurationHandler. Back it up, then remove its finalizer.
The handler disappeared at once. I had also restarted the vcfa-service-manager deployment just before this. I did not wait to see whether the restart alone would help, and I would not expect it to: the fault was in the data, not in a stuck process.
2. The callback service account. The cleanup then stopped on the next object. These objects are named by UUID, so searching for “argocd” finds nothing; use the label.
kubectl -n prelude get vcfaserviceaccount,secret \
-l services.vcfa.broadcom.com/refName=argocd-service.vsphere.vmware.com
kubectl -n prelude get vcfaserviceaccount $SA \
-o jsonpath='deletionTimestamp={.metadata.deletionTimestamp}{"\n"}finalizers={.metadata.finalizers}{"\n"}'
deletionTimestamp=2026-11-11T09:34:08Z
finalizers=["services.vcfa.broadcom.com/vcfa-service",
"services.vcfa.broadcom.com/configurationhandler.6866f8ff-..."]
Two finalizers, and they are not equal. The second belongs to the handler I had just removed, so nothing would ever clear it. The first is the controller’s own, and it should stay, so that VCF Automation still deletes the account properly on its side. I removed only the orphan:
kubectl -n prelude get vcfaserviceaccount $SA -o yaml > vsa-backup.yaml
# confirm which index holds the configurationhandler finalizer before removing it
kubectl -n prelude get vcfaserviceaccount $SA -o jsonpath='{.metadata.finalizers[1]}{"\n"}'
kubectl -n prelude patch vcfaserviceaccount $SA \
--type=json -p='[{"op":"remove","path":"/metadata/finalizers/1"}]'
3. The Secret. The controller picked the cleanup up again and reported what it was waiting for:
requeue after 10s
Unhealthy
waiting for controller-created secret to be fully deleted
The Secret with the same UUID was held by one more handler-owned finalizer:
The Secret backup contains credentials. Keep it only as long as you need it for a support case.
4. Check that nothing is left. A moment later both objects were gone. The service account going away by itself is the good sign: the controller released its own finalizer.
# on the VCF Automation node: should return nothing
kubectl -n prelude get vcfaserviceaccount,secret | grep 8df98ef4
# on the Supervisor: should return nothing
kubectl get pkgi -A | grep -i argo
kubectl get ns | grep -i argo
kubectl get crd | grep -i argocd
In the provider portal, check Administration → Access Control for leftover Argo CD service accounts or roles, and that the service card is back to Not Installed. Leftovers there are what make a reinstall fail with “already exists” (KB 453499). In my lab everything was gone, and the reinstall in the next step went through. Keep the three backup files and a note of the order of events; that is what a support case will ask for.
The second attempt used build 25642124 and went Healthy within minutes.
Build 25642124: all eight Healthy.
I cannot yet say the first build is bad. Both builds were uploaded to VCF Automation at the time, and I have not repeated the install of 25568464 on its own. Until I do, my advice is simply to upload one build, and make it 25642124.
On the Supervisor, the result looks like this:
$ kubectl get pkgi -A | grep argo
svc-argocd-service... 1.2.0-25642124 Reconcile succeeded
$ kubectl get crd | grep argocd
argocds · argocdversions · managedentities · entitymanagementpolicies
Step 4: a tenant creates its own instance
Everything from here is done as a tenant user in the org portal. Under Build & Deploy → Services → GitOps there is now a button to create an Argo CD instance. That page existing at all is the proof that the install wired the tenant side correctly.
The GitOps page in the org portal.
I named the instance org2-argo, chose the project and namespace, and picked version 3.4.4. The wizard warns that project admins and project advanced users become admins of this Argo CD. Take that seriously: project membership is your security boundary.
VCF Automation is the identity provider by default.
The RBAC page holds ordinary Argo CD policy. If you have not read one before:
g, org-admin, role:admin # g assigns a user or group to a role
p, role:app-deployer, applications, sync, default/*, allow # p says what a role may do
A few minutes after submitting, the instance showed Success, with seven pods in the tenant namespace and a load-balancer address of its own.
The instance is up.The provider installs once. Each tenant runs its own Argo CD.
Step 5: sign in with VCF Automation
The button I was hoping for.
This button is what the failed ConfigurationHandler would have cost me. It worked on the first try. The User Info page shows what Argo CD received from VCF Automation, and those are the values to use in RBAC.
Signed in. No applications yet.
One thing I have not settled: my test user is literally named org-admin, so I cannot tell whether the group Argo CD sees comes from the VCF Automation role or from the username. I need to repeat this with a differently named user.
Step 6: give it somewhere to deploy
A new instance has no destinations, not even its own namespace. You add them from the Scope tab in the org portal. I attached the namespace org2-ns1-mrrtd.
Any namespace you can reach in the region, from any project.The namespace appears in Argo CD as a destination behind the VCF Automation proxy.
Look at the URL in that last screenshot. Argo CD does not hold a kubeconfig for the Supervisor. It reaches the namespace through the VCF Automation Kubernetes proxy, using a service account the platform created for the instance. So there are two separate gates: who may use Argo CD, and what Argo CD itself may change.
An Argo CD admin still cannot make Argo CD create something its service account is not allowed to.
What I would tell a customer so far
Three observations from this half. Each was seen once and still needs reproducing.
Install Regional Harbor before any Supervisor Service. The depot service did not exist until Harbor created it.
The online depot did not bring the Argo CD Service. I had to download and upload it.
Build 25568464 failed for me and 25642124 did not. Whether that was the build, or having two builds uploaded, is still open.
At this point a tenant has a private Argo CD, single sign-on, and one namespace attached. In Part 2 I point it at a Git repository and let it build a virtual machine, vSphere Pods, a VKS cluster and the application inside that cluster, then deploy a twelve-service shop from the Argo CD GUI and fix a real fault along the way.
Most VKS guides stop at “cluster created”. A production build has more to it: VLANs that must be proven before vSphere touches them, a private registry the cluster must trust from its first boot, and shared storage that pods can actually write to.
This post walks through a production build I did on VCF 9.1 with the Foundation Load Balancer, Harbor, a 7-node VKS cluster, NFS for ReadWriteMany volumes, and Istio, Contour and Headlamp on top. The commands that matter are all here, in the order you need them.
All names and IP addresses are examples from a made-up network plan, and the terminal screenshots were re-typed with those example values.
What is in this post
Before you start: what to have ready, including the full list of URLs to allow
Step 0: prove the network · Step 1: the jump server · Step 2: the Supervisor and load balancer
Step 3: Harbor · Step 4: the vSphere Namespace · Step 5: the cluster · Step 6: access
Step 7: ReadWriteMany storage · Step 8: Istio and Contour · Step 9: Headlamp · Final checks
What we are building
Three VLANs: management, Supervisor workload with the load balancer VIPs, and the VKS node network with the NFS server on it.
NFSv4.1 export with csi-driver-nfs (a Dell PowerStore in my case; any NFSv4 server works)
Add-ons
cert-manager, Istio, Contour (Gateway API CRDs ship with the VKr)
Console
Headlamp
All work is done from a Windows jump server with the VCF CLI, kubectl, Helm and crane. No Docker is needed anywhere. Every third-party image is copied into Harbor first, so the cluster pulls from one registry you control.
Supervisor workload network, Harbor, FLB data NICs (.200 to .209) and VIPs (.210 to .239)
300
10.30.0.0/26
VKS cluster nodes (.10 to .50) and the NFS server interface (.60)
Inside the cluster: pod CIDR 10.244.0.0/20 (room for 16 nodes) and service CIDR 10.100.0.0/24. Neither can be changed after the cluster exists, so check they don’t overlap anything the pods need to reach.
Before you start
Have these in place before Step 2:
Three VLANs trunked on every host uplink, with gateways, routes in both directions, and NAT for internet egress.
DNS records for the Supervisor API and for Harbor, created once their VIPs are known (the first VIP from the range goes to the Supervisor).
Internal firewall rules: load balancer data NICs → nodes (TCP 6443, 80, 443, 30000 to 32767); nodes → Harbor and Supervisor (443, 6443); jump server → VIPs and nodes (443, 6443, 22).
An NFS server with an interface on the node VLAN and NFSv4 enabled.
administrator@vsphere.local for every vCenter step, and a Broadcom Support Portal login for the downloads.
A Windows jump server with internet access, for the CLI, Helm and crane.
Internet allow list
This is the complete list of what had to be reachable, by source. Allow by FQDN or SNI, never by IP: every one of these is CDN-backed and the addresses rotate. If the firewall does TLS inspection, exclude these names or make the Supervisor, Harbor and the nodes trust the inspection CA. The CDN names are the ones that get missed: a rule for registry.k8s.io without *.pkg.dev gives you pulls that connect and then hang. To find a CDN host, run curl.exe -sI https://registry.k8s.io/v2/pause/manifests/3.10 and read the location: header.
Copying the NFS CSI driver and test images into Harbor with crane
Jump server
ghcr.io, pkg-containers.githubusercontent.com
Copying the Headlamp image into Harbor
Jump server
ocsp.digicert.com, crl3.digicert.com, crl4.digicert.com on TCP 80
Code-signing revocation checks; without them the CLI starts very slowly
Harbor (only if you use its proxy cache)
Docker Hub: registry-1.docker.io, auth.docker.io, production.cloudflare.docker.com · GHCR: ghcr.io, pkg-containers.githubusercontent.com · Quay: quay.io, cdn01.quay.io to cdn03.quay.io · registry.k8s.io with *.pkg.dev · your <name>.azurecr.io
Pulling upstream images through Harbor instead of copying them by hand, and the Trivy vulnerability database
Step 0: prove the network before vSphere uses it
Before you hand a VLAN to the Supervisor, put a small Linux VM on its port group, once on every host, and check:
the gateway answers,
a VM on another VLAN answers, and the reply comes back,
curl -I https://projects.packages.broadcom.com returns an HTTP status line.
Test every host, not just one: a VLAN missing from one host’s uplink trunk only breaks traffic between VMs on different hosts, which is hard to spot later. While the test VM is up, also check the allow list above from each VLAN that needs it.
Step 1: prepare the jump server
Download the VCF CLI for Windows and its plugin bundle from the Broadcom Support Portal. Unblock the downloaded files first, or Windows scans every CLI call and the tools run very slowly:
Get-ChildItem C:\tools -Recurse | Unblock-File
vcf plugin install all --local-source C:\tools\vcf-plugin-bundle
Trust the vCenter root CA in an administrator PowerShell:
tar -xzf C:\temp\crane.tar.gz -C C:\tools crane.exe
Step 2: enable the Supervisor with the Foundation Load Balancer
In the vSphere Client: Supervisor Management → Get Started, VDS networking. The values that matter:
Wizard page
Value
Load balancer
Foundation Load Balancer, One-Arm, Medium, 2 VMs
LB management network
Management port group, 10.10.0.29 to 10.10.0.32 (a couple of spare IPs beyond the two VMs)
LB virtual server network
Workload port group; data NIC IPs 10.20.0.200 to .209; VIP range 10.20.0.210 to .239 (the two ranges must not overlap)
Management network
Starting IP 10.10.0.24 (uses 5 in a row)
Workload network
10.20.0.40 to .99, service CIDR 10.96.0.0/23
Control plane size
Medium (it can grow later, never shrink)
How the Foundation Load Balancer uses its IPs
The wizard asks for three ranges, and each has one job. Management IPs (one per LB VM) are how the Supervisor configures the load balancer. Data NIC IPs (one per LB VM) are the source address of all traffic the load balancer sends to backends, including health checks. The VIP range is handed out on demand: every Service of type LoadBalancer, on the Supervisor or inside a cluster, takes the next free VIP and gives it back when deleted. Size it for one Supervisor API, one per Supervisor Service, one per cluster and one per app ingress, plus headroom.
The three FLB IP ranges, which NIC each lands on, and how a VIP is assigned to a LoadBalancer Service.
At run time, clients only ever connect to a VIP. The active LB VM forwards the connection to a control-plane node on 6443 (Kubernetes API) or to a worker NodePort (apps), and rewrites the source to its data NIC IP, so replies always come back through it. Two consequences: inbound firewall rules only need LB data NICs → nodes, and pod logs show the data NIC IP as the client. In the other direction, traffic leaving a pod is NATed to the node’s IP, so outbound firewall rules use the node subnet as the source, never the pod or service CIDRs.
Traffic paths: Kubernetes API (A), applications through a NodePort (B), and replies returning to the LB data NIC (C).
When the Supervisor is Running, the first VIP (.210) is the API. Log in:
Without a VCF Operations software depot, use Harbor’s legacy Supervisor Service manifest. Register it under Supervisor Management → Services, then install it with a data-values file. The keys that matter:
hostname: harbor.corp.example.com
enableNginxLoadBalancer: true
enableContourHttpProxy: false
harborAdminPassword: "<strong password>"
secretKey: "<exactly 16 characters>"
core:
xsrfKey: "<exactly 32 characters>"
Harbor runs as vSphere Pods on the Supervisor workload network, so that subnet needs internet egress and a working return route for the image pulls. Once the pods are Running, save Harbor’s CA; every cluster needs it to trust the registry:
$b64 = kubectl get secret harbor-tls -n <svc-harbor-namespace> -o jsonpath="{.data.ca\.crt}"
In the Harbor UI, create a Public project called k8s for the third-party images that come later, and log crane in: crane auth login harbor.corp.example.com -u admin --password-stdin.
Step 4: the vSphere Namespace
Add a second workload network for the node VLAN (10.30.0.10 to .50) under Supervisor → Configure → Network → Workload Network.
Create a subscribed content library for Kubernetes releases (https://wp-content.broadcom.com/v2/latest/lib.json) and synchronize the version you will use, so it is stored locally.
Create custom VM classes: cp-12c-48g (12 vCPU / 48 GB) and worker-24c-64g (24 vCPU / 64 GB).
Create the namespace apps on the node network (it can only be chosen at creation), then add the storage policy, the VM classes and the content library.
Step 5: the cluster
First, the Harbor trust secret. Its name must be <cluster>-user-trusted-ca-secret, and it must exist before the cluster is applied. Deleting a cluster removes its secret too, so recreate it before any re-apply:
Then the cluster. On VKS 3.x, trust goes under the osConfiguration variable; the old top-level trust variable is rejected:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: prod-vks01
namespace: apps
spec:
clusterNetwork:
pods:
cidrBlocks: ["10.244.0.0/20"]
services:
cidrBlocks: ["10.100.0.0/24"]
serviceDomain: cluster.local
topology:
class: builtin-generic-v3.6.0
classNamespace: vmware-system-vks-public
version: v1.35.5---vmware.1-vkr.1
controlPlane:
replicas: 3
workers:
machineDeployments:
- class: node-pool
name: np1
replicas: 4
variables:
overrides:
- name: vmClass
value: worker-24c-64g
- name: volumes
value:
- name: containerd
mountPath: /var/lib/containerd
capacity: 500Gi
storageClass: vks-storage-policy
variables:
- name: vmClass
value: cp-12c-48g
- name: storageClass
value: vks-storage-policy
- name: osConfiguration
value:
trust:
additionalTrustedCAs:
- caCert:
secretRef:
name: prod-vks01-user-trusted-ca-secret
key: harbor-ca
The volumes override gives each worker a dedicated 500 GB disk for container images, so large images never fill the root disk.
kubectl apply -f C:\temp\prod-vks01.yaml
kubectl get cluster,machines -n apps
Two minutes after the apply, check that the cluster’s endpoint matches its load balancer service: kubectl get cluster prod-vks01 -n apps -o jsonpath='{.spec.controlPlaneEndpoint.host}' must equal the EXTERNAL-IP of kubectl get svc prod-vks01 -n apps. If you ever rebuild, give the new cluster a name that has not been used in that namespace before.
The cluster with 3/3 control-plane and 4/4 worker machines.
In about 20 minutes the cluster shows AVAILABLE True. Log in:
Everyone logs in with their own directory account through the vSphere Namespace, and the namespace role decides what they get. Grant the roles from the namespace’s Summary → Permissions card as administrator@vsphere.local:
Namespace role
Inside the cluster
Give to
Can view
Login only; rights come from RoleBindings you add
Application teams
Can edit
cluster-admin, created automatically
Platform admins
For each application, create its namespace, set a Pod Security level, and bind the team’s group to the built-in edit role there. VKS clusters enforce restricted by default, which refuses most images that run as root; baseline suits most applications:
An NFS datastore mounted on ESXi gives pods only ReadWriteOnce volumes. For volumes shared across pods, the cluster needs an NFS CSI driver talking to the NFS server directly. Putting the server’s interface on the node VLAN keeps NFS off routers and firewalls entirely.
On the NFS server, create the export /k8s-rwx with NFSv4 enabled, and give host access to the whole node subnet (10.30.0.0/26) as read/write with root allowed (no root squash). The subnet rather than node IPs, because nodes are replaced on every upgrade; root, because the driver creates a folder per volume as root.
What the export needs: node subnet access, read/write, root allowed, NFSv4.1 on the node VLAN.
Test a write from a worker before installing anything. The mount succeeds even when root is squashed; only the write tells you the export is right:
The CSI controller (5/5) and one node pod per node (3/3), all pulled from Harbor.
Then the StorageClass. Retain keeps the data folder on the NFS server when a volume is deleted, which is what you want in production:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-rwx
provisioner: nfs.csi.k8s.io
parameters:
server: 10.30.0.60
share: /k8s-rwx
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
- nfsvers=4.1
- hard
Prove it with a two-replica Deployment, forced onto different nodes with pod anti-affinity, both appending to the same file on one ReadWriteMany PVC. If tail shows both pod names, RWX works across nodes. Note that this driver does not enforce a size per volume; everything shares the export’s capacity.
Step 8: Istio, Contour and Gateway API
VKS add-ons are installed from the Supervisor namespace context, not from inside the cluster:
There is no need for the gateway-api add-on on VKr v1.35.5: the Gateway API CRDs ship with the Kubernetes release, and Istio’s istio GatewayClass is accepted straight away. Check with kubectl get crd | Select-String gateway.networking before installing it on other releases.
Gateway API CRDs present from cluster creation, and the istio GatewayClass accepted.
Contour’s Envoy defaults to NodePort. To get a LoadBalancer VIP, pass a values file, which requires an explicit version:
A fresh Envoy resets connections until at least one Ingress or HTTPProxy exists, so a connection reset on a new ingress VIP is expected. To test Istio end to end, apply a Gateway (class istio, port 80), a small web Deployment and Service, and an HTTPRoute pointing at it. Istio creates the gateway pod and its LoadBalancer service from the Gateway itself; when the Gateway shows PROGRAMMED True, curl its address.
Step 9: Headlamp as the cluster console
Headlamp gives the team a browser view of workloads, logs and events without handing out kubeconfigs. Same pattern as the NFS driver: copy the image into Harbor, install with Helm, expose on a VIP. The chart already creates a ClusterRoleBinding called headlamp-admin, so the login account’s binding needs a different name:
Open http://<EXTERNAL-IP>, choose Token, and paste it. For a production URL, put it behind Contour with a hostname and a certificate, and switch the Service back to ClusterIP.
Final checks
Before handing the platform to application teams, confirm all of these:
kubectl get cluster -n apps shows AVAILABLE True, and 7 nodes are Ready on the expected IPs.
The cluster’s controlPlaneEndpoint equals the EXTERNAL-IP of its Service on the Supervisor.
The two-pod RWX test shows both pod names in the shared file.
vcf addon install list shows cert-manager, Istio and Contour READY, and Contour’s envoy Service has a VIP.
The Istio gateway test returns the test page from its VIP.
A platform admin logs in with a directory account and kubectl auth can-i '*' '*' -A prints yes; an application user can deploy in their namespace only.
Test namespaces are deleted and their VIPs released. All LoadBalancer Services share the same 30-address pool.
If you are building something similar and run into something this post doesn’t cover, let me know in the comments.
VKS makes it easy to give every team its own Kubernetes cluster: VCF Automation hands them out, Cluster API keeps them reconciled, and the Supervisor runs them on the vSphere you already operate. As the number of clusters grows, the natural next step is to see the whole fleet at once: which clusters need attention, what is about to run out, and what a change will do before you make it. That is what I built vks-fleet for.
vks-fleet is an open-source plugin for Headlamp that brings the Supervisor, vCenter, each cluster’s API and its Prometheus together in one view, for operators, read-only admins and tenants alike.
A personal project. vks-fleet is something I built in my own time. It is not a Broadcom or VMware product, not endorsed or supported by either, and it uses only public APIs. It is Apache 2.0, and the code is on GitHub.
The fleet dashboard, in demo mode: a score, what needs you now, eight blocks that each open their page, and every cluster as a tile.
Day-2 questions at fleet scale
VKS gives you a lot out of the box. Cluster API reconciles your clusters, the Supervisor carries the control plane, and VCF Automation hands out self-service. Once there are many clusters, operators start asking fleet-level questions like these:
Which of my clusters needs attention right now, and which of those is actually urgent?
A node has been “deleting” for a day. Is that a PodDisruptionBudget, a volume that won’t detach, or the ESXi host underneath?
Which disk, volume or certificate runs out in the next 30 days?
If I upgrade this cluster, or scale this pool down, what breaks, what moves, and does the namespace have room while it rolls?
What is open security-wise across every cluster, not just the one I happen to be logged into?
Every answer is already there, spread across a kubectl session per cluster, vCenter, the Supervisor’s own namespaces and each cluster’s Prometheus. vks-fleet brings them onto one screen, so the same questions take seconds whether you run three clusters or thirty.
Why a Headlamp plugin
I started out thinking about forking a Kubernetes UI and went the other way. Headlamp is an open-source Kubernetes web UI with a proper plugin system: it already handles sign-in, kubeconfig contexts, per-cluster views, logs, shells and YAML. A plugin can add pages, read any cluster Headlamp is signed in to, and hand off to Headlamp’s own views when you want the raw objects. That meant I could spend the effort on the fleet layer and let Headlamp do the rest.
It also keeps the footprint small. The plugin runs in your browser with Headlamp’s sign-ins. There is no server of its own, no database, no agent in your clusters and no telemetry.
How it is built to work
The plugin follows the way an operator actually works through a problem:
ObserveOne dashboard for the fleet, a score with its trend, and what needs you now.
DetectRanked issues, each with its cause, and the steps that clear the most of them.
ExplainA cluster’s timeline beside a walk down the layers under a node.
PredictWhat runs out in 30 days, and what a change will do before you start it.
ActControlled changes, each with its checks and a server-side dry run.
Watch it in a minute: a one-minute walk through demo mode (the dashboard, investigating a node, a fix with its dry run, pre-flight, Supervisor health, security and what to do first) is at the top of the project’s README.
Observe: one screen for the fleet
The fleet page opens on a dashboard. At the top is a fleet score (the average of each cluster’s best-practice checks) with a 30-day trend, and a short Needs you now list that puts time-bound problems first: a node disk that fills in two days outranks a warning that has been there for a month.
Below it are eight blocks: clusters, Supervisor, capacity, the next 30 days, lifecycle, security, governance and network. Each is one number with one line of context, and each opens the page that owns the detail. Under them, every cluster is a tile on a wall, coloured by its worst open problem and grouped by org.
Detect: issues with a cause, not just a list of findings
The plugin folds raw signals into issues that say what is wrong, why, and what to do. A stuck node, a drain blocked by a PodDisruptionBudget, a crash-looping pod, a certificate close to expiry, quota and memory overcommit, controllers that stopped reconciling, Supervisor services all running on one host, packages that fail to reconcile, backups that stopped: each becomes one line with its fix or the decision it needs.
On top of that sits Do these first: the recommended steps that clear the most issues, ranked by effect, each opening where it is done. A Simulate view shows the fleet as it would look with the fixes applied, so you can see the payoff before spending the change window.
Issues: the steps to do first, then every open issue with its fix or the decision it needs.
Explain: from a pod down to the ESXi host
This is the part I use most. Investigate puts one cluster’s timeline (changes, alerts, events and anomalies) beside a walk down the layers under a node: pod, node, Machine, VM, ESXi host, namespace and Supervisor. It names the deepest layer in trouble, so you know whether you are looking at a Kubernetes problem, a VM problem or a host problem. When two troubled nodes share a host, it says so. It also drafts the post-mortem from the same timeline.
Investigate: what happened, beside every layer under the node. Here the deepest layer in trouble is the Machine, stuck deleting.
Predict: the next 30 days, and pre-flight
Forecasts from each cluster’s Prometheus (node disks, memory, volumes), the Supervisor’s control-plane disk from vCenter, and certificate expiries all land on one 30-day timeline.
Pre-flight answers what will this change do? before an upgrade, a node pool scale or a VM class change. It works out which nodes are replaced, which pods move, which single-replica workloads take a brief outage, which disruption budgets or pinned pods would block it, whether the namespace has room for the extra nodes while it rolls, and whether the Supervisor itself is healthy enough to start. Then it gives a verdict: go ahead, go ahead with care, or stop.
Pre-flight for scaling a pool down by one. The verdict is Stop: a node in the pool is already stuck deleting.
For upgrades across many clusters there is an upgrade planner that arranges clusters in waves (a small canary first, then the rest) with the same checks for each cluster.
Act: controlled changes
Upgrades, node pool scaling, node replacement, drain timeouts, package updates, VM power operations and new clusters can all be started from the plugin. Every change shows its checks first and runs a server-side dry run (?dryRun=All), so anything the API or the VKS admission webhooks would refuse is caught before anything changes. Destructive actions ask you to type the cluster’s name.
For shared installations there is a read by default, elevate to change mode: viewing uses a read-only sign-in, and a change needs a time-limited elevation with a reason, recorded on the object it changed.
Supervisor health
Clusters are only as healthy as the Supervisor running them. A Supervisor page shows the control plane, the ESXi hosts and what runs on each, the leader-election leases of the controllers that manage your clusters (a lease that stops renewing means a controller is stuck, even when its pod says Running), Supervisor services, and the reconcile backlog. Click a host and it shows what that host failing would take down. With an optional, read-only collector it also reads vCenter: configuration status, alarms, utilisation and the control-plane VM’s disks.
Supervisor health: the hosts, what runs on each, and what vCenter reports.
Security: what is open, across every cluster
The Security overview shows what is open across every cluster you are signed in to: privileged namespaces, privileged and host-level pods, cluster-admin grants, and image vulnerabilities from Trivy Operator, counted once per image and split into your images and the ones VKS ships. Policy results from Kyverno or any engine that writes PolicyReports appear beside them. A cluster without a scanner reads not measured, never zero.
Security overview: what is open across the fleet, then one row per cluster.
Principles behind it
Principle
What it means in practice
Public APIs only
Cluster API, VM Operator, NSX VPC, Carvel, Prometheus, and vCenter through a small read-only collector. No private interfaces.
Nothing installed in your clusters
Everything is read through the APIs that are already there. Trivy, Kyverno and Prometheus are used when present, never required.
Your RBAC decides what you see
Operators see every org, read-only admins see everything and change nothing, tenants see only their own org. It follows the account Headlamp signs in with, not a mode in the UI.
Honest numbers
Data that can’t be read is “not measured”, not zero or passing. Every figure says where it came from.
Dry run before every change
Checks, then a server-side dry run, then confirmation. If a multi-step change fails partway, it says which steps took effect.
Try it without a lab
The fastest way to see it is demo mode: a fictional fleet with realistic problems built in, where nothing is ever changed. Install the plugin, open VKS fleet and choose Try the demo. You can scale the demo fleet up to 50 clusters to see how the pages behave at size.
Headlamp desktop app or Docker: download vks-fleet.tar.gz from the latest release, extract it into Headlamp’s plugins folder, restart Headlamp and open VKS fleet.
In a cluster, with Helm: the chart installs Headlamp, the plugin, and a job that keeps the Supervisor sign-in fresh. The quick start has the full commands, including trusting vCenter’s CA so certificates are verified; the core of it is:
Then connect your Supervisor under Settings → Plugins → vks-fleet. It is also listed on Artifact Hub as a Headlamp plugin.
Where it stands, honestly
It is developed and tested against VCF 9.1 with VKS 3.7 on Headlamp 0.45, on a real lab with one Supervisor, two VKS clusters and VCF Automation tenants. Every read and view has been checked there, as have the vCenter collector and changes through the VKS admission webhooks: scaling node pools up and down, pausing and resuming clusters, and a clean Helm install under restricted Pod Security with verified TLS.
What is not yet confirmed on a live system: deletes through the plugin, drain timeouts on machines that are already deleting, and some links into Headlamp’s own pages. The checks also run from the browser, which suits tens of clusters; demo mode exercises the pages at 50, but a real fleet has so far been tested at two. The README’s validation status keeps the current list.
If you run VKS, I would genuinely like to hear how it behaves on your fleet, especially with more clusters, more than one Supervisor, or tenants coming in through VCF Automation.
vks-fleet is a personal open-source project under the Apache 2.0 licence. It is not affiliated with, endorsed by, or supported by Broadcom or VMware. Product names are used only to describe what it works with.
A VKS cluster is built in layers, and different teams own different layers. A pod sits below anything kubectl will show you, and above anything vCenter will show you. That is true of Kubernetes on any virtual infrastructure, but it does mean very few people ever see the whole picture at once. This post walks the entire path, one layer at a time: what happens inside the node, what happens inside the virtual switch in the guest, and what happens when someone exposes a pod with a LoadBalancer Service and a user connects to it from outside.
I have drawn this on a whiteboard often enough that it seemed worth writing down properly. If you have ever had to answer one of these, this post is for you:
Where should a pod-level firewall rule actually be written?
Why does my application log every request as coming from the same address?
Is AntreaProxy a process? What happens to my traffic if it restarts?
How does a LoadBalancer Service actually get its address?
Where does a packet really go when one pod talks to another?
Every one of those is the same question in a different costume: where does the packet go, and who rewrites it on the way?
One note before we start. Networking people say SNAT when they mean “rewrite the source address” and DNAT when they mean “rewrite the destination address.” Both appear a lot below, and almost everything confusing in this post comes down to one of them happening somewhere you did not expect.
The whole path, bottom up
Read this from the bottom up, in the direction an arriving packet travels. Everything here is the NSX overlay design. The other switching model is a post of its own.
Count the layers as you go. That number explains most “why is this so hard to troubleshoot” conversations on this platform.
The VKS data path on NSX overlay networking. Three hypervisor hops, and four inside the guest where vCenter can no longer see them.
What the diagram makes obvious is that the path crosses three ownership boundaries: the guest operating system and its CNI, the ESXi virtual switch, and the physical network. Each one is well instrumented on its own. None of them shows you the other two. That is the real reason a networking question here is hard to answer from a single console, and it is why this post walks the whole path instead of any one layer of it.
Inside the node: one switch, everything plugged into it
Now zoom in on the guest. The correction I make most often in a room is this: br-intis the OVS bridge. There is no OVS on one side and a bridge on the other. There is one bridge, it is called br-int, and everything plugs into it.
One node, drawn honestly. Each pod’s veth peer is a port on br-int. antrea-gw0 is the pod network gateway; antrea-tun0 carries pod traffic between nodes.
Every pod is connected by a veth pair, which is best thought of as a short virtual cable. One end is the pod’s eth0. The other end is a port on br-int. Two other ports matter: antrea-gw0 is the pod network’s gateway on this node, and antrea-tun0 is the tunnel port used only when a pod talks to a pod on a different node.
Three things live on that bridge, and it is worth being precise about which does what, because the names mislead:
Component
What it actually is
ovs-vswitchd
The user-space daemon that holds the flow tables. The kernel module underneath is what actually moves packets.
antrea-agent
Control plane only. It writes the rules. It never touches a packet. If it stops, existing traffic keeps flowing.
AntreaProxy
Not a process at all. It is a set of rules in the bridge: one table picks a backend pod, the next rewrites the destination to that pod’s address. This is what replaces kube-proxy.
The correction that lands hardest. There is no proxy process in the data path. When people say “Antrea does proxying on the bridge,” what actually happens is that a packet matches a rule and has its destination rewritten, in the kernel, with nothing in user space involved. That is why AntreaProxy costs almost nothing, and why restarting antrea-agent does not drop traffic.
Inside br-int: the tables a packet walks
This is the level almost nobody goes to, and it is the level where the rest of the post stops being mysterious. A packet enters at a port, walks through a series of tables in order, and leaves at a port. That is the whole model.
The pipeline inside the bridge. A packet enters at a port, walks the tables in order, and leaves at a port. Service resolution and NetworkPolicy are both just tables in that sequence.
Two things are worth noticing about that sequence.
First, a Service is just a table. A ClusterIP is not an address that anything holds. No network card anywhere is configured with it. It is only something to match on. The packet arrives addressed to the ClusterIP, hits the table that handles Services, one backend pod is selected, and the destination is rewritten to that pod’s real address. Ping a ClusterIP and you may well get nothing back. That is not a fault. You are pinging something that was never meant to answer.
Second, NetworkPolicy is also just a table, and it sits in this same pipeline, right at the pod’s own interface. Where it sits matters more than it sounds, and it comes back later.
Three ways out, and that is all of pod networking
Once you can see the bridge, every packet a pod sends ends in one of exactly three ways. There is no fourth.
Every packet a pod sends ends one of three ways — and each one tells you where its policy belongs.
Destination
What happens
What that means
Pod on the same node
Straight out another port on the same bridge. The packet never reaches a vNIC at all.
Policy for this hop has to live where the traffic is: NetworkPolicy in the bridge, at the pod’s own interface.
Pod on another node
Out antrea-tun0, wrapped in a tunnel header, to the tunnel port on the other node.
At the vNIC it looks like UDP 6081 between two node addresses, so pod-level policy again belongs in the bridge.
Anything outside the cluster
Out antrea-gw0, then the node’s eth0, with the source address rewritten on the way.
The far end sees an infrastructure address instead of the pod, the same as any other masqueraded traffic.
That table is worth reading twice, because it tells you where to write policy. A firewall at the vNIC — on any platform, with any CNI — can only act on traffic it can see, and two of those three paths either never reach a vNIC or arrive inside a tunnel. So pod-level policy belongs at the pod’s own interface, which is exactly where Antrea puts it.
You can write it with kubectl, or author and audit it in NSX and let the Antrea–NSX adapter push it down. Either way the bridge is what enforces it, so you get one policy plane for pods whichever door you came in through. The controls at the vNIC stay for what they are genuinely good at: keeping one cluster’s nodes away from another’s, and limiting both to the Supervisor and the registry.
The three address blocks
The packet walk below needs three names, so here they are in the shortest useful form. An NSX VPC draws addresses from three separate blocks:
Block
What comes from it
Routable?
External
LoadBalancer VIPs, cluster API endpoints, outbound SNAT addresses
Yes — size this one deliberately
Private transit gateway
Addresses reachable across VPCs in the same project
Internally
Private VPC CIDR
Node VMs, and the slice of pod addresses given to each node
No
How those get allocated, and what you commit to on the Workload Network page of the activation wizard, deserves its own post. For now: VIPs come from External, pods and nodes come from the private VPC CIDR, and the VPC gateway sits between the two.
Exposing a pod: what a LoadBalancer Service really does
There are two separate stories here, and merging them is what makes this hard to explain. First an address gets allocated, which is a chain of controllers across three clusters and never touches a packet. Then a client actually connects.
Part one: allocation
The guest cluster has no load balancer controller of its own. When someone applies a Service of type LoadBalancer:
The paravirtual cloud provider, running on the guest cluster’s control plane, notices it. It holds scoped credentials to the Supervisor.
It creates a VirtualMachineService in the Supervisor namespace, selecting that cluster’s node VMs.
VM Operator turns that into a Service plus an Endpoints object whose members are the node VM addresses.
The NSX reconciler watches Supervisor Services and calls the NSX API. Classic deployments run NCP; VPC-mode builds may run the NSX VPC Operator instead. Worth checking which one you have, although the NSX objects are named with an ncp- prefix either way, so the naming will not tell you.
NSX creates a virtual server and a server pool on the VPC gateway, and allocates a VIP from the External block.
The address travels back down the same chain into the guest Service’s status.loadBalancer.ingress.
This is why <pending> is almost never a guest-cluster problem. There is no controller in that cluster that could allocate an address in the first place. If it is stuck, the chain broke further up. Check the VirtualMachineService in the Supervisor namespace first, then the NSX reconciler, then whether the External block has any free addresses left.
Part two: the packet
Now a client connects to the VIP. The left column is where the packet is. The right column is what its addresses look like at that exact point, with anything rewritten highlighted.
Four rewrites — and the one that destroys the client address happens at the load balancer, before Kubernetes ever sees the packet.
The rewrite that matters happens earlier than most people expect. The load balancer changes the source address as well as the destination, so the client’s address is already gone at the VPC gateway. That is before the packet has reached a node, and long before Kubernetes has any say in it. It is also why your access logs are full of the same address over and over.
What the pod sees as the source address
The standard advice is that externalTrafficPolicy: Local preserves the client’s address. On this path it does not. It only changes which infrastructure address the pod sees, and I only know that because I went and measured it.
First, where this setting actually lives
externalTrafficPolicy is a field on the Kubernetes Service. You set it with kubectl, in YAML, next to type: LoadBalancer. You never set it in NSX Manager. It is plain upstream Kubernetes, not a VMware addition, and it works the same way on EKS, GKE, OpenShift, or a bare-metal cluster running MetalLB. If you already know this setting from another platform, you know it here.
What is platform-specific is how the setting gets carried out. Choosing Local makes Kubernetes allocate a healthCheckNodePort. Every node then answers a simple health check on that port: 200 if it is running a pod for this Service, 503 if it is not. The controller that reconciles the Service turns that into two things on the load balancer, its pool membership and its health monitor. Nodes that answer 503 fail the check and stop receiving traffic.
So NSX Manager is where you see the result, not where you configure it. If you edit that pool by hand, it will be put back. The Service object is the source of truth.
Where externalTrafficPolicy sits. It controls the hop inside the cluster — the second rewrite, not the first. Addresses are from the lab measurement later in this post.
What each setting does
Cluster — the default
Every node accepts the traffic, whether or not it is running a pod for this Service.
If the chosen pod is on a different node, the packet is forwarded on, and the source is rewritten to antrea-gw0, the pod network’s gateway on the node that received it.
Load spreads evenly across pods. An extra node-to-node hop is common on clusters with several nodes.
The pod sees the pod network gateway.
Local — opt in
Only nodes actually running a pod for this Service answer, decided by the health check described above.
No extra hop and no second rewrite inside the cluster. Both are real benefits.
Traffic is spread across nodes rather than across pods, so uneven scheduling gives uneven load.
The pod still does not see the client. It sees the load balancer’s own address.
Here is the measurement, from an NSX VPC lab. Three replicas of traefik/whoami behind a LoadBalancer Service, called with curl from an admin host outside the cluster:
externalTrafficPolicy: Cluster
RemoteAddr: 192.168.145.1 ← antrea-gw0, the rewrite inside the cluster
externalTrafficPolicy: Local
RemoteAddr: 100.64.0.1 ← the VPC gateway load balancer
real client address
never visible on this path at all
That node’s pod addresses came from 192.168.145.0/24, so .1 is its antrea-gw0. And 100.64.0.1 is not a guess. The NSX server pool for that Service shows SNAT Translation Mode: Automap, which replaces the source with the load balancer’s own address before handing the packet to a pool member. Every pool the reconciler creates has it set.
The virtual server is also plain TCP, on a Fast TCP profile, with no awareness of HTTP. So the source address the pod reads is the load balancer’s, not the caller’s. That is how any layer 4 load balancer doing source NAT behaves, on any platform, and it is exactly why terminating at layer 7 exists as a design pattern.
One clarification, because this is easy to overstate. The reconciler does create an HTTP profile on the same load balancer, alongside the TCP and UDP ones. The VPC load balancer is perfectly capable of layer 7. But that profile is used by virtual servers created from Ingress objects. A Service of type LoadBalancer always gets the layer 4 one. The limit is on the Service path, not on the load balancer.
The design consequence. The rewrite happens one hop before Kubernetes sees the packet, which makes this an architecture decision rather than a Service setting. The same is true of any layer 4 load balancer doing source NAT, wherever it runs. If an application needs the caller’s address, put something that terminates at layer 7 in front of the rewrite, where the client is still visible. Decide that alongside the ingress design, not when an application team asks the question.
This is worth stating plainly, because the usual Kubernetes advice about Local assumes a load balancer that is not rewriting the source. It is not a defect. It is the documented behaviour of a layer 4 virtual server whose pool is set to Automap. But it does mean Local buys you a shorter path and one less rewrite, not the client’s address.
Automap is a property of the pool, and NSX supports other modes, so in principle it is configurable. In practice these pools are created and maintained for you, so editing them by hand is not a supported workaround. Treat it as a property of the platform and design around it.
So when is Local worth setting?
Not for the client’s address. That is the whole point of the measurement above. But there are two honest reasons to use it. It removes a node-to-node hop, which is worth having on a latency-sensitive path and keeps that traffic off the tunnel between nodes. And it removes one layer of rewriting, which makes a packet capture much easier to read.
The cost is load distribution, and the arithmetic is worth doing. Take three nodes and four replicas: two on node B, one on node C, none on node A. The load balancer sees two healthy nodes and splits the traffic 50/50. Node B’s two pods share its half, so each gets a quarter. Node C’s single pod takes the other half on its own. That one pod is now handling twice the traffic of its peers, and nothing in Kubernetes will correct it. With Cluster, traffic spreads across all three nodes and then across all four pods, so each gets roughly a quarter, at the price of that extra hop.
So Local fits best where pod placement is already even by design: a workload that runs one pod per node, or one where you actually enforce a topology spread constraint. Set it on purpose, not by habit.
Verified two ways
What an application reports is suggestive, not proof. traefik/whoami tells you what its own socket says, which is one tool’s opinion. So I checked the same thing a completely different way, using Antrea Traceflow in liveTraffic mode. That captures the real frame inside the bridge instead of asking an application about it.
Two unrelated tools, the same two addresses. The load balancer’s Automap rewrite under Local, antrea-gw0 under Cluster, and in neither case the client.
The TTL is 123 in both captures. That is consistent with the rewrite inside the cluster being done by a rule in the bridge rather than being a routing hop. I used to present it as proof, and a reviewer was right to push back: an identical TTL does not prove no proxy was involved, because a proxy ends one connection and starts a new packet with its own TTL anyway. What actually settles it is the pipeline itself and the Traceflow capture, which show the packet being rewritten in the bridge with nothing in user space terminating anything. The TTL supports the conclusion; it does not carry it. (For the curious: starting from a Windows client’s 128, five hops were taken off on the way in, which matches the fabric, T0, transit gateway and VPC gateway.)
This also answers “can I see this flow with the product tooling?” Partly. Antrea Traceflow covers the half inside the guest and will capture live traffic. NSX Traceflow draws the overlay half, but cannot follow a VIP through the load balancer, because it injects a packet between two ports and a virtual server has no port to inject at. Neither tool does the whole path on its own. With the Antrea–NSX adapter registered, Antrea’s results appear in the NSX UI, which is the closest thing to one view across both.
The direction nobody asks about: outbound
Everything above is traffic arriving. Traffic leaving behaves differently, and it is routinely overlooked right up until a partner’s firewall gets involved.
A pod calling something outside leaves through antrea-gw0, out the node’s eth0, and through the VPC gateway, which rewrites the source again. So the mirror image of the client-address story applies: coming in, your pod cannot see who called it; going out, the far end cannot see which pod called them. Same mechanism, opposite direction. Neither is a fault. Both are what a rewrite does.
Which means that when a partner asks you to allow-list “the payments pod” at their firewall, by default you cannot. With the standard VPC gateway rewrite, the finest identity available at that boundary is the namespace, and only where Antrea Egress is in use. Without it, traffic leaves as the node’s address and even that distinction disappears.
Worth knowing that this ceiling has moved. VCF 9.1 supports Antrea Egress with the EgressSeparateSubnet feature gate, which takes egress addresses from a child segment instead of letting the VPC gateway rewrite them again. A chosen set of pods can then keep a dedicated, routable identity that NSX can match policy on. I have not tested this yet, so treat it as documented capability rather than a lab result. But if you have previously told a customer that namespace granularity is a hard ceiling, that answer now needs an asterisk.
One more thing worth saying plainly, because people assume the opposite: outbound traffic is not blocked by default. Nothing stops a pod reaching the internet unless you write policy that does.
Seeing it for yourself
Reading about address rewriting is far less convincing than watching it happen. Two demos, neither of which needs a packet capture.
Watch the allocation chain fire. Three terminals, then apply the Service in the first:
SUPERVISOR_NS=my-namespace
# T1 — guest cluster
kubectl get svc my-app -w
# T2 — Supervisor namespace
kubectl -n $SUPERVISOR_NS get virtualmachineservice,svc,endpoints -w
# T3 — the NSX reconciler. Find it first, the pod name varies by build:
kubectl -n vmware-system-nsx get pods
kubectl -n vmware-system-nsx logs -f <that-pod> --tail=50
T1 shows <pending>. T2 shows the VirtualMachineService appear, then the Service and Endpoints. T3 shows the reconciler working. Then T1 fills in. About fifteen seconds of wall clock, and the whole chain becomes obvious.
Prove the rewrite. This is the two-minute version that makes the entire packet diagram land:
kubectl create deploy whoami --image=traefik/whoami --replicas=3
kubectl expose deploy whoami --type=LoadBalancer --port=80
kubectl get svc whoami -w # wait for EXTERNAL-IP
VIP=$(kubectl get svc whoami -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
# Cluster (default): the rewrite inside the cluster
curl http://$VIP/ # RemoteAddr = the pod network gateway
# repeat — Hostname rotates across replicas
kubectl patch svc whoami -p '{"spec":{"externalTrafficPolicy":"Local"}}'
# Local: no rewrite inside the cluster — and still not you
curl http://$VIP/ # RemoteAddr = the VPC gateway LB address
# repeat — Hostname sticks to local backends
traefik/whoami prints both RemoteAddr and Hostname. Watch RemoteAddr change between the two runs. It moves from the pod network gateway to the load balancer’s address, and never to yours. That is a more interesting demo than the one I expected to be giving, because it shows the limit rather than a workaround.
If you want to go deeper, the tools split by layer: pktcap-uw on the ESXi host (use net-stats -l to find the switchport ID), plain tcpdump inside the node VM, and antctl inside the antrea-agent pod for the bridge itself. That last one is the most underused tool on the platform. It will show you the actual tables from the figures above, on your own cluster.
What I would actually do with this
Four design positions, ranked, with the one I would defend hardest first.
Size the External block against every consumer, not one. Count one VIP per guest cluster endpoint, one per LoadBalancer Service, and one outbound address per namespace, then add the growth rate of each. Everything else on this list can be fixed later. Running out of routable addresses mid-project cannot.
Put Services behind one layer 7 proxy instead of exposing each one. Ten microservices exposed as ten LoadBalancer Services is ten routable addresses. Behind one ingress it is one. Note what this does not buy you: an ingress inside the cluster still sits downstream of the VPC gateway rewrite, so it cannot see your client either. Consolidate for the address economics and the routing, not for the client’s address.
Be generous with private VPC CIDRs. They are free, they can repeat across VPCs, and a tight range is what stops a cluster growing eighteen months from now, when nobody remembers why the number was chosen.
Decide the pod-level security story explicitly, and early. Pod-to-pod traffic is enforced at the pod’s own interface, which is what the three-ways-out table means in practice. Either you accept NetworkPolicy inside the cluster as the enforcement point, or you deploy the Antrea–NSX adapter so those same policies are authored and audited from NSX. Both are defensible. Finding out which one you have during a security review is not.
None of this is exotic. It follows from a layered platform where each layer does its own job well and none of them can see the others. Once you can hold the whole path in your head at once, the policy question, the source-address question and the address-planning question all turn out to be the same picture from different angles.
Written against VCF 9.1, with the data-plane addresses measured in a live lab rather than assumed. The TTL argument in an earlier version of this post was overstated and has been corrected here. Defaults and object names move between releases, so verify against the version in front of you before designing to them. Corrections and disagreements welcome. I would rather fix this than have it be quietly wrong.
If you administer vSphere, you already know how to build a VM. This post shows you how to build the same VM in VMware Cloud Foundation Automation using nothing but a command line — no UI clicks, no cloud template, no prior Kubernetes experience assumed. It takes about less then 10 minutes, and by the end you’ll have a running web server with a load-balanced IP that answers from your browser.
I’m deliberately keeping this simple. There is a longer, deeper reference post for the full API surface, but this one is the on-ramp: the shortest honest path from “I’ve never used the VCF Automation API” to “I just built something with it.”
What we’re going to do
Install the CLI and log in to your organization
Create a project (the tenancy boundary for a team)
Create a Supervisor Namespace (where workloads actually live)
Create a VM with cloud-init that installs nginx
Publish it behind a load balancer and browse to it
Build a Kubernetes cluster in the same namespace, and run a workload on it
Why bother, when the UI works fine?
Fair question, and I’d give three honest answers.
Repeatability. A VM you clicked into existence is a VM nobody can rebuild exactly. A VM defined in a YAML file is one you can commit, review, and recreate identically a year later.
It’s where the platform is going. In VCF 9, the consumption layer is a Kubernetes-style API. The UI is a client of that API, not the other way round. Learning the objects underneath means the UI stops being magic.
It’s the foundation for automation. Everything you’d want later — GitOps, pipelines, self-service catalogs — is built on these same objects. This is the first rung.
None of that means you should stop using the UI. I use both. But knowing what’s underneath makes you considerably harder to stump.
Before you start
What
Where to get it
VCF CLI (the vcf command)
Broadcom Support portal — it replaces the older CCI kubectl plugin
kubectl
Any recent version; the CLI generates a kubeconfig for it
An API token
Log into your VCF Automation org portal → your user menu → API Tokens → Generate. Copy it — it’s shown once.
Org admin rights
Needed to create projects. If you only have consumer rights, skip to Step 4 and use an existing namespace.
Everything below is run from PowerShell on Windows. It works identically on Linux and macOS; only the quoting changes.
Step 1 — Log in
First, store your token in a variable. Note the quotes — without them PowerShell tries to execute the token as a command, which produces a confusing error.
$vcfa_token = "your-api-token-here"
Now create a context. This is the CLI’s word for “a saved connection to an environment.”
Substitute your own endpoint and tenant name. The backtick is PowerShell’s line-continuation character — or just put it all on one line. Use --insecure-skip-tls-verify only in a lab; in production supply your CA certificate with --ca-certificate instead.
A successful login. Note that the CLI lists every context you now have access to — one for the organization, plus one for each Supervisor Namespace you can reach.
That output is worth reading carefully, because it explains the whole model. You get an organization context (org2-ctgw) and a namespace context for each namespace you can use (org2-ctgw:org2-ns1-4hm32:default-project).
The one concept to take away
There are two levels. The organization context is where you manage tenancy — projects, namespaces, who gets what. The namespace context is where you build things — VMs, clusters, networks. Almost every “why can’t I see that?” moment is being in the wrong one. If a command returns “the server doesn’t have a resource type”, you’re at the org level asking a workload question.
Step 2 — Create a project
A project is the boundary you’d hand to a team. Save this as project.yml:
apiVersion: project.cci.vmware.com/v1alpha2
kind: Project
metadata:
name: team-phoenix
spec:
description: Phoenix dev team
kubectl --context org2-ctgw create -f project.yml
# project.project.cci.vmware.com/team-phoenix created
Use create, not apply
This will save you a puzzling ten minutes. kubectl apply stores a tracking annotation on the object, and this API rejects it outright with “Annotation updates are not supported.” Everywhere in this post, use kubectl create.
Step 3 — Create a namespace
The project is just a boundary. A Supervisor Namespace is the thing with actual capacity — CPU, memory, storage, a network — where your VMs will live. Think of it as the resource pool plus the network plus the quota, expressed as one object.
You need four pieces of information, all of which you can read off an existing namespace or get from whoever set up your environment:
Two things that catch people here
The namespace name is generated with a random suffix (phoenix-ns-kj9g6), so read it from the previous command rather than guessing. And the context name is the full three-part string <org>:<namespace>:<project> — using just the namespace name gives you “no context exists.”
You can confirm the same thing in the UI, which is a reassuring way to check your work while you’re learning:
The namespace created from the command line, shown in the VCF Automation UI — Active, in the team-phoenix project, with its CPU, memory and storage allocation. The API and the UI are two views of the same object.
Why the long classConfigOverrides block?
When you create a namespace through the portal, it fills these values in for you from the class. Creating one through the API, you have to state them yourself — and the zone entry needs all four CPU and memory fields, not just a name. If you leave them out, the API tells you exactly what’s missing, one field at a time. That’s actually a decent way to learn the object: submit a minimal version and let the error messages guide you.
Step 4 — Create a VM
Now the part that will feel familiar. We need three things: a size, an image, and a storage policy. Ask the namespace what it has:
kubectl get virtualmachineclasses # the sizes available to you
kubectl get virtualmachineimages # images from your content library
kubectl get clustervirtualmachineimages | Select-String "ubuntu"
If virtualmachineimages comes back empty, you’re not stuck.
That list is empty when no content library is attached to your namespace class. You can still deploy from a cluster-wide image by referencing its vmi- identifier directly, which is what I do below. Getting a content library attached is the better long-term answer — it gives you friendly names instead of hashes — but it isn’t a blocker today.
Here’s the VM, together with a cloud-init secret that creates a login and installs nginx. Save both in one file:
Two objects, one command: the cloud-init secret and the VM itself.
Watch it come up. The VM will report an IP address once VMware Tools starts reporting:
kubectl get vm web01
kubectl get vm web01 -o jsonpath='{.status.network.primaryIP4}'
# 172.30.0.66
Notice what you didn’t have to specify: no port group, no IP address, no datastore. The VM picked up an address from the namespace’s default network automatically and landed on storage matching the policy you named. That’s the platform doing the placement work, exactly as it does for a Kubernetes pod.
Cloud-init only runs on first boot
You cannot add a bootstrap section to a VM that already exists. If you forget it, the VM boots fine but has no user account and no way in — you delete and recreate. Decide your bootstrap before you provision.
Step 5 — Publish it
A VirtualMachineService puts a load-balanced IP in front of one or more VMs, selected by label. Our VM carries app: web, so the service will find it.
The load balancer service, created. NSX allocates the external address within seconds.
Reading the external IPkubectl get vmservice prints only name, type and age — the address isn’t in its default columns. Use kubectl get svc instead, or ask for the field directly with -o jsonpath='{.status.loadBalancer.ingress[0].ip}'.
Step 6 — Confirm it works
kubectl get svc
# web-lb LoadBalancer 172.29.255.24 40.60.0.4 80/TCP 2m31s
kubectl get endpoints
# web-lb 172.30.0.66:80 2m34s
curl.exe http://40.60.0.4
The whole chain proven: external IP 40.60.0.4, backend 172.30.0.66:80, and nginx answering.
That’s the full path working. The load balancer address is allocated from the NSX VPC, the endpoint is your VM’s own IP, and nginx — installed by cloud-init at first boot — is serving. Nothing in that sequence involved the UI.
Step 7 — Build a Kubernetes cluster
Here’s the part that surprises people: a Kubernetes cluster is just another object in the same namespace you’ve been working in. Same context, same kubectl create, same mental model. You describe the cluster you want and the platform builds the nodes.
You need to know two things first — and both are places where a sensible-looking manifest fails.
Where the cluster template lives. Clusters are built from a ClusterClass, which is a template your platform team maintains. Crucially, it does not live in your namespace:
kubectl get clusterclasses
# NAME VARIABLES READY AGE
# builtin-generic-v3.1.0 True 164m
# builtin-generic-v3.2.0 True 164m
# builtin-generic-v3.3.0 True 164m
That list is not the whole story. The class you actually want may live in a shared system namespace and won’t appear here at all — you reference it explicitly by name and namespace.
Which Kubernetes version you can use. Ask the platform rather than guessing:
kubectl get kubernetesreleases
You’ll get a long list. Only the rows showing bothREADY: True and COMPATIBLE: True are usable. Pick one of those.
Now the cluster itself — one control plane node, one worker, deliberately small:
kubectl create -f cluster.yml
kubectl get cluster phoenix-k8s -w
Provisioning takes five to six minutes for this size. PHASE: Provisioned appears early and only means the objects were created — the column to watch is AVAILABLE, which flips to True once the control plane is genuinely up. kubectl get machines gives you the per-node view while you wait.
Three things in that manifest are not obvious1.classRef needs a namespace. The ClusterClass lives in a shared system namespace, and without it you’ll get a “not found” even though the class plainly exists. 2. The version string is the long form — v1.35.5---vmware.1-vkr.1, with three dashes, not the +vmware.1 form shown elsewhere. 3. The resolve-os-image annotation is often required. If your environment has the same node image registered in more than one content library — common in labs that have been upgraded a few times — the resolver refuses to guess and fails with “Multiple OSImages resolved.” Adding content-library= narrows it to one. You’ll find the library ID in your Content Libraries list.
The shortcut worth knowing
Rather than assembling this by hand, start a cluster creation in the VCF Automation UI and look at the YAML it generates. It will show you the right ClusterClass namespace, the right version format, and any annotations your environment needs — all correct for your specific build. Copy that, then edit it. This is far quicker than reverse-engineering from error messages, and it’s how I arrived at the manifest above.
Step 8 — Get onto the cluster and run something
Once AVAILABLE is True, download the cluster’s kubeconfig from the VCF Automation UI (the cluster’s detail page offers it) and point kubectl at it:
kubectl --kubeconfig .\phoenix-k8s.yaml get nodes
# NAME STATUS ROLES AGE VERSION
# phoenix-k8s-... Ready control-plane 12m v1.35.5+vmware.1
# phoenix-k8s-...-np-1-... Ready <none> 9m v1.35.5+vmware.1
You’re now a cluster administrator on an ordinary, conformant Kubernetes cluster. Everything you know about Kubernetes applies from here.
With one important exception, which is worth meeting deliberately rather than by accident.
Your first deployment will probably be rejected — and that’s correct
Try the obvious thing:
kubectl --kubeconfig .\phoenix-k8s.yaml create deployment web --image=nginx --replicas=2
The deployment is created, but no pods ever appear. Looking at the ReplicaSet tells you why:
pods "web-..." is forbidden: violates PodSecurity "restricted:latest":
allowPrivilegeEscalation != false
unrestricted capabilities (must set capabilities.drop=["ALL"])
runAsNonRoot != true
seccompProfile (must set seccompProfile.type to "RuntimeDefault")
This is Pod Security Admission enforcing the restricted profile, and it’s switched on by default. A stock container image that runs as root is refused at creation — not warned about, refused. For a vSphere admin used to “deploy first, harden later,” this is a genuine change of habit, and a welcome one: the cluster will not run a privileged workload just because you forgot.
The fix is to declare what the workload actually needs. Each field below answers one of the four complaints, and the image is swapped for a non-root variant that listens on 8080 rather than binding port 80 as root:
kubectl --kubeconfig .\phoenix-k8s.yaml apply -f web.yml
kubectl --kubeconfig .\phoenix-k8s.yaml expose deployment web --port=80 --target-port=8080 --type=LoadBalancer
kubectl --kubeconfig .\phoenix-k8s.yaml get svc web
The external IP is allocated by the same NSX load balancer that served your VM in Step 5 — usually within seconds. Confirm there are backends behind it:
Note that kubectl apply works here.
The “use create, not apply” rule applies to the VCF Automation API only. Inside a VKS cluster you’re talking to a normal Kubernetes API server, and all your usual habits — apply, Helm, Kustomize — work exactly as expected.
The five things that will trip you up
These cost me time so they don’t have to cost you any.
Use kubectl create, never apply. The API rejects the annotation that apply depends on.
Context names are three parts.<org>:<namespace>:<project>, not just the namespace.
Know which level you’re at. “The server doesn’t have a resource type” almost always means you’re at the org context asking a workload question, or vice versa.
Tokens expire mid-session. A sudden Unauthorized, or a strange complaint about downloading the OpenAPI schema, usually just means your session lapsed. vcf context use <org> re-authenticates.
Check the API version against your build. Published examples lag. On the environment used here, namespaces are served at v1alpha3 and VMs at v1alpha5. The most reliable way to learn the current shape of any object is to dump a working one: kubectl get vm <name> -o yaml. Copy the structure, change the values.
That last one is worth internalising as a general habit. Rather than hunting for documentation that matches your exact release, clone something that already works in your own environment. It is always correct, by definition.
Where to go next
You now have the pattern for everything else on this platform. The same two-level model, the same create workflow, and the same “clone a working object” trick apply to Kubernetes clusters, persistent volumes, NSX security policies, and micro-segmentation rules — all of which are just more objects in the same API.
A few natural next steps, in the order I’d take them:
Delete and rebuild what you just made. Change the VM size in the YAML, recreate, and watch it come back identically. That’s the payoff of declarative infrastructure, and it’s more convincing when you do it than when you read about it.
Scale the cluster. Change replicas on the worker pool, re-apply, and watch nodes roll in. Then bump version to a newer compatible release and watch a rolling upgrade — that’s a one-line change doing something that used to be a project.
Write a security policy as YAML and watch NSX distributed firewall rules appear.
Put it all in Git. Once your infrastructure is files, a GitOps engine can reconcile it. That’s the subject of my next post.
If you’re a vSphere admin who has been eyeing all this with mild suspicion, I’d encourage you to run through the steps above once in a lab. The concepts are less foreign than the vocabulary suggests — a namespace is a resource pool with a network and a quota, a VM is still a VM, and a load balancer service is the thing you’d have raised a ticket for. The difference is that now it’s a file you own.
Questions or something that didn’t work in your environment? Leave a comment — I read all of them, and environment-specific quirks are exactly what makes these posts better.