In Part 1 I installed Regional Harbor and the Argo CD Service on VCF 9.1.1, and created an Argo CD instance with VCF Automation sign-in. Part 2 puts it to work.
The idea is simple to state. A DevOps user on VCF writes Kubernetes manifests, pushes them to Git, and gets a virtual machine, some pods, a whole Kubernetes cluster and an application running inside that cluster, without opening vCenter or NSX Manager and without holding a kubeconfig for the new cluster.
Four kinds of thing, one API, one Git workflow.
This post walks through how it fits together, the details worth getting right, and then a second application deployed and tuned entirely from the Argo CD GUI.
The application and where each tier runs
I used Yelb, a small restaurant-voting app that splits cleanly into tiers:
Tier
Runs as
Created from
Database (Postgres)
A VM Service virtual machine, Ubuntu 24.04, built by cloud-init
apps/ five pointers for Argo CD
10-yelb-db.yaml
20-yelb-appserver.yaml
30-yelb-vks-cluster.yaml
40-yelb-registration.yaml
50-yelb-ui-appset.yaml
infra/
db-vm/ VirtualMachine + cloud-init
appserver/ Deployments + LoadBalancer Service
vks-cluster/ Cluster
registration/ ManagedEntity
workloads/
yelb-ui/ Deployment + LoadBalancer Service
The only command run by hand is kubectl apply -f apps/. That creates four Argo CD Applications and one ApplicationSet. No infrastructure exists at that point; Argo CD pulls the repo and the VCF controllers build the rest.
Three of these kinds come from VCF: VirtualMachine, Cluster with the VKS ClusterClass, and ManagedEntity. That is the point: the platform’s own resources are Kubernetes objects, so they fit the same workflow as a Deployment. The VM class, image and storage policy names are the ones your platform team has assigned to the namespace.
The part worth showing live: a cluster that registers itself
Git never contains the new cluster’s address or credentials. Two small pieces connect it. A ManagedEntity points at the Cluster and asks for a label:
The Argo CD Service sees it, registers the cluster in Argo CD, and puts app: yelb on the cluster secret. An ordinary upstream ApplicationSet then deploys the UI to every cluster carrying that label:
The repository already has these built in. If you adapt it to your own environment, these are the details that matter.
Assign the storage policy to the namespace. A policy that exists on the Supervisor is not enough. The VM and the Cluster are admitted only with a storage class that is assigned to the namespace they are created in.
Give vSphere Pods an explicit CPU request. A vSphere Pod’s CPU request becomes a resource-pool reservation, and with no request set it takes the value of the limit. requests: { cpu: "0" } keeps small pods from reserving capacity they do not need.
After changing a manifest, start a fresh sync. An automated sync that is retrying stays on the commit it started with. Terminate the running operation and sync again to pick up the new commit.
Connect the tiers with hostAliases. Argo CD 3.x leaves EndpointSlices alone by default, so they are not a good bridge between runtimes. A hostAliases entry in the Deployment is plain, visible and managed like everything else.
Reach the database on the VM’s own address. The app server in a vSphere Pod talks to Postgres on the VM’s IP. The address lives in Git, so update it if you rebuild the VM.
Check the ManagedEntity once the cluster is ready. If it was created before the cluster finished building, touch it with an annotation and it reconciles within a minute.
Match the database’s authentication to the app’s client. Yelb’s database client predates SCRAM, the Postgres 16 default, so the cloud-init sets md5 for this app.
The last one shows the nicest property of the whole setup. To apply a new cloud-init, commit the change and run kubectl delete vm yelb-db. Argo CD notices the VM is missing and creates a new one from Git, with the new cloud-init. Rebuilding a server becomes a one-line operation.
The result
Everything Healthy and Synced.The UI in VKS, calling the app server in a vSphere Pod, reading restaurants from Postgres on the VM.Who talks to whom.
Day 2: more capacity is a commit
Next I wanted a bigger worker for a larger application. That was a few lines added to cluster.yaml and a push:
Argo CD synced the change and VKS added the node pool. The Cluster carries a Prune=false annotation, so Git can change it but removing the file would not delete it.
All three clusters in the namespace Available.
A twelve-service shop from the Argo CD GUI
Not everything needs a repository on day one. I deployed Google’s Online Boutique demo, twelve services, entirely from Argo CD’s New App form: no commit, no kubectl, no Helm CLI.
The cluster that registered itself is simply there to pick.Synced within seconds; Progressing while images pull.
NSX handed out a new load-balancer address, and the shop was up.
Served from yelb-vks through an NSX load balancer.
Tune it and change it from the same GUI
Two changes followed, both made under Details → Parameters → Edit in Argo CD.
The first was tuning. One of the twelve services, emailservice, needed more CPU than the chart’s default 200m limit to start inside its liveness window. I raised emailService.resources.limits.cpu to 1, and Argo CD replaced that one pod.
The second was a feature flag. Setting frontend.cymbalBranding to false rolled only the frontend and changed the shop’s branding.
Editing a Helm parameter in place.Two edits, two pods replaced. The other ten never restarted.After the branding change. The cart kept its item, because the cart lives in Redis.An order, end to end.
Git or GUI? Yelb lives entirely in Git. The Online Boutique application and its overrides live in Argo CD. The GUI is fast for an experiment; for anything that should last, move the overrides into a values file in a repository.
What this adds up to
One repository built a VM, vSphere Pods and a VKS cluster. The cluster registered itself, and the application arrived without anyone attaching anything. A second application went in from a browser and was tuned from the same place.
For a platform team, that is a private, declarative slice of the private cloud for each team. Requests become pull requests, drift is corrected from Git, and every change has an author and a diff. For the teams using it, virtual machines, pods and clusters all arrive through the one workflow they already know.
The repository is public. Clone it, change the names to your own namespace, image and storage policy, and try it.
Most VKS guides stop at “cluster created”. A production build has more to it: VLANs that must be proven before vSphere touches them, a private registry the cluster must trust from its first boot, and shared storage that pods can actually write to.
This post walks through a production build I did on VCF 9.1 with the Foundation Load Balancer, Harbor, a 7-node VKS cluster, NFS for ReadWriteMany volumes, and Istio, Contour and Headlamp on top. The commands that matter are all here, in the order you need them.
All names and IP addresses are examples from a made-up network plan, and the terminal screenshots were re-typed with those example values.
What is in this post
Before you start: what to have ready, including the full list of URLs to allow
Step 0: prove the network · Step 1: the jump server · Step 2: the Supervisor and load balancer
Step 3: Harbor · Step 4: the vSphere Namespace · Step 5: the cluster · Step 6: access
Step 7: ReadWriteMany storage · Step 8: Istio and Contour · Step 9: Headlamp · Final checks
What we are building
Three VLANs: management, Supervisor workload with the load balancer VIPs, and the VKS node network with the NFS server on it.
NFSv4.1 export with csi-driver-nfs (a Dell PowerStore in my case; any NFSv4 server works)
Add-ons
cert-manager, Istio, Contour (Gateway API CRDs ship with the VKr)
Console
Headlamp
All work is done from a Windows jump server with the VCF CLI, kubectl, Helm and crane. No Docker is needed anywhere. Every third-party image is copied into Harbor first, so the cluster pulls from one registry you control.
Supervisor workload network, Harbor, FLB data NICs (.200 to .209) and VIPs (.210 to .239)
300
10.30.0.0/26
VKS cluster nodes (.10 to .50) and the NFS server interface (.60)
Inside the cluster: pod CIDR 10.244.0.0/20 (room for 16 nodes) and service CIDR 10.100.0.0/24. Neither can be changed after the cluster exists, so check they don’t overlap anything the pods need to reach.
Before you start
Have these in place before Step 2:
Three VLANs trunked on every host uplink, with gateways, routes in both directions, and NAT for internet egress.
DNS records for the Supervisor API and for Harbor, created once their VIPs are known (the first VIP from the range goes to the Supervisor).
Internal firewall rules: load balancer data NICs → nodes (TCP 6443, 80, 443, 30000 to 32767); nodes → Harbor and Supervisor (443, 6443); jump server → VIPs and nodes (443, 6443, 22).
An NFS server with an interface on the node VLAN and NFSv4 enabled.
administrator@vsphere.local for every vCenter step, and a Broadcom Support Portal login for the downloads.
A Windows jump server with internet access, for the CLI, Helm and crane.
Internet allow list
This is the complete list of what had to be reachable, by source. Allow by FQDN or SNI, never by IP: every one of these is CDN-backed and the addresses rotate. If the firewall does TLS inspection, exclude these names or make the Supervisor, Harbor and the nodes trust the inspection CA. The CDN names are the ones that get missed: a rule for registry.k8s.io without *.pkg.dev gives you pulls that connect and then hang. To find a CDN host, run curl.exe -sI https://registry.k8s.io/v2/pause/manifests/3.10 and read the location: header.
Copying the NFS CSI driver and test images into Harbor with crane
Jump server
ghcr.io, pkg-containers.githubusercontent.com
Copying the Headlamp image into Harbor
Jump server
ocsp.digicert.com, crl3.digicert.com, crl4.digicert.com on TCP 80
Code-signing revocation checks; without them the CLI starts very slowly
Harbor (only if you use its proxy cache)
Docker Hub: registry-1.docker.io, auth.docker.io, production.cloudflare.docker.com · GHCR: ghcr.io, pkg-containers.githubusercontent.com · Quay: quay.io, cdn01.quay.io to cdn03.quay.io · registry.k8s.io with *.pkg.dev · your <name>.azurecr.io
Pulling upstream images through Harbor instead of copying them by hand, and the Trivy vulnerability database
Step 0: prove the network before vSphere uses it
Before you hand a VLAN to the Supervisor, put a small Linux VM on its port group, once on every host, and check:
the gateway answers,
a VM on another VLAN answers, and the reply comes back,
curl -I https://projects.packages.broadcom.com returns an HTTP status line.
Test every host, not just one: a VLAN missing from one host’s uplink trunk only breaks traffic between VMs on different hosts, which is hard to spot later. While the test VM is up, also check the allow list above from each VLAN that needs it.
Step 1: prepare the jump server
Download the VCF CLI for Windows and its plugin bundle from the Broadcom Support Portal. Unblock the downloaded files first, or Windows scans every CLI call and the tools run very slowly:
Get-ChildItem C:\tools -Recurse | Unblock-File
vcf plugin install all --local-source C:\tools\vcf-plugin-bundle
Trust the vCenter root CA in an administrator PowerShell:
tar -xzf C:\temp\crane.tar.gz -C C:\tools crane.exe
Step 2: enable the Supervisor with the Foundation Load Balancer
In the vSphere Client: Supervisor Management → Get Started, VDS networking. The values that matter:
Wizard page
Value
Load balancer
Foundation Load Balancer, One-Arm, Medium, 2 VMs
LB management network
Management port group, 10.10.0.29 to 10.10.0.32 (a couple of spare IPs beyond the two VMs)
LB virtual server network
Workload port group; data NIC IPs 10.20.0.200 to .209; VIP range 10.20.0.210 to .239 (the two ranges must not overlap)
Management network
Starting IP 10.10.0.24 (uses 5 in a row)
Workload network
10.20.0.40 to .99, service CIDR 10.96.0.0/23
Control plane size
Medium (it can grow later, never shrink)
How the Foundation Load Balancer uses its IPs
The wizard asks for three ranges, and each has one job. Management IPs (one per LB VM) are how the Supervisor configures the load balancer. Data NIC IPs (one per LB VM) are the source address of all traffic the load balancer sends to backends, including health checks. The VIP range is handed out on demand: every Service of type LoadBalancer, on the Supervisor or inside a cluster, takes the next free VIP and gives it back when deleted. Size it for one Supervisor API, one per Supervisor Service, one per cluster and one per app ingress, plus headroom.
The three FLB IP ranges, which NIC each lands on, and how a VIP is assigned to a LoadBalancer Service.
At run time, clients only ever connect to a VIP. The active LB VM forwards the connection to a control-plane node on 6443 (Kubernetes API) or to a worker NodePort (apps), and rewrites the source to its data NIC IP, so replies always come back through it. Two consequences: inbound firewall rules only need LB data NICs → nodes, and pod logs show the data NIC IP as the client. In the other direction, traffic leaving a pod is NATed to the node’s IP, so outbound firewall rules use the node subnet as the source, never the pod or service CIDRs.
Traffic paths: Kubernetes API (A), applications through a NodePort (B), and replies returning to the LB data NIC (C).
When the Supervisor is Running, the first VIP (.210) is the API. Log in:
Without a VCF Operations software depot, use Harbor’s legacy Supervisor Service manifest. Register it under Supervisor Management → Services, then install it with a data-values file. The keys that matter:
hostname: harbor.corp.example.com
enableNginxLoadBalancer: true
enableContourHttpProxy: false
harborAdminPassword: "<strong password>"
secretKey: "<exactly 16 characters>"
core:
xsrfKey: "<exactly 32 characters>"
Harbor runs as vSphere Pods on the Supervisor workload network, so that subnet needs internet egress and a working return route for the image pulls. Once the pods are Running, save Harbor’s CA; every cluster needs it to trust the registry:
$b64 = kubectl get secret harbor-tls -n <svc-harbor-namespace> -o jsonpath="{.data.ca\.crt}"
In the Harbor UI, create a Public project called k8s for the third-party images that come later, and log crane in: crane auth login harbor.corp.example.com -u admin --password-stdin.
Step 4: the vSphere Namespace
Add a second workload network for the node VLAN (10.30.0.10 to .50) under Supervisor → Configure → Network → Workload Network.
Create a subscribed content library for Kubernetes releases (https://wp-content.broadcom.com/v2/latest/lib.json) and synchronize the version you will use, so it is stored locally.
Create custom VM classes: cp-12c-48g (12 vCPU / 48 GB) and worker-24c-64g (24 vCPU / 64 GB).
Create the namespace apps on the node network (it can only be chosen at creation), then add the storage policy, the VM classes and the content library.
Step 5: the cluster
First, the Harbor trust secret. Its name must be <cluster>-user-trusted-ca-secret, and it must exist before the cluster is applied. Deleting a cluster removes its secret too, so recreate it before any re-apply:
Then the cluster. On VKS 3.x, trust goes under the osConfiguration variable; the old top-level trust variable is rejected:
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
name: prod-vks01
namespace: apps
spec:
clusterNetwork:
pods:
cidrBlocks: ["10.244.0.0/20"]
services:
cidrBlocks: ["10.100.0.0/24"]
serviceDomain: cluster.local
topology:
class: builtin-generic-v3.6.0
classNamespace: vmware-system-vks-public
version: v1.35.5---vmware.1-vkr.1
controlPlane:
replicas: 3
workers:
machineDeployments:
- class: node-pool
name: np1
replicas: 4
variables:
overrides:
- name: vmClass
value: worker-24c-64g
- name: volumes
value:
- name: containerd
mountPath: /var/lib/containerd
capacity: 500Gi
storageClass: vks-storage-policy
variables:
- name: vmClass
value: cp-12c-48g
- name: storageClass
value: vks-storage-policy
- name: osConfiguration
value:
trust:
additionalTrustedCAs:
- caCert:
secretRef:
name: prod-vks01-user-trusted-ca-secret
key: harbor-ca
The volumes override gives each worker a dedicated 500 GB disk for container images, so large images never fill the root disk.
kubectl apply -f C:\temp\prod-vks01.yaml
kubectl get cluster,machines -n apps
Two minutes after the apply, check that the cluster’s endpoint matches its load balancer service: kubectl get cluster prod-vks01 -n apps -o jsonpath='{.spec.controlPlaneEndpoint.host}' must equal the EXTERNAL-IP of kubectl get svc prod-vks01 -n apps. If you ever rebuild, give the new cluster a name that has not been used in that namespace before.
The cluster with 3/3 control-plane and 4/4 worker machines.
In about 20 minutes the cluster shows AVAILABLE True. Log in:
Everyone logs in with their own directory account through the vSphere Namespace, and the namespace role decides what they get. Grant the roles from the namespace’s Summary → Permissions card as administrator@vsphere.local:
Namespace role
Inside the cluster
Give to
Can view
Login only; rights come from RoleBindings you add
Application teams
Can edit
cluster-admin, created automatically
Platform admins
For each application, create its namespace, set a Pod Security level, and bind the team’s group to the built-in edit role there. VKS clusters enforce restricted by default, which refuses most images that run as root; baseline suits most applications:
An NFS datastore mounted on ESXi gives pods only ReadWriteOnce volumes. For volumes shared across pods, the cluster needs an NFS CSI driver talking to the NFS server directly. Putting the server’s interface on the node VLAN keeps NFS off routers and firewalls entirely.
On the NFS server, create the export /k8s-rwx with NFSv4 enabled, and give host access to the whole node subnet (10.30.0.0/26) as read/write with root allowed (no root squash). The subnet rather than node IPs, because nodes are replaced on every upgrade; root, because the driver creates a folder per volume as root.
What the export needs: node subnet access, read/write, root allowed, NFSv4.1 on the node VLAN.
Test a write from a worker before installing anything. The mount succeeds even when root is squashed; only the write tells you the export is right:
The CSI controller (5/5) and one node pod per node (3/3), all pulled from Harbor.
Then the StorageClass. Retain keeps the data folder on the NFS server when a volume is deleted, which is what you want in production:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-rwx
provisioner: nfs.csi.k8s.io
parameters:
server: 10.30.0.60
share: /k8s-rwx
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true
mountOptions:
- nfsvers=4.1
- hard
Prove it with a two-replica Deployment, forced onto different nodes with pod anti-affinity, both appending to the same file on one ReadWriteMany PVC. If tail shows both pod names, RWX works across nodes. Note that this driver does not enforce a size per volume; everything shares the export’s capacity.
Step 8: Istio, Contour and Gateway API
VKS add-ons are installed from the Supervisor namespace context, not from inside the cluster:
There is no need for the gateway-api add-on on VKr v1.35.5: the Gateway API CRDs ship with the Kubernetes release, and Istio’s istio GatewayClass is accepted straight away. Check with kubectl get crd | Select-String gateway.networking before installing it on other releases.
Gateway API CRDs present from cluster creation, and the istio GatewayClass accepted.
Contour’s Envoy defaults to NodePort. To get a LoadBalancer VIP, pass a values file, which requires an explicit version:
A fresh Envoy resets connections until at least one Ingress or HTTPProxy exists, so a connection reset on a new ingress VIP is expected. To test Istio end to end, apply a Gateway (class istio, port 80), a small web Deployment and Service, and an HTTPRoute pointing at it. Istio creates the gateway pod and its LoadBalancer service from the Gateway itself; when the Gateway shows PROGRAMMED True, curl its address.
Step 9: Headlamp as the cluster console
Headlamp gives the team a browser view of workloads, logs and events without handing out kubeconfigs. Same pattern as the NFS driver: copy the image into Harbor, install with Helm, expose on a VIP. The chart already creates a ClusterRoleBinding called headlamp-admin, so the login account’s binding needs a different name:
Open http://<EXTERNAL-IP>, choose Token, and paste it. For a production URL, put it behind Contour with a hostname and a certificate, and switch the Service back to ClusterIP.
Final checks
Before handing the platform to application teams, confirm all of these:
kubectl get cluster -n apps shows AVAILABLE True, and 7 nodes are Ready on the expected IPs.
The cluster’s controlPlaneEndpoint equals the EXTERNAL-IP of its Service on the Supervisor.
The two-pod RWX test shows both pod names in the shared file.
vcf addon install list shows cert-manager, Istio and Contour READY, and Contour’s envoy Service has a VIP.
The Istio gateway test returns the test page from its VIP.
A platform admin logs in with a directory account and kubectl auth can-i '*' '*' -A prints yes; an application user can deploy in their namespace only.
Test namespaces are deleted and their VIPs released. All LoadBalancer Services share the same 30-address pool.
If you are building something similar and run into something this post doesn’t cover, let me know in the comments.
VKS makes it easy to give every team its own Kubernetes cluster: VCF Automation hands them out, Cluster API keeps them reconciled, and the Supervisor runs them on the vSphere you already operate. As the number of clusters grows, the natural next step is to see the whole fleet at once: which clusters need attention, what is about to run out, and what a change will do before you make it. That is what I built vks-fleet for.
vks-fleet is an open-source plugin for Headlamp that brings the Supervisor, vCenter, each cluster’s API and its Prometheus together in one view, for operators, read-only admins and tenants alike.
A personal project. vks-fleet is something I built in my own time. It is not a Broadcom or VMware product, not endorsed or supported by either, and it uses only public APIs. It is Apache 2.0, and the code is on GitHub.
The fleet dashboard, in demo mode: a score, what needs you now, eight blocks that each open their page, and every cluster as a tile.
Day-2 questions at fleet scale
VKS gives you a lot out of the box. Cluster API reconciles your clusters, the Supervisor carries the control plane, and VCF Automation hands out self-service. Once there are many clusters, operators start asking fleet-level questions like these:
Which of my clusters needs attention right now, and which of those is actually urgent?
A node has been “deleting” for a day. Is that a PodDisruptionBudget, a volume that won’t detach, or the ESXi host underneath?
Which disk, volume or certificate runs out in the next 30 days?
If I upgrade this cluster, or scale this pool down, what breaks, what moves, and does the namespace have room while it rolls?
What is open security-wise across every cluster, not just the one I happen to be logged into?
Every answer is already there, spread across a kubectl session per cluster, vCenter, the Supervisor’s own namespaces and each cluster’s Prometheus. vks-fleet brings them onto one screen, so the same questions take seconds whether you run three clusters or thirty.
Why a Headlamp plugin
I started out thinking about forking a Kubernetes UI and went the other way. Headlamp is an open-source Kubernetes web UI with a proper plugin system: it already handles sign-in, kubeconfig contexts, per-cluster views, logs, shells and YAML. A plugin can add pages, read any cluster Headlamp is signed in to, and hand off to Headlamp’s own views when you want the raw objects. That meant I could spend the effort on the fleet layer and let Headlamp do the rest.
It also keeps the footprint small. The plugin runs in your browser with Headlamp’s sign-ins. There is no server of its own, no database, no agent in your clusters and no telemetry.
How it is built to work
The plugin follows the way an operator actually works through a problem:
ObserveOne dashboard for the fleet, a score with its trend, and what needs you now.
DetectRanked issues, each with its cause, and the steps that clear the most of them.
ExplainA cluster’s timeline beside a walk down the layers under a node.
PredictWhat runs out in 30 days, and what a change will do before you start it.
ActControlled changes, each with its checks and a server-side dry run.
Watch it in a minute: a one-minute walk through demo mode (the dashboard, investigating a node, a fix with its dry run, pre-flight, Supervisor health, security and what to do first) is at the top of the project’s README.
Observe: one screen for the fleet
The fleet page opens on a dashboard. At the top is a fleet score (the average of each cluster’s best-practice checks) with a 30-day trend, and a short Needs you now list that puts time-bound problems first: a node disk that fills in two days outranks a warning that has been there for a month.
Below it are eight blocks: clusters, Supervisor, capacity, the next 30 days, lifecycle, security, governance and network. Each is one number with one line of context, and each opens the page that owns the detail. Under them, every cluster is a tile on a wall, coloured by its worst open problem and grouped by org.
Detect: issues with a cause, not just a list of findings
The plugin folds raw signals into issues that say what is wrong, why, and what to do. A stuck node, a drain blocked by a PodDisruptionBudget, a crash-looping pod, a certificate close to expiry, quota and memory overcommit, controllers that stopped reconciling, Supervisor services all running on one host, packages that fail to reconcile, backups that stopped: each becomes one line with its fix or the decision it needs.
On top of that sits Do these first: the recommended steps that clear the most issues, ranked by effect, each opening where it is done. A Simulate view shows the fleet as it would look with the fixes applied, so you can see the payoff before spending the change window.
Issues: the steps to do first, then every open issue with its fix or the decision it needs.
Explain: from a pod down to the ESXi host
This is the part I use most. Investigate puts one cluster’s timeline (changes, alerts, events and anomalies) beside a walk down the layers under a node: pod, node, Machine, VM, ESXi host, namespace and Supervisor. It names the deepest layer in trouble, so you know whether you are looking at a Kubernetes problem, a VM problem or a host problem. When two troubled nodes share a host, it says so. It also drafts the post-mortem from the same timeline.
Investigate: what happened, beside every layer under the node. Here the deepest layer in trouble is the Machine, stuck deleting.
Predict: the next 30 days, and pre-flight
Forecasts from each cluster’s Prometheus (node disks, memory, volumes), the Supervisor’s control-plane disk from vCenter, and certificate expiries all land on one 30-day timeline.
Pre-flight answers what will this change do? before an upgrade, a node pool scale or a VM class change. It works out which nodes are replaced, which pods move, which single-replica workloads take a brief outage, which disruption budgets or pinned pods would block it, whether the namespace has room for the extra nodes while it rolls, and whether the Supervisor itself is healthy enough to start. Then it gives a verdict: go ahead, go ahead with care, or stop.
Pre-flight for scaling a pool down by one. The verdict is Stop: a node in the pool is already stuck deleting.
For upgrades across many clusters there is an upgrade planner that arranges clusters in waves (a small canary first, then the rest) with the same checks for each cluster.
Act: controlled changes
Upgrades, node pool scaling, node replacement, drain timeouts, package updates, VM power operations and new clusters can all be started from the plugin. Every change shows its checks first and runs a server-side dry run (?dryRun=All), so anything the API or the VKS admission webhooks would refuse is caught before anything changes. Destructive actions ask you to type the cluster’s name.
For shared installations there is a read by default, elevate to change mode: viewing uses a read-only sign-in, and a change needs a time-limited elevation with a reason, recorded on the object it changed.
Supervisor health
Clusters are only as healthy as the Supervisor running them. A Supervisor page shows the control plane, the ESXi hosts and what runs on each, the leader-election leases of the controllers that manage your clusters (a lease that stops renewing means a controller is stuck, even when its pod says Running), Supervisor services, and the reconcile backlog. Click a host and it shows what that host failing would take down. With an optional, read-only collector it also reads vCenter: configuration status, alarms, utilisation and the control-plane VM’s disks.
Supervisor health: the hosts, what runs on each, and what vCenter reports.
Security: what is open, across every cluster
The Security overview shows what is open across every cluster you are signed in to: privileged namespaces, privileged and host-level pods, cluster-admin grants, and image vulnerabilities from Trivy Operator, counted once per image and split into your images and the ones VKS ships. Policy results from Kyverno or any engine that writes PolicyReports appear beside them. A cluster without a scanner reads not measured, never zero.
Security overview: what is open across the fleet, then one row per cluster.
Principles behind it
Principle
What it means in practice
Public APIs only
Cluster API, VM Operator, NSX VPC, Carvel, Prometheus, and vCenter through a small read-only collector. No private interfaces.
Nothing installed in your clusters
Everything is read through the APIs that are already there. Trivy, Kyverno and Prometheus are used when present, never required.
Your RBAC decides what you see
Operators see every org, read-only admins see everything and change nothing, tenants see only their own org. It follows the account Headlamp signs in with, not a mode in the UI.
Honest numbers
Data that can’t be read is “not measured”, not zero or passing. Every figure says where it came from.
Dry run before every change
Checks, then a server-side dry run, then confirmation. If a multi-step change fails partway, it says which steps took effect.
Try it without a lab
The fastest way to see it is demo mode: a fictional fleet with realistic problems built in, where nothing is ever changed. Install the plugin, open VKS fleet and choose Try the demo. You can scale the demo fleet up to 50 clusters to see how the pages behave at size.
Headlamp desktop app or Docker: download vks-fleet.tar.gz from the latest release, extract it into Headlamp’s plugins folder, restart Headlamp and open VKS fleet.
In a cluster, with Helm: the chart installs Headlamp, the plugin, and a job that keeps the Supervisor sign-in fresh. The quick start has the full commands, including trusting vCenter’s CA so certificates are verified; the core of it is:
Then connect your Supervisor under Settings → Plugins → vks-fleet. It is also listed on Artifact Hub as a Headlamp plugin.
Where it stands, honestly
It is developed and tested against VCF 9.1 with VKS 3.7 on Headlamp 0.45, on a real lab with one Supervisor, two VKS clusters and VCF Automation tenants. Every read and view has been checked there, as have the vCenter collector and changes through the VKS admission webhooks: scaling node pools up and down, pausing and resuming clusters, and a clean Helm install under restricted Pod Security with verified TLS.
What is not yet confirmed on a live system: deletes through the plugin, drain timeouts on machines that are already deleting, and some links into Headlamp’s own pages. The checks also run from the browser, which suits tens of clusters; demo mode exercises the pages at 50, but a real fleet has so far been tested at two. The README’s validation status keeps the current list.
If you run VKS, I would genuinely like to hear how it behaves on your fleet, especially with more clusters, more than one Supervisor, or tenants coming in through VCF Automation.
vks-fleet is a personal open-source project under the Apache 2.0 licence. It is not affiliated with, endorsed by, or supported by Broadcom or VMware. Product names are used only to describe what it works with.
A VKS cluster is built in layers, and different teams own different layers. A pod sits below anything kubectl will show you, and above anything vCenter will show you. That is true of Kubernetes on any virtual infrastructure, but it does mean very few people ever see the whole picture at once. This post walks the entire path, one layer at a time: what happens inside the node, what happens inside the virtual switch in the guest, and what happens when someone exposes a pod with a LoadBalancer Service and a user connects to it from outside.
I have drawn this on a whiteboard often enough that it seemed worth writing down properly. If you have ever had to answer one of these, this post is for you:
Where should a pod-level firewall rule actually be written?
Why does my application log every request as coming from the same address?
Is AntreaProxy a process? What happens to my traffic if it restarts?
How does a LoadBalancer Service actually get its address?
Where does a packet really go when one pod talks to another?
Every one of those is the same question in a different costume: where does the packet go, and who rewrites it on the way?
One note before we start. Networking people say SNAT when they mean “rewrite the source address” and DNAT when they mean “rewrite the destination address.” Both appear a lot below, and almost everything confusing in this post comes down to one of them happening somewhere you did not expect.
The whole path, bottom up
Read this from the bottom up, in the direction an arriving packet travels. Everything here is the NSX overlay design. The other switching model is a post of its own.
Count the layers as you go. That number explains most “why is this so hard to troubleshoot” conversations on this platform.
The VKS data path on NSX overlay networking. Three hypervisor hops, and four inside the guest where vCenter can no longer see them.
What the diagram makes obvious is that the path crosses three ownership boundaries: the guest operating system and its CNI, the ESXi virtual switch, and the physical network. Each one is well instrumented on its own. None of them shows you the other two. That is the real reason a networking question here is hard to answer from a single console, and it is why this post walks the whole path instead of any one layer of it.
Inside the node: one switch, everything plugged into it
Now zoom in on the guest. The correction I make most often in a room is this: br-intis the OVS bridge. There is no OVS on one side and a bridge on the other. There is one bridge, it is called br-int, and everything plugs into it.
One node, drawn honestly. Each pod’s veth peer is a port on br-int. antrea-gw0 is the pod network gateway; antrea-tun0 carries pod traffic between nodes.
Every pod is connected by a veth pair, which is best thought of as a short virtual cable. One end is the pod’s eth0. The other end is a port on br-int. Two other ports matter: antrea-gw0 is the pod network’s gateway on this node, and antrea-tun0 is the tunnel port used only when a pod talks to a pod on a different node.
Three things live on that bridge, and it is worth being precise about which does what, because the names mislead:
Component
What it actually is
ovs-vswitchd
The user-space daemon that holds the flow tables. The kernel module underneath is what actually moves packets.
antrea-agent
Control plane only. It writes the rules. It never touches a packet. If it stops, existing traffic keeps flowing.
AntreaProxy
Not a process at all. It is a set of rules in the bridge: one table picks a backend pod, the next rewrites the destination to that pod’s address. This is what replaces kube-proxy.
The correction that lands hardest. There is no proxy process in the data path. When people say “Antrea does proxying on the bridge,” what actually happens is that a packet matches a rule and has its destination rewritten, in the kernel, with nothing in user space involved. That is why AntreaProxy costs almost nothing, and why restarting antrea-agent does not drop traffic.
Inside br-int: the tables a packet walks
This is the level almost nobody goes to, and it is the level where the rest of the post stops being mysterious. A packet enters at a port, walks through a series of tables in order, and leaves at a port. That is the whole model.
The pipeline inside the bridge. A packet enters at a port, walks the tables in order, and leaves at a port. Service resolution and NetworkPolicy are both just tables in that sequence.
Two things are worth noticing about that sequence.
First, a Service is just a table. A ClusterIP is not an address that anything holds. No network card anywhere is configured with it. It is only something to match on. The packet arrives addressed to the ClusterIP, hits the table that handles Services, one backend pod is selected, and the destination is rewritten to that pod’s real address. Ping a ClusterIP and you may well get nothing back. That is not a fault. You are pinging something that was never meant to answer.
Second, NetworkPolicy is also just a table, and it sits in this same pipeline, right at the pod’s own interface. Where it sits matters more than it sounds, and it comes back later.
Three ways out, and that is all of pod networking
Once you can see the bridge, every packet a pod sends ends in one of exactly three ways. There is no fourth.
Every packet a pod sends ends one of three ways — and each one tells you where its policy belongs.
Destination
What happens
What that means
Pod on the same node
Straight out another port on the same bridge. The packet never reaches a vNIC at all.
Policy for this hop has to live where the traffic is: NetworkPolicy in the bridge, at the pod’s own interface.
Pod on another node
Out antrea-tun0, wrapped in a tunnel header, to the tunnel port on the other node.
At the vNIC it looks like UDP 6081 between two node addresses, so pod-level policy again belongs in the bridge.
Anything outside the cluster
Out antrea-gw0, then the node’s eth0, with the source address rewritten on the way.
The far end sees an infrastructure address instead of the pod, the same as any other masqueraded traffic.
That table is worth reading twice, because it tells you where to write policy. A firewall at the vNIC — on any platform, with any CNI — can only act on traffic it can see, and two of those three paths either never reach a vNIC or arrive inside a tunnel. So pod-level policy belongs at the pod’s own interface, which is exactly where Antrea puts it.
You can write it with kubectl, or author and audit it in NSX and let the Antrea–NSX adapter push it down. Either way the bridge is what enforces it, so you get one policy plane for pods whichever door you came in through. The controls at the vNIC stay for what they are genuinely good at: keeping one cluster’s nodes away from another’s, and limiting both to the Supervisor and the registry.
The three address blocks
The packet walk below needs three names, so here they are in the shortest useful form. An NSX VPC draws addresses from three separate blocks:
Block
What comes from it
Routable?
External
LoadBalancer VIPs, cluster API endpoints, outbound SNAT addresses
Yes — size this one deliberately
Private transit gateway
Addresses reachable across VPCs in the same project
Internally
Private VPC CIDR
Node VMs, and the slice of pod addresses given to each node
No
How those get allocated, and what you commit to on the Workload Network page of the activation wizard, deserves its own post. For now: VIPs come from External, pods and nodes come from the private VPC CIDR, and the VPC gateway sits between the two.
Exposing a pod: what a LoadBalancer Service really does
There are two separate stories here, and merging them is what makes this hard to explain. First an address gets allocated, which is a chain of controllers across three clusters and never touches a packet. Then a client actually connects.
Part one: allocation
The guest cluster has no load balancer controller of its own. When someone applies a Service of type LoadBalancer:
The paravirtual cloud provider, running on the guest cluster’s control plane, notices it. It holds scoped credentials to the Supervisor.
It creates a VirtualMachineService in the Supervisor namespace, selecting that cluster’s node VMs.
VM Operator turns that into a Service plus an Endpoints object whose members are the node VM addresses.
The NSX reconciler watches Supervisor Services and calls the NSX API. Classic deployments run NCP; VPC-mode builds may run the NSX VPC Operator instead. Worth checking which one you have, although the NSX objects are named with an ncp- prefix either way, so the naming will not tell you.
NSX creates a virtual server and a server pool on the VPC gateway, and allocates a VIP from the External block.
The address travels back down the same chain into the guest Service’s status.loadBalancer.ingress.
This is why <pending> is almost never a guest-cluster problem. There is no controller in that cluster that could allocate an address in the first place. If it is stuck, the chain broke further up. Check the VirtualMachineService in the Supervisor namespace first, then the NSX reconciler, then whether the External block has any free addresses left.
Part two: the packet
Now a client connects to the VIP. The left column is where the packet is. The right column is what its addresses look like at that exact point, with anything rewritten highlighted.
Four rewrites — and the one that destroys the client address happens at the load balancer, before Kubernetes ever sees the packet.
The rewrite that matters happens earlier than most people expect. The load balancer changes the source address as well as the destination, so the client’s address is already gone at the VPC gateway. That is before the packet has reached a node, and long before Kubernetes has any say in it. It is also why your access logs are full of the same address over and over.
What the pod sees as the source address
The standard advice is that externalTrafficPolicy: Local preserves the client’s address. On this path it does not. It only changes which infrastructure address the pod sees, and I only know that because I went and measured it.
First, where this setting actually lives
externalTrafficPolicy is a field on the Kubernetes Service. You set it with kubectl, in YAML, next to type: LoadBalancer. You never set it in NSX Manager. It is plain upstream Kubernetes, not a VMware addition, and it works the same way on EKS, GKE, OpenShift, or a bare-metal cluster running MetalLB. If you already know this setting from another platform, you know it here.
What is platform-specific is how the setting gets carried out. Choosing Local makes Kubernetes allocate a healthCheckNodePort. Every node then answers a simple health check on that port: 200 if it is running a pod for this Service, 503 if it is not. The controller that reconciles the Service turns that into two things on the load balancer, its pool membership and its health monitor. Nodes that answer 503 fail the check and stop receiving traffic.
So NSX Manager is where you see the result, not where you configure it. If you edit that pool by hand, it will be put back. The Service object is the source of truth.
Where externalTrafficPolicy sits. It controls the hop inside the cluster — the second rewrite, not the first. Addresses are from the lab measurement later in this post.
What each setting does
Cluster — the default
Every node accepts the traffic, whether or not it is running a pod for this Service.
If the chosen pod is on a different node, the packet is forwarded on, and the source is rewritten to antrea-gw0, the pod network’s gateway on the node that received it.
Load spreads evenly across pods. An extra node-to-node hop is common on clusters with several nodes.
The pod sees the pod network gateway.
Local — opt in
Only nodes actually running a pod for this Service answer, decided by the health check described above.
No extra hop and no second rewrite inside the cluster. Both are real benefits.
Traffic is spread across nodes rather than across pods, so uneven scheduling gives uneven load.
The pod still does not see the client. It sees the load balancer’s own address.
Here is the measurement, from an NSX VPC lab. Three replicas of traefik/whoami behind a LoadBalancer Service, called with curl from an admin host outside the cluster:
externalTrafficPolicy: Cluster
RemoteAddr: 192.168.145.1 ← antrea-gw0, the rewrite inside the cluster
externalTrafficPolicy: Local
RemoteAddr: 100.64.0.1 ← the VPC gateway load balancer
real client address
never visible on this path at all
That node’s pod addresses came from 192.168.145.0/24, so .1 is its antrea-gw0. And 100.64.0.1 is not a guess. The NSX server pool for that Service shows SNAT Translation Mode: Automap, which replaces the source with the load balancer’s own address before handing the packet to a pool member. Every pool the reconciler creates has it set.
The virtual server is also plain TCP, on a Fast TCP profile, with no awareness of HTTP. So the source address the pod reads is the load balancer’s, not the caller’s. That is how any layer 4 load balancer doing source NAT behaves, on any platform, and it is exactly why terminating at layer 7 exists as a design pattern.
One clarification, because this is easy to overstate. The reconciler does create an HTTP profile on the same load balancer, alongside the TCP and UDP ones. The VPC load balancer is perfectly capable of layer 7. But that profile is used by virtual servers created from Ingress objects. A Service of type LoadBalancer always gets the layer 4 one. The limit is on the Service path, not on the load balancer.
The design consequence. The rewrite happens one hop before Kubernetes sees the packet, which makes this an architecture decision rather than a Service setting. The same is true of any layer 4 load balancer doing source NAT, wherever it runs. If an application needs the caller’s address, put something that terminates at layer 7 in front of the rewrite, where the client is still visible. Decide that alongside the ingress design, not when an application team asks the question.
This is worth stating plainly, because the usual Kubernetes advice about Local assumes a load balancer that is not rewriting the source. It is not a defect. It is the documented behaviour of a layer 4 virtual server whose pool is set to Automap. But it does mean Local buys you a shorter path and one less rewrite, not the client’s address.
Automap is a property of the pool, and NSX supports other modes, so in principle it is configurable. In practice these pools are created and maintained for you, so editing them by hand is not a supported workaround. Treat it as a property of the platform and design around it.
So when is Local worth setting?
Not for the client’s address. That is the whole point of the measurement above. But there are two honest reasons to use it. It removes a node-to-node hop, which is worth having on a latency-sensitive path and keeps that traffic off the tunnel between nodes. And it removes one layer of rewriting, which makes a packet capture much easier to read.
The cost is load distribution, and the arithmetic is worth doing. Take three nodes and four replicas: two on node B, one on node C, none on node A. The load balancer sees two healthy nodes and splits the traffic 50/50. Node B’s two pods share its half, so each gets a quarter. Node C’s single pod takes the other half on its own. That one pod is now handling twice the traffic of its peers, and nothing in Kubernetes will correct it. With Cluster, traffic spreads across all three nodes and then across all four pods, so each gets roughly a quarter, at the price of that extra hop.
So Local fits best where pod placement is already even by design: a workload that runs one pod per node, or one where you actually enforce a topology spread constraint. Set it on purpose, not by habit.
Verified two ways
What an application reports is suggestive, not proof. traefik/whoami tells you what its own socket says, which is one tool’s opinion. So I checked the same thing a completely different way, using Antrea Traceflow in liveTraffic mode. That captures the real frame inside the bridge instead of asking an application about it.
Two unrelated tools, the same two addresses. The load balancer’s Automap rewrite under Local, antrea-gw0 under Cluster, and in neither case the client.
The TTL is 123 in both captures. That is consistent with the rewrite inside the cluster being done by a rule in the bridge rather than being a routing hop. I used to present it as proof, and a reviewer was right to push back: an identical TTL does not prove no proxy was involved, because a proxy ends one connection and starts a new packet with its own TTL anyway. What actually settles it is the pipeline itself and the Traceflow capture, which show the packet being rewritten in the bridge with nothing in user space terminating anything. The TTL supports the conclusion; it does not carry it. (For the curious: starting from a Windows client’s 128, five hops were taken off on the way in, which matches the fabric, T0, transit gateway and VPC gateway.)
This also answers “can I see this flow with the product tooling?” Partly. Antrea Traceflow covers the half inside the guest and will capture live traffic. NSX Traceflow draws the overlay half, but cannot follow a VIP through the load balancer, because it injects a packet between two ports and a virtual server has no port to inject at. Neither tool does the whole path on its own. With the Antrea–NSX adapter registered, Antrea’s results appear in the NSX UI, which is the closest thing to one view across both.
The direction nobody asks about: outbound
Everything above is traffic arriving. Traffic leaving behaves differently, and it is routinely overlooked right up until a partner’s firewall gets involved.
A pod calling something outside leaves through antrea-gw0, out the node’s eth0, and through the VPC gateway, which rewrites the source again. So the mirror image of the client-address story applies: coming in, your pod cannot see who called it; going out, the far end cannot see which pod called them. Same mechanism, opposite direction. Neither is a fault. Both are what a rewrite does.
Which means that when a partner asks you to allow-list “the payments pod” at their firewall, by default you cannot. With the standard VPC gateway rewrite, the finest identity available at that boundary is the namespace, and only where Antrea Egress is in use. Without it, traffic leaves as the node’s address and even that distinction disappears.
Worth knowing that this ceiling has moved. VCF 9.1 supports Antrea Egress with the EgressSeparateSubnet feature gate, which takes egress addresses from a child segment instead of letting the VPC gateway rewrite them again. A chosen set of pods can then keep a dedicated, routable identity that NSX can match policy on. I have not tested this yet, so treat it as documented capability rather than a lab result. But if you have previously told a customer that namespace granularity is a hard ceiling, that answer now needs an asterisk.
One more thing worth saying plainly, because people assume the opposite: outbound traffic is not blocked by default. Nothing stops a pod reaching the internet unless you write policy that does.
Seeing it for yourself
Reading about address rewriting is far less convincing than watching it happen. Two demos, neither of which needs a packet capture.
Watch the allocation chain fire. Three terminals, then apply the Service in the first:
SUPERVISOR_NS=my-namespace
# T1 — guest cluster
kubectl get svc my-app -w
# T2 — Supervisor namespace
kubectl -n $SUPERVISOR_NS get virtualmachineservice,svc,endpoints -w
# T3 — the NSX reconciler. Find it first, the pod name varies by build:
kubectl -n vmware-system-nsx get pods
kubectl -n vmware-system-nsx logs -f <that-pod> --tail=50
T1 shows <pending>. T2 shows the VirtualMachineService appear, then the Service and Endpoints. T3 shows the reconciler working. Then T1 fills in. About fifteen seconds of wall clock, and the whole chain becomes obvious.
Prove the rewrite. This is the two-minute version that makes the entire packet diagram land:
kubectl create deploy whoami --image=traefik/whoami --replicas=3
kubectl expose deploy whoami --type=LoadBalancer --port=80
kubectl get svc whoami -w # wait for EXTERNAL-IP
VIP=$(kubectl get svc whoami -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
# Cluster (default): the rewrite inside the cluster
curl http://$VIP/ # RemoteAddr = the pod network gateway
# repeat — Hostname rotates across replicas
kubectl patch svc whoami -p '{"spec":{"externalTrafficPolicy":"Local"}}'
# Local: no rewrite inside the cluster — and still not you
curl http://$VIP/ # RemoteAddr = the VPC gateway LB address
# repeat — Hostname sticks to local backends
traefik/whoami prints both RemoteAddr and Hostname. Watch RemoteAddr change between the two runs. It moves from the pod network gateway to the load balancer’s address, and never to yours. That is a more interesting demo than the one I expected to be giving, because it shows the limit rather than a workaround.
If you want to go deeper, the tools split by layer: pktcap-uw on the ESXi host (use net-stats -l to find the switchport ID), plain tcpdump inside the node VM, and antctl inside the antrea-agent pod for the bridge itself. That last one is the most underused tool on the platform. It will show you the actual tables from the figures above, on your own cluster.
What I would actually do with this
Four design positions, ranked, with the one I would defend hardest first.
Size the External block against every consumer, not one. Count one VIP per guest cluster endpoint, one per LoadBalancer Service, and one outbound address per namespace, then add the growth rate of each. Everything else on this list can be fixed later. Running out of routable addresses mid-project cannot.
Put Services behind one layer 7 proxy instead of exposing each one. Ten microservices exposed as ten LoadBalancer Services is ten routable addresses. Behind one ingress it is one. Note what this does not buy you: an ingress inside the cluster still sits downstream of the VPC gateway rewrite, so it cannot see your client either. Consolidate for the address economics and the routing, not for the client’s address.
Be generous with private VPC CIDRs. They are free, they can repeat across VPCs, and a tight range is what stops a cluster growing eighteen months from now, when nobody remembers why the number was chosen.
Decide the pod-level security story explicitly, and early. Pod-to-pod traffic is enforced at the pod’s own interface, which is what the three-ways-out table means in practice. Either you accept NetworkPolicy inside the cluster as the enforcement point, or you deploy the Antrea–NSX adapter so those same policies are authored and audited from NSX. Both are defensible. Finding out which one you have during a security review is not.
None of this is exotic. It follows from a layered platform where each layer does its own job well and none of them can see the others. Once you can hold the whole path in your head at once, the policy question, the source-address question and the address-planning question all turn out to be the same picture from different angles.
Written against VCF 9.1, with the data-plane addresses measured in a live lab rather than assumed. The TTL argument in an earlier version of this post was overstated and has been corrected here. Defaults and object names move between releases, so verify against the version in front of you before designing to them. Corrections and disagreements welcome. I would rather fix this than have it be quietly wrong.
If you administer vSphere, you already know how to build a VM. This post shows you how to build the same VM in VMware Cloud Foundation Automation using nothing but a command line — no UI clicks, no cloud template, no prior Kubernetes experience assumed. It takes about less then 10 minutes, and by the end you’ll have a running web server with a load-balanced IP that answers from your browser.
I’m deliberately keeping this simple. There is a longer, deeper reference post for the full API surface, but this one is the on-ramp: the shortest honest path from “I’ve never used the VCF Automation API” to “I just built something with it.”
What we’re going to do
Install the CLI and log in to your organization
Create a project (the tenancy boundary for a team)
Create a Supervisor Namespace (where workloads actually live)
Create a VM with cloud-init that installs nginx
Publish it behind a load balancer and browse to it
Build a Kubernetes cluster in the same namespace, and run a workload on it
Why bother, when the UI works fine?
Fair question, and I’d give three honest answers.
Repeatability. A VM you clicked into existence is a VM nobody can rebuild exactly. A VM defined in a YAML file is one you can commit, review, and recreate identically a year later.
It’s where the platform is going. In VCF 9, the consumption layer is a Kubernetes-style API. The UI is a client of that API, not the other way round. Learning the objects underneath means the UI stops being magic.
It’s the foundation for automation. Everything you’d want later — GitOps, pipelines, self-service catalogs — is built on these same objects. This is the first rung.
None of that means you should stop using the UI. I use both. But knowing what’s underneath makes you considerably harder to stump.
Before you start
What
Where to get it
VCF CLI (the vcf command)
Broadcom Support portal — it replaces the older CCI kubectl plugin
kubectl
Any recent version; the CLI generates a kubeconfig for it
An API token
Log into your VCF Automation org portal → your user menu → API Tokens → Generate. Copy it — it’s shown once.
Org admin rights
Needed to create projects. If you only have consumer rights, skip to Step 4 and use an existing namespace.
Everything below is run from PowerShell on Windows. It works identically on Linux and macOS; only the quoting changes.
Step 1 — Log in
First, store your token in a variable. Note the quotes — without them PowerShell tries to execute the token as a command, which produces a confusing error.
$vcfa_token = "your-api-token-here"
Now create a context. This is the CLI’s word for “a saved connection to an environment.”
Substitute your own endpoint and tenant name. The backtick is PowerShell’s line-continuation character — or just put it all on one line. Use --insecure-skip-tls-verify only in a lab; in production supply your CA certificate with --ca-certificate instead.
A successful login. Note that the CLI lists every context you now have access to — one for the organization, plus one for each Supervisor Namespace you can reach.
That output is worth reading carefully, because it explains the whole model. You get an organization context (org2-ctgw) and a namespace context for each namespace you can use (org2-ctgw:org2-ns1-4hm32:default-project).
The one concept to take away
There are two levels. The organization context is where you manage tenancy — projects, namespaces, who gets what. The namespace context is where you build things — VMs, clusters, networks. Almost every “why can’t I see that?” moment is being in the wrong one. If a command returns “the server doesn’t have a resource type”, you’re at the org level asking a workload question.
Step 2 — Create a project
A project is the boundary you’d hand to a team. Save this as project.yml:
apiVersion: project.cci.vmware.com/v1alpha2
kind: Project
metadata:
name: team-phoenix
spec:
description: Phoenix dev team
kubectl --context org2-ctgw create -f project.yml
# project.project.cci.vmware.com/team-phoenix created
Use create, not apply
This will save you a puzzling ten minutes. kubectl apply stores a tracking annotation on the object, and this API rejects it outright with “Annotation updates are not supported.” Everywhere in this post, use kubectl create.
Step 3 — Create a namespace
The project is just a boundary. A Supervisor Namespace is the thing with actual capacity — CPU, memory, storage, a network — where your VMs will live. Think of it as the resource pool plus the network plus the quota, expressed as one object.
You need four pieces of information, all of which you can read off an existing namespace or get from whoever set up your environment:
Two things that catch people here
The namespace name is generated with a random suffix (phoenix-ns-kj9g6), so read it from the previous command rather than guessing. And the context name is the full three-part string <org>:<namespace>:<project> — using just the namespace name gives you “no context exists.”
You can confirm the same thing in the UI, which is a reassuring way to check your work while you’re learning:
The namespace created from the command line, shown in the VCF Automation UI — Active, in the team-phoenix project, with its CPU, memory and storage allocation. The API and the UI are two views of the same object.
Why the long classConfigOverrides block?
When you create a namespace through the portal, it fills these values in for you from the class. Creating one through the API, you have to state them yourself — and the zone entry needs all four CPU and memory fields, not just a name. If you leave them out, the API tells you exactly what’s missing, one field at a time. That’s actually a decent way to learn the object: submit a minimal version and let the error messages guide you.
Step 4 — Create a VM
Now the part that will feel familiar. We need three things: a size, an image, and a storage policy. Ask the namespace what it has:
kubectl get virtualmachineclasses # the sizes available to you
kubectl get virtualmachineimages # images from your content library
kubectl get clustervirtualmachineimages | Select-String "ubuntu"
If virtualmachineimages comes back empty, you’re not stuck.
That list is empty when no content library is attached to your namespace class. You can still deploy from a cluster-wide image by referencing its vmi- identifier directly, which is what I do below. Getting a content library attached is the better long-term answer — it gives you friendly names instead of hashes — but it isn’t a blocker today.
Here’s the VM, together with a cloud-init secret that creates a login and installs nginx. Save both in one file:
Two objects, one command: the cloud-init secret and the VM itself.
Watch it come up. The VM will report an IP address once VMware Tools starts reporting:
kubectl get vm web01
kubectl get vm web01 -o jsonpath='{.status.network.primaryIP4}'
# 172.30.0.66
Notice what you didn’t have to specify: no port group, no IP address, no datastore. The VM picked up an address from the namespace’s default network automatically and landed on storage matching the policy you named. That’s the platform doing the placement work, exactly as it does for a Kubernetes pod.
Cloud-init only runs on first boot
You cannot add a bootstrap section to a VM that already exists. If you forget it, the VM boots fine but has no user account and no way in — you delete and recreate. Decide your bootstrap before you provision.
Step 5 — Publish it
A VirtualMachineService puts a load-balanced IP in front of one or more VMs, selected by label. Our VM carries app: web, so the service will find it.
The load balancer service, created. NSX allocates the external address within seconds.
Reading the external IPkubectl get vmservice prints only name, type and age — the address isn’t in its default columns. Use kubectl get svc instead, or ask for the field directly with -o jsonpath='{.status.loadBalancer.ingress[0].ip}'.
Step 6 — Confirm it works
kubectl get svc
# web-lb LoadBalancer 172.29.255.24 40.60.0.4 80/TCP 2m31s
kubectl get endpoints
# web-lb 172.30.0.66:80 2m34s
curl.exe http://40.60.0.4
The whole chain proven: external IP 40.60.0.4, backend 172.30.0.66:80, and nginx answering.
That’s the full path working. The load balancer address is allocated from the NSX VPC, the endpoint is your VM’s own IP, and nginx — installed by cloud-init at first boot — is serving. Nothing in that sequence involved the UI.
Step 7 — Build a Kubernetes cluster
Here’s the part that surprises people: a Kubernetes cluster is just another object in the same namespace you’ve been working in. Same context, same kubectl create, same mental model. You describe the cluster you want and the platform builds the nodes.
You need to know two things first — and both are places where a sensible-looking manifest fails.
Where the cluster template lives. Clusters are built from a ClusterClass, which is a template your platform team maintains. Crucially, it does not live in your namespace:
kubectl get clusterclasses
# NAME VARIABLES READY AGE
# builtin-generic-v3.1.0 True 164m
# builtin-generic-v3.2.0 True 164m
# builtin-generic-v3.3.0 True 164m
That list is not the whole story. The class you actually want may live in a shared system namespace and won’t appear here at all — you reference it explicitly by name and namespace.
Which Kubernetes version you can use. Ask the platform rather than guessing:
kubectl get kubernetesreleases
You’ll get a long list. Only the rows showing bothREADY: True and COMPATIBLE: True are usable. Pick one of those.
Now the cluster itself — one control plane node, one worker, deliberately small:
kubectl create -f cluster.yml
kubectl get cluster phoenix-k8s -w
Provisioning takes five to six minutes for this size. PHASE: Provisioned appears early and only means the objects were created — the column to watch is AVAILABLE, which flips to True once the control plane is genuinely up. kubectl get machines gives you the per-node view while you wait.
Three things in that manifest are not obvious1.classRef needs a namespace. The ClusterClass lives in a shared system namespace, and without it you’ll get a “not found” even though the class plainly exists. 2. The version string is the long form — v1.35.5---vmware.1-vkr.1, with three dashes, not the +vmware.1 form shown elsewhere. 3. The resolve-os-image annotation is often required. If your environment has the same node image registered in more than one content library — common in labs that have been upgraded a few times — the resolver refuses to guess and fails with “Multiple OSImages resolved.” Adding content-library= narrows it to one. You’ll find the library ID in your Content Libraries list.
The shortcut worth knowing
Rather than assembling this by hand, start a cluster creation in the VCF Automation UI and look at the YAML it generates. It will show you the right ClusterClass namespace, the right version format, and any annotations your environment needs — all correct for your specific build. Copy that, then edit it. This is far quicker than reverse-engineering from error messages, and it’s how I arrived at the manifest above.
Step 8 — Get onto the cluster and run something
Once AVAILABLE is True, download the cluster’s kubeconfig from the VCF Automation UI (the cluster’s detail page offers it) and point kubectl at it:
kubectl --kubeconfig .\phoenix-k8s.yaml get nodes
# NAME STATUS ROLES AGE VERSION
# phoenix-k8s-... Ready control-plane 12m v1.35.5+vmware.1
# phoenix-k8s-...-np-1-... Ready <none> 9m v1.35.5+vmware.1
You’re now a cluster administrator on an ordinary, conformant Kubernetes cluster. Everything you know about Kubernetes applies from here.
With one important exception, which is worth meeting deliberately rather than by accident.
Your first deployment will probably be rejected — and that’s correct
Try the obvious thing:
kubectl --kubeconfig .\phoenix-k8s.yaml create deployment web --image=nginx --replicas=2
The deployment is created, but no pods ever appear. Looking at the ReplicaSet tells you why:
pods "web-..." is forbidden: violates PodSecurity "restricted:latest":
allowPrivilegeEscalation != false
unrestricted capabilities (must set capabilities.drop=["ALL"])
runAsNonRoot != true
seccompProfile (must set seccompProfile.type to "RuntimeDefault")
This is Pod Security Admission enforcing the restricted profile, and it’s switched on by default. A stock container image that runs as root is refused at creation — not warned about, refused. For a vSphere admin used to “deploy first, harden later,” this is a genuine change of habit, and a welcome one: the cluster will not run a privileged workload just because you forgot.
The fix is to declare what the workload actually needs. Each field below answers one of the four complaints, and the image is swapped for a non-root variant that listens on 8080 rather than binding port 80 as root:
kubectl --kubeconfig .\phoenix-k8s.yaml apply -f web.yml
kubectl --kubeconfig .\phoenix-k8s.yaml expose deployment web --port=80 --target-port=8080 --type=LoadBalancer
kubectl --kubeconfig .\phoenix-k8s.yaml get svc web
The external IP is allocated by the same NSX load balancer that served your VM in Step 5 — usually within seconds. Confirm there are backends behind it:
Note that kubectl apply works here.
The “use create, not apply” rule applies to the VCF Automation API only. Inside a VKS cluster you’re talking to a normal Kubernetes API server, and all your usual habits — apply, Helm, Kustomize — work exactly as expected.
The five things that will trip you up
These cost me time so they don’t have to cost you any.
Use kubectl create, never apply. The API rejects the annotation that apply depends on.
Context names are three parts.<org>:<namespace>:<project>, not just the namespace.
Know which level you’re at. “The server doesn’t have a resource type” almost always means you’re at the org context asking a workload question, or vice versa.
Tokens expire mid-session. A sudden Unauthorized, or a strange complaint about downloading the OpenAPI schema, usually just means your session lapsed. vcf context use <org> re-authenticates.
Check the API version against your build. Published examples lag. On the environment used here, namespaces are served at v1alpha3 and VMs at v1alpha5. The most reliable way to learn the current shape of any object is to dump a working one: kubectl get vm <name> -o yaml. Copy the structure, change the values.
That last one is worth internalising as a general habit. Rather than hunting for documentation that matches your exact release, clone something that already works in your own environment. It is always correct, by definition.
Where to go next
You now have the pattern for everything else on this platform. The same two-level model, the same create workflow, and the same “clone a working object” trick apply to Kubernetes clusters, persistent volumes, NSX security policies, and micro-segmentation rules — all of which are just more objects in the same API.
A few natural next steps, in the order I’d take them:
Delete and rebuild what you just made. Change the VM size in the YAML, recreate, and watch it come back identically. That’s the payoff of declarative infrastructure, and it’s more convincing when you do it than when you read about it.
Scale the cluster. Change replicas on the worker pool, re-apply, and watch nodes roll in. Then bump version to a newer compatible release and watch a rolling upgrade — that’s a one-line change doing something that used to be a project.
Write a security policy as YAML and watch NSX distributed firewall rules appear.
Put it all in Git. Once your infrastructure is files, a GitOps engine can reconcile it. That’s the subject of my next post.
If you’re a vSphere admin who has been eyeing all this with mild suspicion, I’d encourage you to run through the steps above once in a lab. The concepts are less foreign than the vocabulary suggests — a namespace is a resource pool with a network and a quota, a VM is still a VM, and a load balancer service is the thing you’d have raised a ticket for. The difference is that now it’s a file you own.
Questions or something that didn’t work in your environment? Leave a comment — I read all of them, and environment-specific quirks are exactly what makes these posts better.