Author: vmtechie

  • Health, Forecasts and Pre-Flight for Every VKS Cluster: Introducing vks-fleet

    Health, Forecasts and Pre-Flight for Every VKS Cluster: Introducing vks-fleet

    VKS makes it easy to give every team its own Kubernetes cluster: VCF Automation hands them out, Cluster API keeps them reconciled, and the Supervisor runs them on the vSphere you already operate. As the number of clusters grows, the natural next step is to see the whole fleet at once: which clusters need attention, what is about to run out, and what a change will do before you make it. That is what I built vks-fleet for.

    vks-fleet is an open-source plugin for Headlamp that brings the Supervisor, vCenter, each cluster’s API and its Prometheus together in one view, for operators, read-only admins and tenants alike.

    A personal project. vks-fleet is something I built in my own time. It is not a Broadcom or VMware product, not endorsed or supported by either, and it uses only public APIs. It is Apache 2.0, and the code is on GitHub.

    The vks-fleet dashboard: a fleet score with a 30-day trend, what needs you now, eight blocks and a wall of clusters
    The fleet dashboard, in demo mode: a score, what needs you now, eight blocks that each open their page, and every cluster as a tile.

    Day-2 questions at fleet scale

    VKS gives you a lot out of the box. Cluster API reconciles your clusters, the Supervisor carries the control plane, and VCF Automation hands out self-service. Once there are many clusters, operators start asking fleet-level questions like these:

    • Which of my clusters needs attention right now, and which of those is actually urgent?
    • A node has been “deleting” for a day. Is that a PodDisruptionBudget, a volume that won’t detach, or the ESXi host underneath?
    • Which disk, volume or certificate runs out in the next 30 days?
    • If I upgrade this cluster, or scale this pool down, what breaks, what moves, and does the namespace have room while it rolls?
    • What is open security-wise across every cluster, not just the one I happen to be logged into?

    Every answer is already there, spread across a kubectl session per cluster, vCenter, the Supervisor’s own namespaces and each cluster’s Prometheus. vks-fleet brings them onto one screen, so the same questions take seconds whether you run three clusters or thirty.

    Why a Headlamp plugin

    I started out thinking about forking a Kubernetes UI and went the other way. Headlamp is an open-source Kubernetes web UI with a proper plugin system: it already handles sign-in, kubeconfig contexts, per-cluster views, logs, shells and YAML. A plugin can add pages, read any cluster Headlamp is signed in to, and hand off to Headlamp’s own views when you want the raw objects. That meant I could spend the effort on the fleet layer and let Headlamp do the rest.

    It also keeps the footprint small. The plugin runs in your browser with Headlamp’s sign-ins. There is no server of its own, no database, no agent in your clusters and no telemetry.

    How it is built to work

    The plugin follows the way an operator actually works through a problem:

    ObserveOne dashboard for the fleet, a score with its trend, and what needs you now.
    DetectRanked issues, each with its cause, and the steps that clear the most of them.
    ExplainA cluster’s timeline beside a walk down the layers under a node.
    PredictWhat runs out in 30 days, and what a change will do before you start it.
    ActControlled changes, each with its checks and a server-side dry run.

    Watch it in a minute: a one-minute walk through demo mode (the dashboard, investigating a node, a fix with its dry run, pre-flight, Supervisor health, security and what to do first) is at the top of the project’s README.

    Observe: one screen for the fleet

    The fleet page opens on a dashboard. At the top is a fleet score (the average of each cluster’s best-practice checks) with a 30-day trend, and a short Needs you now list that puts time-bound problems first: a node disk that fills in two days outranks a warning that has been there for a month.

    Below it are eight blocks: clusters, Supervisor, capacity, the next 30 days, lifecycle, security, governance and network. Each is one number with one line of context, and each opens the page that owns the detail. Under them, every cluster is a tile on a wall, coloured by its worst open problem and grouped by org.

    Detect: issues with a cause, not just a list of findings

    The plugin folds raw signals into issues that say what is wrong, why, and what to do. A stuck node, a drain blocked by a PodDisruptionBudget, a crash-looping pod, a certificate close to expiry, quota and memory overcommit, controllers that stopped reconciling, Supervisor services all running on one host, packages that fail to reconcile, backups that stopped: each becomes one line with its fix or the decision it needs.

    On top of that sits Do these first: the recommended steps that clear the most issues, ranked by effect, each opening where it is done. A Simulate view shows the fleet as it would look with the fixes applied, so you can see the payoff before spending the change window.

    The Issues tab: steps to do first, ranked by effect, then every open issue
    Issues: the steps to do first, then every open issue with its fix or the decision it needs.

    Explain: from a pod down to the ESXi host

    This is the part I use most. Investigate puts one cluster’s timeline (changes, alerts, events and anomalies) beside a walk down the layers under a node: pod, node, Machine, VM, ESXi host, namespace and Supervisor. It names the deepest layer in trouble, so you know whether you are looking at a Kubernetes problem, a VM problem or a host problem. When two troubled nodes share a host, it says so. It also drafts the post-mortem from the same timeline.

    Investigate: the cluster's timeline beside the layers under a stuck node, down to the ESXi host
    Investigate: what happened, beside every layer under the node. Here the deepest layer in trouble is the Machine, stuck deleting.

    Predict: the next 30 days, and pre-flight

    Forecasts from each cluster’s Prometheus (node disks, memory, volumes), the Supervisor’s control-plane disk from vCenter, and certificate expiries all land on one 30-day timeline.

    Pre-flight answers what will this change do? before an upgrade, a node pool scale or a VM class change. It works out which nodes are replaced, which pods move, which single-replica workloads take a brief outage, which disruption budgets or pinned pods would block it, whether the namespace has room for the extra nodes while it rolls, and whether the Supervisor itself is healthy enough to start. Then it gives a verdict: go ahead, go ahead with care, or stop.

    Pre-flight for scaling a node pool down: a blocker, the nodes and pods affected, and each check
    Pre-flight for scaling a pool down by one. The verdict is Stop: a node in the pool is already stuck deleting.

    For upgrades across many clusters there is an upgrade planner that arranges clusters in waves (a small canary first, then the rest) with the same checks for each cluster.

    Act: controlled changes

    Upgrades, node pool scaling, node replacement, drain timeouts, package updates, VM power operations and new clusters can all be started from the plugin. Every change shows its checks first and runs a server-side dry run (?dryRun=All), so anything the API or the VKS admission webhooks would refuse is caught before anything changes. Destructive actions ask you to type the cluster’s name.

    For shared installations there is a read by default, elevate to change mode: viewing uses a read-only sign-in, and a change needs a time-limited elevation with a reason, recorded on the object it changed.

    Supervisor health

    Clusters are only as healthy as the Supervisor running them. A Supervisor page shows the control plane, the ESXi hosts and what runs on each, the leader-election leases of the controllers that manage your clusters (a lease that stops renewing means a controller is stuck, even when its pod says Running), Supervisor services, and the reconcile backlog. Click a host and it shows what that host failing would take down. With an optional, read-only collector it also reads vCenter: configuration status, alarms, utilisation and the control-plane VM’s disks.

    Supervisor health: its score, the ESXi hosts and what runs on each, what vCenter reports, and the controllers
    Supervisor health: the hosts, what runs on each, and what vCenter reports.

    Security: what is open, across every cluster

    The Security overview shows what is open across every cluster you are signed in to: privileged namespaces, privileged and host-level pods, cluster-admin grants, and image vulnerabilities from Trivy Operator, counted once per image and split into your images and the ones VKS ships. Policy results from Kyverno or any engine that writes PolicyReports appear beside them. A cluster without a scanner reads not measured, never zero.

    Security overview: privileged pods and admin grants, critical vulnerabilities, privileged namespaces and accepted risks, then one row per cluster
    Security overview: what is open across the fleet, then one row per cluster.

    Principles behind it

    PrincipleWhat it means in practice
    Public APIs onlyCluster API, VM Operator, NSX VPC, Carvel, Prometheus, and vCenter through a small read-only collector. No private interfaces.
    Nothing installed in your clustersEverything is read through the APIs that are already there. Trivy, Kyverno and Prometheus are used when present, never required.
    Your RBAC decides what you seeOperators see every org, read-only admins see everything and change nothing, tenants see only their own org. It follows the account Headlamp signs in with, not a mode in the UI.
    Honest numbersData that can’t be read is “not measured”, not zero or passing. Every figure says where it came from.
    Dry run before every changeChecks, then a server-side dry run, then confirmation. If a multi-step change fails partway, it says which steps took effect.

    Try it without a lab

    The fastest way to see it is demo mode: a fictional fleet with realistic problems built in, where nothing is ever changed. Install the plugin, open VKS fleet and choose Try the demo. You can scale the demo fleet up to 50 clusters to see how the pages behave at size.

    Headlamp desktop app or Docker: download vks-fleet.tar.gz from the latest release, extract it into Headlamp’s plugins folder, restart Headlamp and open VKS fleet.

    In a cluster, with Helm: the chart installs Headlamp, the plugin, and a job that keeps the Supervisor sign-in fresh. The quick start has the full commands, including trusting vCenter’s CA so certificates are verified; the core of it is:

    VERSION=1.43.0
    helm install vks-fleet "https://github.com/avnish80/vks-fleet/releases/download/v$VERSION/vks-fleet-$VERSION.tgz" \
      -n vks-fleet --set plugin.source=download \
      --set refresher.supervisors="<your Supervisor address>" --set tls.caSecret=vks-fleet-ca

    Then connect your Supervisor under Settings → Plugins → vks-fleet. It is also listed on Artifact Hub as a Headlamp plugin.

    Where it stands, honestly

    It is developed and tested against VCF 9.1 with VKS 3.7 on Headlamp 0.45, on a real lab with one Supervisor, two VKS clusters and VCF Automation tenants. Every read and view has been checked there, as have the vCenter collector and changes through the VKS admission webhooks: scaling node pools up and down, pausing and resuming clusters, and a clean Helm install under restricted Pod Security with verified TLS.

    What is not yet confirmed on a live system: deletes through the plugin, drain timeouts on machines that are already deleting, and some links into Headlamp’s own pages. The checks also run from the browser, which suits tens of clusters; demo mode exercises the pages at 50, but a real fleet has so far been tested at two. The README’s validation status keeps the current list.

    If you run VKS, I would genuinely like to hear how it behaves on your fleet, especially with more clusters, more than one Supervisor, or tenants coming in through VCF Automation.

    vks-fleet is a personal open-source project under the Apache 2.0 licence. It is not affiliated with, endorsed by, or supported by Broadcom or VMware. Product names are used only to describe what it works with.

  • Demystifying VKS Networking with VCF

    Demystifying VKS Networking with VCF

    A VKS cluster is built in layers, and different teams own different layers. A pod sits below anything kubectl will show you, and above anything vCenter will show you. That is true of Kubernetes on any virtual infrastructure, but it does mean very few people ever see the whole picture at once. This post walks the entire path, one layer at a time: what happens inside the node, what happens inside the virtual switch in the guest, and what happens when someone exposes a pod with a LoadBalancer Service and a user connects to it from outside.

    I have drawn this on a whiteboard often enough that it seemed worth writing down properly. If you have ever had to answer one of these, this post is for you:

    • Where should a pod-level firewall rule actually be written?
    • Why does my application log every request as coming from the same address?
    • Is AntreaProxy a process? What happens to my traffic if it restarts?
    • How does a LoadBalancer Service actually get its address?
    • Where does a packet really go when one pod talks to another?

    Every one of those is the same question in a different costume: where does the packet go, and who rewrites it on the way?

    One note before we start. Networking people say SNAT when they mean “rewrite the source address” and DNAT when they mean “rewrite the destination address.” Both appear a lot below, and almost everything confusing in this post comes down to one of them happening somewhere you did not expect.

    The whole path, bottom up

    Read this from the bottom up, in the direction an arriving packet travels. Everything here is the NSX overlay design. The other switching model is a post of its own.

    Count the layers as you go. That number explains most “why is this so hard to troubleshoot” conversations on this platform.

    GUEST OS · NODE VM ESXi VIRTUAL SWITCH HOST · FABRIC Pod · eth0 address from the node’s podCIDR (Antrea IPAM) veth pair peer end is an OVS port on br-int antrea-agent · OVS br-int veth ports + antrea-gw0 · tun0 only for the inter-node hop Node eth0 (vmxnet3) node IP · Antrea transport interface vNIC → segment port / dvPort the last object vCenter owns · vNIC-level policy applies here NSX OVERLAY NSX segment · distributed router VPC gateway → TGW → T0, upstream on the Edge TEP vmk10/11 · Geneve decap UDP 6081 · VNI selects the segment VDS uplink · teaming / LAG uplink profile · failover order vmnic0 — the physical NIC RSS · NetQueue · Geneve offload ToR switch port VLAN trunk · TEP + workload VLANs packet direction ↑
    The VKS data path on NSX overlay networking. Three hypervisor hops, and four inside the guest where vCenter can no longer see them.

    What the diagram makes obvious is that the path crosses three ownership boundaries: the guest operating system and its CNI, the ESXi virtual switch, and the physical network. Each one is well instrumented on its own. None of them shows you the other two. That is the real reason a networking question here is hard to answer from a single console, and it is why this post walks the whole path instead of any one layer of it.

    Inside the node: one switch, everything plugged into it

    Now zoom in on the guest. The correction I make most often in a room is this: br-int is the OVS bridge. There is no OVS on one side and a bridge on the other. There is one bridge, it is called br-int, and everything plugs into it.

    Guest cluster worker node (a VM) pod A · netns 192.168.145.12 eth0 pod B · netns 192.168.145.13 eth0 pod C · netns 192.168.145.14 eth0 veth pair veth pair veth pair br-int — the OVS integration bridge (one per node) forwarding happens in the kernel datapath · flow tables held by ovs-vswitchd port port port antrea-gw0 antrea-tun0 antrea- agent control plane only programs flow rules ↓ · never sees a packet node’s own IP stack / routing 172.30.0.2 Geneve encap vNIC → vmnic the physical NIC is NOT attached to br-int in encap mode
    One node, drawn honestly. Each pod’s veth peer is a port on br-int. antrea-gw0 is the pod network gateway; antrea-tun0 carries pod traffic between nodes.

    Every pod is connected by a veth pair, which is best thought of as a short virtual cable. One end is the pod’s eth0. The other end is a port on br-int. Two other ports matter: antrea-gw0 is the pod network’s gateway on this node, and antrea-tun0 is the tunnel port used only when a pod talks to a pod on a different node.

    Three things live on that bridge, and it is worth being precise about which does what, because the names mislead:

    ComponentWhat it actually is
    ovs-vswitchdThe user-space daemon that holds the flow tables. The kernel module underneath is what actually moves packets.
    antrea-agentControl plane only. It writes the rules. It never touches a packet. If it stops, existing traffic keeps flowing.
    AntreaProxyNot a process at all. It is a set of rules in the bridge: one table picks a backend pod, the next rewrites the destination to that pod’s address. This is what replaces kube-proxy.

    The correction that lands hardest. There is no proxy process in the data path. When people say “Antrea does proxying on the bridge,” what actually happens is that a packet matches a rule and has its destination rewritten, in the kernel, with nothing in user space involved. That is why AntreaProxy costs almost nothing, and why restarting antrea-agent does not drop traffic.

    Inside br-int: the tables a packet walks

    This is the level almost nobody goes to, and it is the level where the rest of the post stops being mysterious. A packet enters at a port, walks through a series of tables in order, and leaves at a port. That is the whole model.

    podeth0 SpoofGuard is this really pod A’s IP+MAC? ServiceLB ClusterIP hit → pick a bucket Endpoint DNAT rewrite dst to the real pod IP Network Policy allow or drop (egress + ingress) L3 Forward + SNAT which port out? rewrite src if needed Output send to port destinationport A packet from pod A to a Service — the tables it walks, left to right all of this is one pass through one bridge · no userspace, no extra hop, TTL unchanged What the header looks like as it goes in src 192.168.145.12 dst 10.96.41.7:80 ← the Service ClusterIP, an address no machine owns src 192.168.145.12 dst 192.168.145.14:80 ← after ServiceLB + EndpointDNAT picked a backend out src 192.168.145.1 dst 192.168.145.14:80 ← after SNAT to antrea-gw0, if the path needs it
    The pipeline inside the bridge. A packet enters at a port, walks the tables in order, and leaves at a port. Service resolution and NetworkPolicy are both just tables in that sequence.

    Two things are worth noticing about that sequence.

    First, a Service is just a table. A ClusterIP is not an address that anything holds. No network card anywhere is configured with it. It is only something to match on. The packet arrives addressed to the ClusterIP, hits the table that handles Services, one backend pod is selected, and the destination is rewritten to that pod’s real address. Ping a ClusterIP and you may well get nothing back. That is not a fault. You are pinging something that was never meant to answer.

    Second, NetworkPolicy is also just a table, and it sits in this same pipeline, right at the pod’s own interface. Where it sits matters more than it sounds, and it comes back later.

    Three ways out, and that is all of pod networking

    Once you can see the bridge, every packet a pod sends ends in one of exactly three ways. There is no fourth.

    A · same node out another port on the same switch pod A br-int pod C Never leaves the VM. No encapsulation, no NSX, no physical network at all. Two ports, one lookup. B · another node wrapped in Geneve, out the vmnic antrea-tun0 node stack → vmnic → NSX peer tun0 The outer envelope costs bytes on every frame — which has consequences for how you size the fabric. Node-to-node only. Never north-south. C · outside the cluster through the door to the node’s stack antrea-gw0 node routing → VPC gateway gw0 is an OVS internal port that is also a real Linux interface with an IP — 192.168.145.1 That address is the in-cluster masquerade.
    Every packet a pod sends ends one of three ways — and each one tells you where its policy belongs.
    DestinationWhat happensWhat that means
    Pod on the same nodeStraight out another port on the same bridge. The packet never reaches a vNIC at all.Policy for this hop has to live where the traffic is: NetworkPolicy in the bridge, at the pod’s own interface.
    Pod on another nodeOut antrea-tun0, wrapped in a tunnel header, to the tunnel port on the other node.At the vNIC it looks like UDP 6081 between two node addresses, so pod-level policy again belongs in the bridge.
    Anything outside the clusterOut antrea-gw0, then the node’s eth0, with the source address rewritten on the way.The far end sees an infrastructure address instead of the pod, the same as any other masqueraded traffic.

    That table is worth reading twice, because it tells you where to write policy. A firewall at the vNIC — on any platform, with any CNI — can only act on traffic it can see, and two of those three paths either never reach a vNIC or arrive inside a tunnel. So pod-level policy belongs at the pod’s own interface, which is exactly where Antrea puts it.

    You can write it with kubectl, or author and audit it in NSX and let the Antrea–NSX adapter push it down. Either way the bridge is what enforces it, so you get one policy plane for pods whichever door you came in through. The controls at the vNIC stay for what they are genuinely good at: keeping one cluster’s nodes away from another’s, and limiting both to the Supervisor and the registry.

    The three address blocks

    The packet walk below needs three names, so here they are in the shortest useful form. An NSX VPC draws addresses from three separate blocks:

    BlockWhat comes from itRoutable?
    ExternalLoadBalancer VIPs, cluster API endpoints, outbound SNAT addressesYes — size this one deliberately
    Private transit gatewayAddresses reachable across VPCs in the same projectInternally
    Private VPC CIDRNode VMs, and the slice of pod addresses given to each nodeNo

    How those get allocated, and what you commit to on the Workload Network page of the activation wizard, deserves its own post. For now: VIPs come from External, pods and nodes come from the private VPC CIDR, and the VPC gateway sits between the two.

    Exposing a pod: what a LoadBalancer Service really does

    There are two separate stories here, and merging them is what makes this hard to explain. First an address gets allocated, which is a chain of controllers across three clusters and never touches a packet. Then a client actually connects.

    Part one: allocation

    The guest cluster has no load balancer controller of its own. When someone applies a Service of type LoadBalancer:

    1. The paravirtual cloud provider, running on the guest cluster’s control plane, notices it. It holds scoped credentials to the Supervisor.
    2. It creates a VirtualMachineService in the Supervisor namespace, selecting that cluster’s node VMs.
    3. VM Operator turns that into a Service plus an Endpoints object whose members are the node VM addresses.
    4. The NSX reconciler watches Supervisor Services and calls the NSX API. Classic deployments run NCP; VPC-mode builds may run the NSX VPC Operator instead. Worth checking which one you have, although the NSX objects are named with an ncp- prefix either way, so the naming will not tell you.
    5. NSX creates a virtual server and a server pool on the VPC gateway, and allocates a VIP from the External block.
    6. The address travels back down the same chain into the guest Service’s status.loadBalancer.ingress.

    This is why <pending> is almost never a guest-cluster problem. There is no controller in that cluster that could allocate an address in the first place. If it is stuck, the chain broke further up. Check the VirtualMachineService in the Supervisor namespace first, then the NSX reconciler, then whether the External block has any free addresses left.

    Part two: the packet

    Now a client connects to the VIP. The left column is where the packet is. The right column is what its addresses look like at that exact point, with anything rewritten highlighted.

    WHERE THE PACKET IS WHAT IT LOOKS LIKE THERE External client somewhere on the corporate network SRC<client>:52310 DST40.60.0.2:80 the allocated VIP Fabric → Tier-0 BGP carries the External block only SRC<client>:52310 DST40.60.0.2:80 unchanged — pure routing VPC gateway load balancer SNAT mode: Automap · rewrites BOTH SRC 100.64.0.1 DST 172.30.0.3:31046 SNAT — CLIENT IP LOST HERE DNAT → NODE : NODEPORT NSX overlay to the node’s host wrapped in Geneve, TEP to TEP SRC100.64.0.1 DST172.30.0.3:31046 inner header untouched Node eth0 → OVS br-int decapsulated, into the guest SRC100.64.0.1 DST172.30.0.3:31046 the client IP never got this far AntreaProxy in br-int picks a backend, rewrites again SRC 192.168.145.1 DST 192.168.145.11:8080 SNAT → ANTREA-GW0 (CLUSTER ONLY) DNAT → POD : TARGETPORT With externalTrafficPolicy: Local the in-cluster masquerade is skipped SRC100.64.0.1 DST192.168.145.11:8080 still the LB, still not the client Pod application sees the connection SRC192.168.145.1 or 100.64.0.1 DST192.168.145.11:8080 never the real client, either way Addresses captured from a live VCF 9 lab on NSX VPC networking; only the client address is elided.
    Four rewrites — and the one that destroys the client address happens at the load balancer, before Kubernetes ever sees the packet.

    The rewrite that matters happens earlier than most people expect. The load balancer changes the source address as well as the destination, so the client’s address is already gone at the VPC gateway. That is before the packet has reached a node, and long before Kubernetes has any say in it. It is also why your access logs are full of the same address over and over.

    What the pod sees as the source address

    The standard advice is that externalTrafficPolicy: Local preserves the client’s address. On this path it does not. It only changes which infrastructure address the pod sees, and I only know that because I went and measured it.

    First, where this setting actually lives

    externalTrafficPolicy is a field on the Kubernetes Service. You set it with kubectl, in YAML, next to type: LoadBalancer. You never set it in NSX Manager. It is plain upstream Kubernetes, not a VMware addition, and it works the same way on EKS, GKE, OpenShift, or a bare-metal cluster running MetalLB. If you already know this setting from another platform, you know it here.

    What is platform-specific is how the setting gets carried out. Choosing Local makes Kubernetes allocate a healthCheckNodePort. Every node then answers a simple health check on that port: 200 if it is running a pod for this Service, 503 if it is not. The controller that reconciles the Service turns that into two things on the load balancer, its pool membership and its health monitor. Nodes that answer 503 fail the check and stop receiving traffic.

    So NSX Manager is where you see the result, not where you configure it. If you edit that pool by hand, it will be put back. The Service object is the source of truth.

    Client 203.0.113.50 NSX VPC gateway · LoadBalancer VIP DNAT  VIP:443 → nodePort on a member node SNAT  source rewritten to a gateway address 100.64.0.1 THE CALLER’S ADDRESS IS ALREADY GONE AT THIS LINE externalTrafficPolicy: Cluster a field on the Service · the default externalTrafficPolicy: Local a field on the Service · you have to ask for it WHICH NODES THE LOAD BALANCER WILL SEND TO Every node in the cluster the nodePort is open everywhere, so all nodes pass Only nodes running a backend pod healthCheckNodePort · the rest fail and drop out WHERE THE PACKET LANDS Node A — which may hold no backend pod any node can be chosen, pod or no pod Node B — guaranteed to hold a backend pod the pool contains nothing else THE SECOND HOP Forwarded again — Node A → Node B source rewritten again, to that node’s pod-network gateway so the reply comes back the way it came No second hop nothing to forward, so nothing to rewrite again WHAT THE POD READS AS THE SOURCE 192.168.145.1 antrea-gw0 · the pod-network gateway on that node 100.64.0.1 the VPC gateway load balancer’s own address Local removes the second rewrite. It cannot undo the first one. Both values measured.
    Where externalTrafficPolicy sits. It controls the hop inside the cluster — the second rewrite, not the first. Addresses are from the lab measurement later in this post.

    What each setting does

    Cluster — the default

    • Every node accepts the traffic, whether or not it is running a pod for this Service.
    • If the chosen pod is on a different node, the packet is forwarded on, and the source is rewritten to antrea-gw0, the pod network’s gateway on the node that received it.
    • Load spreads evenly across pods. An extra node-to-node hop is common on clusters with several nodes.
    • The pod sees the pod network gateway.

    Local — opt in

    • Only nodes actually running a pod for this Service answer, decided by the health check described above.
    • No extra hop and no second rewrite inside the cluster. Both are real benefits.
    • Traffic is spread across nodes rather than across pods, so uneven scheduling gives uneven load.
    • The pod still does not see the client. It sees the load balancer’s own address.

    Here is the measurement, from an NSX VPC lab. Three replicas of traefik/whoami behind a LoadBalancer Service, called with curl from an admin host outside the cluster:

    externalTrafficPolicy: Cluster
      RemoteAddr: 192.168.145.1     ← antrea-gw0, the rewrite inside the cluster
    
    externalTrafficPolicy: Local
      RemoteAddr: 100.64.0.1        ← the VPC gateway load balancer
    
    real client address
      never visible on this path at all

    That node’s pod addresses came from 192.168.145.0/24, so .1 is its antrea-gw0. And 100.64.0.1 is not a guess. The NSX server pool for that Service shows SNAT Translation Mode: Automap, which replaces the source with the load balancer’s own address before handing the packet to a pool member. Every pool the reconciler creates has it set.

    The virtual server is also plain TCP, on a Fast TCP profile, with no awareness of HTTP. So the source address the pod reads is the load balancer’s, not the caller’s. That is how any layer 4 load balancer doing source NAT behaves, on any platform, and it is exactly why terminating at layer 7 exists as a design pattern.

    One clarification, because this is easy to overstate. The reconciler does create an HTTP profile on the same load balancer, alongside the TCP and UDP ones. The VPC load balancer is perfectly capable of layer 7. But that profile is used by virtual servers created from Ingress objects. A Service of type LoadBalancer always gets the layer 4 one. The limit is on the Service path, not on the load balancer.

    The design consequence. The rewrite happens one hop before Kubernetes sees the packet, which makes this an architecture decision rather than a Service setting. The same is true of any layer 4 load balancer doing source NAT, wherever it runs. If an application needs the caller’s address, put something that terminates at layer 7 in front of the rewrite, where the client is still visible. Decide that alongside the ingress design, not when an application team asks the question.

    This is worth stating plainly, because the usual Kubernetes advice about Local assumes a load balancer that is not rewriting the source. It is not a defect. It is the documented behaviour of a layer 4 virtual server whose pool is set to Automap. But it does mean Local buys you a shorter path and one less rewrite, not the client’s address.

    Automap is a property of the pool, and NSX supports other modes, so in principle it is configurable. In practice these pools are created and maintained for you, so editing them by hand is not a supported workaround. Treat it as a property of the platform and design around it.

    So when is Local worth setting?

    Not for the client’s address. That is the whole point of the measurement above. But there are two honest reasons to use it. It removes a node-to-node hop, which is worth having on a latency-sensitive path and keeps that traffic off the tunnel between nodes. And it removes one layer of rewriting, which makes a packet capture much easier to read.

    The cost is load distribution, and the arithmetic is worth doing. Take three nodes and four replicas: two on node B, one on node C, none on node A. The load balancer sees two healthy nodes and splits the traffic 50/50. Node B’s two pods share its half, so each gets a quarter. Node C’s single pod takes the other half on its own. That one pod is now handling twice the traffic of its peers, and nothing in Kubernetes will correct it. With Cluster, traffic spreads across all three nodes and then across all four pods, so each gets roughly a quarter, at the price of that extra hop.

    So Local fits best where pod placement is already even by design: a workload that runs one pod per node, or one where you actually enforce a topology spread constraint. Set it on purpose, not by habit.

    Verified two ways

    What an application reports is suggestive, not proof. traefik/whoami tells you what its own socket says, which is one tool’s opinion. So I checked the same thing a completely different way, using Antrea Traceflow in liveTraffic mode. That captures the real frame inside the bridge instead of asking an application about it.

    apiVersion: crd.antrea.io/v1beta1
    kind: Traceflow
    metadata:
      name: tf-live
    spec:
      liveTraffic: true
      timeout: 60
      destination: { namespace: default, pod: whoami-… }
      packet:
        ipHeader: { protocol: 6 }
        transportHeader: { tcp: { dstPort: 8080 } }

    Then call the VIP inside the timeout window and read status.capturedPacket. Both settings, same cluster, minutes apart:

    externalTrafficPolicy: Local
      {"srcIP":"100.64.0.1",    "dstIP":"192.168.145.12",
       "tcp":{"dstPort":8080,"flags":2}, "ttl":123}
    
    externalTrafficPolicy: Cluster
      {"srcIP":"192.168.145.1", "dstIP":"192.168.145.12",
       "tcp":{"dstPort":8080,"flags":2}, "ttl":123}

    Two unrelated tools, the same two addresses. The load balancer’s Automap rewrite under Local, antrea-gw0 under Cluster, and in neither case the client.

    The TTL is 123 in both captures. That is consistent with the rewrite inside the cluster being done by a rule in the bridge rather than being a routing hop. I used to present it as proof, and a reviewer was right to push back: an identical TTL does not prove no proxy was involved, because a proxy ends one connection and starts a new packet with its own TTL anyway. What actually settles it is the pipeline itself and the Traceflow capture, which show the packet being rewritten in the bridge with nothing in user space terminating anything. The TTL supports the conclusion; it does not carry it. (For the curious: starting from a Windows client’s 128, five hops were taken off on the way in, which matches the fabric, T0, transit gateway and VPC gateway.)

    This also answers “can I see this flow with the product tooling?” Partly. Antrea Traceflow covers the half inside the guest and will capture live traffic. NSX Traceflow draws the overlay half, but cannot follow a VIP through the load balancer, because it injects a packet between two ports and a virtual server has no port to inject at. Neither tool does the whole path on its own. With the Antrea–NSX adapter registered, Antrea’s results appear in the NSX UI, which is the closest thing to one view across both.

    The direction nobody asks about: outbound

    Everything above is traffic arriving. Traffic leaving behaves differently, and it is routinely overlooked right up until a partner’s firewall gets involved.

    A pod calling something outside leaves through antrea-gw0, out the node’s eth0, and through the VPC gateway, which rewrites the source again. So the mirror image of the client-address story applies: coming in, your pod cannot see who called it; going out, the far end cannot see which pod called them. Same mechanism, opposite direction. Neither is a fault. Both are what a rewrite does.

    Which means that when a partner asks you to allow-list “the payments pod” at their firewall, by default you cannot. With the standard VPC gateway rewrite, the finest identity available at that boundary is the namespace, and only where Antrea Egress is in use. Without it, traffic leaves as the node’s address and even that distinction disappears.

    Worth knowing that this ceiling has moved. VCF 9.1 supports Antrea Egress with the EgressSeparateSubnet feature gate, which takes egress addresses from a child segment instead of letting the VPC gateway rewrite them again. A chosen set of pods can then keep a dedicated, routable identity that NSX can match policy on. I have not tested this yet, so treat it as documented capability rather than a lab result. But if you have previously told a customer that namespace granularity is a hard ceiling, that answer now needs an asterisk.

    One more thing worth saying plainly, because people assume the opposite: outbound traffic is not blocked by default. Nothing stops a pod reaching the internet unless you write policy that does.

    Seeing it for yourself

    Reading about address rewriting is far less convincing than watching it happen. Two demos, neither of which needs a packet capture.

    Watch the allocation chain fire. Three terminals, then apply the Service in the first:

    SUPERVISOR_NS=my-namespace
    
    # T1 — guest cluster
    kubectl get svc my-app -w
    
    # T2 — Supervisor namespace
    kubectl -n $SUPERVISOR_NS get virtualmachineservice,svc,endpoints -w
    
    # T3 — the NSX reconciler. Find it first, the pod name varies by build:
    kubectl -n vmware-system-nsx get pods
    kubectl -n vmware-system-nsx logs -f <that-pod> --tail=50

    T1 shows <pending>. T2 shows the VirtualMachineService appear, then the Service and Endpoints. T3 shows the reconciler working. Then T1 fills in. About fifteen seconds of wall clock, and the whole chain becomes obvious.

    Prove the rewrite. This is the two-minute version that makes the entire packet diagram land:

    kubectl create deploy whoami --image=traefik/whoami --replicas=3
    kubectl expose deploy whoami --type=LoadBalancer --port=80
    kubectl get svc whoami -w          # wait for EXTERNAL-IP
    
    VIP=$(kubectl get svc whoami -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
    
    # Cluster (default): the rewrite inside the cluster
    curl http://$VIP/                  # RemoteAddr = the pod network gateway
                                       # repeat — Hostname rotates across replicas
    
    kubectl patch svc whoami -p '{"spec":{"externalTrafficPolicy":"Local"}}'
    
    # Local: no rewrite inside the cluster — and still not you
    curl http://$VIP/                  # RemoteAddr = the VPC gateway LB address
                                       # repeat — Hostname sticks to local backends

    traefik/whoami prints both RemoteAddr and Hostname. Watch RemoteAddr change between the two runs. It moves from the pod network gateway to the load balancer’s address, and never to yours. That is a more interesting demo than the one I expected to be giving, because it shows the limit rather than a workaround.

    If you want to go deeper, the tools split by layer: pktcap-uw on the ESXi host (use net-stats -l to find the switchport ID), plain tcpdump inside the node VM, and antctl inside the antrea-agent pod for the bridge itself. That last one is the most underused tool on the platform. It will show you the actual tables from the figures above, on your own cluster.

    What I would actually do with this

    Four design positions, ranked, with the one I would defend hardest first.

    1. Size the External block against every consumer, not one. Count one VIP per guest cluster endpoint, one per LoadBalancer Service, and one outbound address per namespace, then add the growth rate of each. Everything else on this list can be fixed later. Running out of routable addresses mid-project cannot.
    2. Put Services behind one layer 7 proxy instead of exposing each one. Ten microservices exposed as ten LoadBalancer Services is ten routable addresses. Behind one ingress it is one. Note what this does not buy you: an ingress inside the cluster still sits downstream of the VPC gateway rewrite, so it cannot see your client either. Consolidate for the address economics and the routing, not for the client’s address.
    3. Be generous with private VPC CIDRs. They are free, they can repeat across VPCs, and a tight range is what stops a cluster growing eighteen months from now, when nobody remembers why the number was chosen.
    4. Decide the pod-level security story explicitly, and early. Pod-to-pod traffic is enforced at the pod’s own interface, which is what the three-ways-out table means in practice. Either you accept NetworkPolicy inside the cluster as the enforcement point, or you deploy the Antrea–NSX adapter so those same policies are authored and audited from NSX. Both are defensible. Finding out which one you have during a security review is not.

    None of this is exotic. It follows from a layered platform where each layer does its own job well and none of them can see the others. Once you can hold the whole path in your head at once, the policy question, the source-address question and the address-planning question all turn out to be the same picture from different angles.

    Written against VCF 9.1, with the data-plane addresses measured in a live lab rather than assumed. The TTL argument in an earlier version of this post was overstated and has been corrected here. Defaults and object names move between releases, so verify against the version in front of you before designing to them. Corrections and disagreements welcome. I would rather fix this than have it be quietly wrong.

  • Configuring Custom Default Gateways on VPC-Backed Subnets in VCFA

    Create a VPC-backed subnet in VCFA 9 and NSX assigns the default gateway for you — the first usable address in the CIDR. Try to change it in NSX Manager and the field is greyed out. It looks like a platform limitation. It isn’t. The gateway is fully configurable, at creation and afterwards, through the Supervisor API. This post covers the whole path: installing the CLI, establishing organization and namespace context, creating a subnet with a custom gateway, modifying an existing one, and verifying that NSX actually realised the change.

    Contents
    1. Why the NSX Manager field is disabled · 2. Prerequisites · 3. Establishing context with the VCF CLI · 4. Understanding the two Subnet kinds · 5. Creating a subnet with a custom gateway · 6. Modifying an existing subnet · 7. Verifying realisation · 8. Choosing a valid CIDR

    1. Why the NSX Manager field is disabled

    VPC-backed subnets in VCFA 9 are managed declaratively. The Kubernetes object is the source of truth, and a controller reconciles it into NSX continuously. Editing the realised configuration in NSX Manager sits outside that loop — so even when a change appears to save, the controller restores whatever the object specifies. That is also why the value reverts on refresh: a symptom that reads like a defect but is the reconciliation loop working correctly.

    The rule that follows is worth carrying beyond this specific case: in a declarative system, a greyed-out control usually means you are looking at realised state rather than desired state. The edit belongs in the object.

    2. Prerequisites

    RequirementWhere to get it
    VCF CLI (the vcf command)Broadcom Support portal. It supersedes the older kubectl cci plugin.
    kubectlAny recent release. The VCF CLI generates the kubeconfig it uses.
    An API tokenVCF Automation org portal → user menu → API Tokens → Generate. Displayed once; copy it immediately.
    Namespace accessA role that permits creating objects in a Supervisor Namespace. Read-only consumers can follow along but not apply.

    Commands below are PowerShell on Windows. They work identically on Linux and macOS with the usual quoting differences.

    3. Establishing context with the VCF CLI

    Store the token in a variable. The quotes matter — without them PowerShell attempts to execute the token as a command.

    $vcfa_token = "your-api-token-here"

    Create the organization context. This is the CLI’s term for a saved connection to a VCF Automation tenant.

    vcf context create my-org `
      --endpoint https://vcfa.example.com `
      --type cci `
      --auth-type basic `
      --tenant-name My-Org `
      --api-token $vcfa_token `
      --insecure-skip-tls-verify

    Substitute your endpoint and tenant name. Use --insecure-skip-tls-verify only in a lab; in production supply your fleet CA with --ca-certificate instead.

    The CLI creates one context for the organization plus one for every Supervisor Namespace you can reach. List them:

    kubectl config get-contexts
    NAME                                      NAMESPACE
    my-org
    my-org:app-ns-4hm32:default-project       app-ns-4hm32
    my-org:phoenix-ns-kj9g6:team-phoenix      phoenix-ns-kj9g6
    Context names have three parts The format is <org>:<namespace>:<project>. Using the bare namespace name returns “no context exists with the name”. Copy the full string from the output above.

    Switch into the namespace where the subnet will live:

    kubectl config use-context my-org:phoenix-ns-kj9g6:team-phoenix

    Confirm you are on the namespace layer rather than the organization layer — the presence of subnetsets is a reliable indicator:

    kubectl get subnetsets
    NAME          ACCESSMODE   IPV4SUBNETSIZE   NETWORKADDRESSES
    pod-default   PrivateTGW   32
    vm-default    Private      32               172.30.0.64/27
    Tokens expire mid-session An expired token surfaces as Unauthorized, as com.vmware.vapi.std.errors.unauthorized, or — confusingly — as a complaint about downloading the OpenAPI schema on an otherwise valid manifest. All three mean the same thing. Re-authenticate with vcf context use my-org, then switch back to your namespace context.

    4. Understanding the two Subnet kinds

    Before writing any YAML, know which object you are creating. VCFA 9 exposes two distinct Subnet kinds in different API groups, corresponding to the two areas of the VCF Automation UI. They are not interchangeable.

    Build & DeployManage & Govern
    ContextNamespaceOrganization
    API groupcrd.nsx.vmware.com/v1alpha1vpc.nsx.vmware.com/v1alpha1
    ScopeSupervisor NamespaceVPC
    Name formatdb-tier<vpc>:db-tier
    Additional required fields—regionName, vpcName

    Both support custom gateways, so the technique here applies at either level. Most consumers want the namespace-scoped object; the VPC-scoped one is for governance workflows.

    Diagnosing the wrong layer “The server doesn’t have a resource type” almost always means you are on the wrong context rather than that the object does not exist. Check with kubectl config current-context before assuming a CRD is missing.

    5. Creating a subnet with a custom gateway

    By default NSX assigns the first usable address in the range — a subnet allocated 172.30.0.192/27 receives gateway 172.30.0.193. To choose a different address you must supply both the CIDR and the gateway, because the gateway is only meaningful relative to a known range.

    Namespace-scoped (Build & Deploy):

    apiVersion: crd.nsx.vmware.com/v1alpha1
    kind: Subnet
    metadata:
      name: db-tier
      namespace: phoenix-ns-kj9g6
    spec:
      accessMode: Private
      ipAddresses:
        - 172.30.1.224/27
      advancedConfig:
        gatewayAddresses:
          - 172.30.1.240/27

    VPC-scoped (Manage & Govern), run against the organization context:

    apiVersion: vpc.nsx.vmware.com/v1alpha1
    kind: Subnet
    metadata:
      name: default-region-west:db-tier
      namespace: team-phoenix
    spec:
      regionName: region-west
      vpcName: default-region-west
      accessMode: Private
      ipAddresses:
        - 172.30.1.224/27
      advancedConfig:
        gatewayAddresses:
          - 172.30.1.240/27
    kubectl create -f subnet.yaml
    Use create, not apply kubectl apply writes a last-applied-configuration annotation that this API rejects outright with “Annotation updates are not supported.” Use kubectl create for new objects and kubectl edit or kubectl patch for changes. Inside a VKS guest cluster, ordinary apply works normally — the restriction applies to the VCF Automation API only.

    The gateway may sit at any usable address within the CIDR. It does not have to be the first.

    6. Modifying an existing subnet

    Contrary to what the greyed-out NSX Manager field suggests, the gateway can also be changed after creation:

    kubectl edit subnet db-tier

    Or without an interactive editor:

    kubectl patch subnet db-tier --type=merge `
      -p '{"spec":{"advancedConfig":{"gatewayAddresses":["172.30.1.250/27"]}}}'
    PowerShell and inline JSON Wrap the JSON in single quotes and use plain double quotes inside. Escaping the inner quotes with backslashes — the pattern that works in bash — produces “invalid character ‘\\’ looking for beginning of object key string” here.

    Changing the gateway alters routing for every workload on the subnet, so treat it as a change with a maintenance window rather than a routine edit.

    7. Verifying realisation

    This is the step most worth building into your habits, and it generalises well beyond gateways.

    kubectl create and kubectl edit return success the moment the Kubernetes API accepts the object. NSX realisation happens afterwards, asynchronously, and it can fail silently.

    kubectl get subnet db-tier -o jsonpath='{.status.conditions}'

    A healthy subnet:

    [{"reason":"SubnetReady","status":"True","type":"Ready",
      "message":"NSX Subnet with DHCPDeactivated has been successfully created/updated"}]

    A failed one carries the real NSX error in the condition message:

    [{"reason":"SubnetNotReady","status":"False","type":"Ready",
      "message":"... IpAddressBlocks path=[...] do not have spare capacity
       to allocate CIDR of size 32 and startIp 172.30.0.160."}]

    Confirm the realised addressing too, which is where you will see whether your requested gateway was honoured:

    kubectl get subnet db-tier -o jsonpath='{.status.networkAddresses}{"\n"}{.status.gatewayAddresses}'
    A failed subnet still looks created It appears in kubectl get subnets with a name, an access mode, and a spec. It simply has no network behind it. Any automation against this API should poll status.conditions rather than trusting the command’s exit code.

    8. Choosing a valid CIDR

    Specifying ipAddresses disables automatic allocation — ipv4SubnetSize is then ignored entirely, and selecting a valid free range becomes your responsibility.

    Each access mode draws from a different IP block, and the blocks do not overlap:

    Access modeAddress sourceReachable from
    PrivateThe VPC’s own private rangeWithin the VPC; SNAT applied on egress
    PrivateTGWA separate transit-gateway blockOther VPCs on the same transit gateway — routed, not NAT’d
    PublicThe NSX Project’s external blockNatively routed beyond the VPC

    Choose from the wrong pool and NSX names the block it expected:

    CIDR 172.30.1.64/27 does not belong to IpAddressBlocks
    path=[/orgs/default/projects/.../ip-blocks/default-tgw-region-west]

    To find free space, list what is already allocated:

    kubectl get subnets,subnetsets -o custom-columns=`
    'NAME:.metadata.name,CIDR:.status.networkAddresses'
    Allocation is sequential NSX allocates from the next free boundary rather than filling gaps. Apparent gaps in your namespace’s view are usually claimed by other namespaces sharing the same VPC, which you cannot see from a tenant context. Step past the highest allocated block rather than attempting to reuse a gap — and if you hit “do not have spare capacity” on a range that looks free, that is why.

    Summary

    TaskCommand or field
    Authenticatevcf context create <org> --type cci --api-token $token
    Switch to namespacekubectl config use-context <org>:<ns>:<project>
    Set gateway at creationspec.advancedConfig.gatewayAddresses plus spec.ipAddresses
    Change it laterkubectl edit subnet <name>
    Verifykubectl get subnet <name> -o jsonpath='{.status.conditions}'
    Find free rangeskubectl get subnets,subnetsets with status.networkAddresses

    If you are new to this API, the companion post Your First VM and Kubernetes Cluster in VCF Automation walks through projects, namespaces, virtual machines and VKS clusters using the same toolchain.

  • Your First VM and Kubernetes Cluster in VCF Automation — Without Touching the UI

    Your First VM and Kubernetes Cluster in VCF Automation — Without Touching the UI

    If you administer vSphere, you already know how to build a VM. This post shows you how to build the same VM in VMware Cloud Foundation Automation using nothing but a command line — no UI clicks, no cloud template, no prior Kubernetes experience assumed. It takes about less then 10 minutes, and by the end you’ll have a running web server with a load-balanced IP that answers from your browser.

    I’m deliberately keeping this simple. There is a longer, deeper reference post for the full API surface, but this one is the on-ramp: the shortest honest path from “I’ve never used the VCF Automation API” to “I just built something with it.”

    What we’re going to do
    1. Install the CLI and log in to your organization
    2. Create a project (the tenancy boundary for a team)
    3. Create a Supervisor Namespace (where workloads actually live)
    4. Create a VM with cloud-init that installs nginx
    5. Publish it behind a load balancer and browse to it
    6. Build a Kubernetes cluster in the same namespace, and run a workload on it

    Why bother, when the UI works fine?

    Fair question, and I’d give three honest answers.

    Repeatability. A VM you clicked into existence is a VM nobody can rebuild exactly. A VM defined in a YAML file is one you can commit, review, and recreate identically a year later.

    It’s where the platform is going. In VCF 9, the consumption layer is a Kubernetes-style API. The UI is a client of that API, not the other way round. Learning the objects underneath means the UI stops being magic.

    It’s the foundation for automation. Everything you’d want later — GitOps, pipelines, self-service catalogs — is built on these same objects. This is the first rung.

    None of that means you should stop using the UI. I use both. But knowing what’s underneath makes you considerably harder to stump.

    Before you start

    WhatWhere to get it
    VCF CLI (the vcf command)Broadcom Support portal — it replaces the older CCI kubectl plugin
    kubectlAny recent version; the CLI generates a kubeconfig for it
    An API tokenLog into your VCF Automation org portal → your user menu → API Tokens → Generate. Copy it — it’s shown once.
    Org admin rightsNeeded to create projects. If you only have consumer rights, skip to Step 4 and use an existing namespace.

    Everything below is run from PowerShell on Windows. It works identically on Linux and macOS; only the quoting changes.

    Step 1 — Log in

    First, store your token in a variable. Note the quotes — without them PowerShell tries to execute the token as a command, which produces a confusing error.

    $vcfa_token = "your-api-token-here"

    Now create a context. This is the CLI’s word for “a saved connection to an environment.”

    vcf context create org2-ctgw `
      --endpoint https://auto-a.site-a.vcf.lab `
      --type cci `
      --auth-type basic `
      --tenant-name Org2-CTGW `
      --api-token $vcfa_token `
      --insecure-skip-tls-verify

    Substitute your own endpoint and tenant name. The backtick is PowerShell’s line-continuation character — or just put it all on one line. Use --insecure-skip-tls-verify only in a lab; in production supply your CA certificate with --ca-certificate instead.

    VCF CLI context create command output showing successful login and available contexts
    A successful login. Note that the CLI lists every context you now have access to — one for the organization, plus one for each Supervisor Namespace you can reach.

    That output is worth reading carefully, because it explains the whole model. You get an organization context (org2-ctgw) and a namespace context for each namespace you can use (org2-ctgw:org2-ns1-4hm32:default-project).

    The one concept to take away There are two levels. The organization context is where you manage tenancy — projects, namespaces, who gets what. The namespace context is where you build things — VMs, clusters, networks. Almost every “why can’t I see that?” moment is being in the wrong one. If a command returns “the server doesn’t have a resource type”, you’re at the org level asking a workload question.

    Step 2 — Create a project

    A project is the boundary you’d hand to a team. Save this as project.yml:

    apiVersion: project.cci.vmware.com/v1alpha2
    kind: Project
    metadata:
      name: team-phoenix
    spec:
      description: Phoenix dev team
    kubectl --context org2-ctgw create -f project.yml
    # project.project.cci.vmware.com/team-phoenix created
    Use create, not apply This will save you a puzzling ten minutes. kubectl apply stores a tracking annotation on the object, and this API rejects it outright with “Annotation updates are not supported.” Everywhere in this post, use kubectl create.

    Step 3 — Create a namespace

    The project is just a boundary. A Supervisor Namespace is the thing with actual capacity — CPU, memory, storage, a network — where your VMs will live. Think of it as the resource pool plus the network plus the quota, expressed as one object.

    You need four pieces of information, all of which you can read off an existing namespace or get from whoever set up your environment:

    FieldWhat it meansExample
    regionNameWhich region (a vCenter/Supervisor grouping)region-west
    classNameThe T-shirt size defined by your adminsmall
    vpcNameThe NSX VPC providing the networkdefault-region-west
    classConfigOverridesThe zone and storage class, with limitssee below
    apiVersion: infrastructure.cci.vmware.com/v1alpha3
    kind: SupervisorNamespace
    metadata:
      generateName: phoenix-ns-
      namespace: team-phoenix
    spec:
      regionName: region-west
      className: small
      vpcName: default-region-west
      classConfigOverrides:
        storageClasses:
        - name: vsan-default-storage-policy
          limit: 204800Mi
        zones:
        - name: z-wld-a
          cpuLimit: 15000M
          cpuReservation: 0M
          memoryLimit: 15000Mi
          memoryReservation: 0Mi
    kubectl --context org2-ctgw create -f namespace.yml
    kubectl --context org2-ctgw get supervisornamespaces -n team-phoenix

    Wait for STATUS: Created. Then refresh your contexts so the new namespace appears, and switch into it:

    vcf context use org2-ctgw
    kubectl config get-contexts
    kubectl config use-context org2-ctgw:phoenix-ns-kj9g6:team-phoenix
    Two things that catch people here The namespace name is generated with a random suffix (phoenix-ns-kj9g6), so read it from the previous command rather than guessing. And the context name is the full three-part string <org>:<namespace>:<project> — using just the namespace name gives you “no context exists.”

    You can confirm the same thing in the UI, which is a reassuring way to check your work while you’re learning:

    VCF Automation UI showing the newly created namespace phoenix-ns-kj9g6 in project team-phoenix, Active, with CPU, memory and storage quota bars
    The namespace created from the command line, shown in the VCF Automation UI — Active, in the team-phoenix project, with its CPU, memory and storage allocation. The API and the UI are two views of the same object.
    Why the long classConfigOverrides block? When you create a namespace through the portal, it fills these values in for you from the class. Creating one through the API, you have to state them yourself — and the zone entry needs all four CPU and memory fields, not just a name. If you leave them out, the API tells you exactly what’s missing, one field at a time. That’s actually a decent way to learn the object: submit a minimal version and let the error messages guide you.

    Step 4 — Create a VM

    Now the part that will feel familiar. We need three things: a size, an image, and a storage policy. Ask the namespace what it has:

    kubectl get virtualmachineclasses     # the sizes available to you
    kubectl get virtualmachineimages      # images from your content library
    kubectl get clustervirtualmachineimages | Select-String "ubuntu"
    If virtualmachineimages comes back empty, you’re not stuck. That list is empty when no content library is attached to your namespace class. You can still deploy from a cluster-wide image by referencing its vmi- identifier directly, which is what I do below. Getting a content library attached is the better long-term answer — it gives you friendly names instead of hashes — but it isn’t a blocker today.

    Here’s the VM, together with a cloud-init secret that creates a login and installs nginx. Save both in one file:

    apiVersion: v1
    kind: Secret
    metadata:
      name: web01-bootstrap
    stringData:
      user-data: |
        #cloud-config
        users:
        - name: ops
          sudo: ALL=(ALL) NOPASSWD:ALL
          shell: /bin/bash
          ssh_authorized_keys:
          - ssh-ed25519 AAAA...your-key-here
        packages: [nginx]
        runcmd:
        - systemctl enable --now nginx
    ---
    apiVersion: vmoperator.vmware.com/v1alpha5
    kind: VirtualMachine
    metadata:
      name: web01
      labels:
        app: web
    spec:
      className: best-effort-small
      imageName: vmi-5dcfa60c7cc579296
      storageClass: vsan-default-storage-policy
      powerState: PoweredOn
      bootstrap:
        cloudInit:
          rawCloudConfig:
            name: web01-bootstrap
            key: user-data
    kubectl create -f vm.yml
    PowerShell output showing the VirtualMachine manifest and confirmation that the secret and virtual machine were created
    Two objects, one command: the cloud-init secret and the VM itself.

    Watch it come up. The VM will report an IP address once VMware Tools starts reporting:

    kubectl get vm web01
    kubectl get vm web01 -o jsonpath='{.status.network.primaryIP4}'
    # 172.30.0.66

    Notice what you didn’t have to specify: no port group, no IP address, no datastore. The VM picked up an address from the namespace’s default network automatically and landed on storage matching the policy you named. That’s the platform doing the placement work, exactly as it does for a Kubernetes pod.

    Cloud-init only runs on first boot You cannot add a bootstrap section to a VM that already exists. If you forget it, the VM boots fine but has no user account and no way in — you delete and recreate. Decide your bootstrap before you provision.

    Step 5 — Publish it

    A VirtualMachineService puts a load-balanced IP in front of one or more VMs, selected by label. Our VM carries app: web, so the service will find it.

    apiVersion: vmoperator.vmware.com/v1alpha5
    kind: VirtualMachineService
    metadata:
      name: web-lb
    spec:
      type: LoadBalancer
      selector:
        app: web
      ports:
      - name: http
        port: 80
        targetPort: 80
        protocol: TCP
    PowerShell output showing the VirtualMachineService manifest and the service created as type LoadBalancer
    The load balancer service, created. NSX allocates the external address within seconds.
    Reading the external IP kubectl get vmservice prints only name, type and age — the address isn’t in its default columns. Use kubectl get svc instead, or ask for the field directly with -o jsonpath='{.status.loadBalancer.ingress[0].ip}'.

    Step 6 — Confirm it works

    kubectl get svc
    # web-lb   LoadBalancer   172.29.255.24   40.60.0.4   80/TCP   2m31s
    
    kubectl get endpoints
    # web-lb   172.30.0.66:80   2m34s
    
    curl.exe http://40.60.0.4
    PowerShell output showing the service endpoints and a curl request returning the nginx welcome page
    The whole chain proven: external IP 40.60.0.4, backend 172.30.0.66:80, and nginx answering.

    That’s the full path working. The load balancer address is allocated from the NSX VPC, the endpoint is your VM’s own IP, and nginx — installed by cloud-init at first boot — is serving. Nothing in that sequence involved the UI.

    Step 7 — Build a Kubernetes cluster

    Here’s the part that surprises people: a Kubernetes cluster is just another object in the same namespace you’ve been working in. Same context, same kubectl create, same mental model. You describe the cluster you want and the platform builds the nodes.

    You need to know two things first — and both are places where a sensible-looking manifest fails.

    Where the cluster template lives. Clusters are built from a ClusterClass, which is a template your platform team maintains. Crucially, it does not live in your namespace:

    kubectl get clusterclasses
    # NAME                     VARIABLES READY   AGE
    # builtin-generic-v3.1.0   True              164m
    # builtin-generic-v3.2.0   True              164m
    # builtin-generic-v3.3.0   True              164m

    That list is not the whole story. The class you actually want may live in a shared system namespace and won’t appear here at all — you reference it explicitly by name and namespace.

    Which Kubernetes version you can use. Ask the platform rather than guessing:

    kubectl get kubernetesreleases

    You’ll get a long list. Only the rows showing both READY: True and COMPATIBLE: True are usable. Pick one of those.

    Now the cluster itself — one control plane node, one worker, deliberately small:

    apiVersion: cluster.x-k8s.io/v1beta2
    kind: Cluster
    metadata:
      name: phoenix-k8s
    spec:
      clusterNetwork:
        pods:
          cidrBlocks: ["192.168.156.0/20"]
        services:
          cidrBlocks: ["10.96.0.0/12"]
        serviceDomain: cluster.local
      topology:
        classRef:
          name: builtin-generic-v3.6.0
          namespace: vmware-system-vks-public
        version: v1.35.5---vmware.1-vkr.1
        variables:
        - name: vmClass
          value: best-effort-medium
        - name: storageClass
          value: vsan-default-storage-policy
        controlPlane:
          replicas: 1
          metadata:
            annotations:
              run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-96cb71ec140d192ad
        workers:
          machineDeployments:
          - class: node-pool
            name: phoenix-k8s-np-1
            replicas: 1
            metadata:
              annotations:
                run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-96cb71ec140d192ad
    kubectl create -f cluster.yml
    kubectl get cluster phoenix-k8s -w

    Provisioning takes five to six minutes for this size. PHASE: Provisioned appears early and only means the objects were created — the column to watch is AVAILABLE, which flips to True once the control plane is genuinely up. kubectl get machines gives you the per-node view while you wait.

    Three things in that manifest are not obvious 1. classRef needs a namespace. The ClusterClass lives in a shared system namespace, and without it you’ll get a “not found” even though the class plainly exists.
    2. The version string is the long form — v1.35.5---vmware.1-vkr.1, with three dashes, not the +vmware.1 form shown elsewhere.
    3. The resolve-os-image annotation is often required. If your environment has the same node image registered in more than one content library — common in labs that have been upgraded a few times — the resolver refuses to guess and fails with “Multiple OSImages resolved.” Adding content-library= narrows it to one. You’ll find the library ID in your Content Libraries list.
    The shortcut worth knowing Rather than assembling this by hand, start a cluster creation in the VCF Automation UI and look at the YAML it generates. It will show you the right ClusterClass namespace, the right version format, and any annotations your environment needs — all correct for your specific build. Copy that, then edit it. This is far quicker than reverse-engineering from error messages, and it’s how I arrived at the manifest above.

    Step 8 — Get onto the cluster and run something

    Once AVAILABLE is True, download the cluster’s kubeconfig from the VCF Automation UI (the cluster’s detail page offers it) and point kubectl at it:

    kubectl --kubeconfig .\phoenix-k8s.yaml get nodes
    # NAME                        STATUS   ROLES           AGE   VERSION
    # phoenix-k8s-...             Ready    control-plane   12m   v1.35.5+vmware.1
    # phoenix-k8s-...-np-1-...    Ready    <none>          9m    v1.35.5+vmware.1

    You’re now a cluster administrator on an ordinary, conformant Kubernetes cluster. Everything you know about Kubernetes applies from here.

    With one important exception, which is worth meeting deliberately rather than by accident.

    Your first deployment will probably be rejected — and that’s correct

    Try the obvious thing:

    kubectl --kubeconfig .\phoenix-k8s.yaml create deployment web --image=nginx --replicas=2

    The deployment is created, but no pods ever appear. Looking at the ReplicaSet tells you why:

    pods "web-..." is forbidden: violates PodSecurity "restricted:latest":
      allowPrivilegeEscalation != false
      unrestricted capabilities (must set capabilities.drop=["ALL"])
      runAsNonRoot != true
      seccompProfile (must set seccompProfile.type to "RuntimeDefault")

    This is Pod Security Admission enforcing the restricted profile, and it’s switched on by default. A stock container image that runs as root is refused at creation — not warned about, refused. For a vSphere admin used to “deploy first, harden later,” this is a genuine change of habit, and a welcome one: the cluster will not run a privileged workload just because you forgot.

    The fix is to declare what the workload actually needs. Each field below answers one of the four complaints, and the image is swapped for a non-root variant that listens on 8080 rather than binding port 80 as root:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: web
      template:
        metadata:
          labels:
            app: web
        spec:
          securityContext:
            runAsNonRoot: true
            seccompProfile:
              type: RuntimeDefault
          containers:
          - name: nginx
            image: nginxinc/nginx-unprivileged:stable
            ports:
            - containerPort: 8080
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop: ["ALL"]
    kubectl --kubeconfig .\phoenix-k8s.yaml apply -f web.yml
    kubectl --kubeconfig .\phoenix-k8s.yaml expose deployment web --port=80 --target-port=8080 --type=LoadBalancer
    kubectl --kubeconfig .\phoenix-k8s.yaml get svc web

    The external IP is allocated by the same NSX load balancer that served your VM in Step 5 — usually within seconds. Confirm there are backends behind it:

    kubectl --kubeconfig .\phoenix-k8s.yaml get endpointslices -l kubernetes.io/service-name=web
    # ADDRESSTYPE   PORTS   ENDPOINTS
    # IPv4          8080    192.168.147.3,192.168.145.12
    Note that kubectl apply works here. The “use create, not apply” rule applies to the VCF Automation API only. Inside a VKS cluster you’re talking to a normal Kubernetes API server, and all your usual habits — apply, Helm, Kustomize — work exactly as expected.

    The five things that will trip you up

    These cost me time so they don’t have to cost you any.

    1. Use kubectl create, never apply. The API rejects the annotation that apply depends on.
    2. Context names are three parts. <org>:<namespace>:<project>, not just the namespace.
    3. Know which level you’re at. “The server doesn’t have a resource type” almost always means you’re at the org context asking a workload question, or vice versa.
    4. Tokens expire mid-session. A sudden Unauthorized, or a strange complaint about downloading the OpenAPI schema, usually just means your session lapsed. vcf context use <org> re-authenticates.
    5. Check the API version against your build. Published examples lag. On the environment used here, namespaces are served at v1alpha3 and VMs at v1alpha5. The most reliable way to learn the current shape of any object is to dump a working one: kubectl get vm <name> -o yaml. Copy the structure, change the values.

    That last one is worth internalising as a general habit. Rather than hunting for documentation that matches your exact release, clone something that already works in your own environment. It is always correct, by definition.

    Where to go next

    You now have the pattern for everything else on this platform. The same two-level model, the same create workflow, and the same “clone a working object” trick apply to Kubernetes clusters, persistent volumes, NSX security policies, and micro-segmentation rules — all of which are just more objects in the same API.

    A few natural next steps, in the order I’d take them:

    • Delete and rebuild what you just made. Change the VM size in the YAML, recreate, and watch it come back identically. That’s the payoff of declarative infrastructure, and it’s more convincing when you do it than when you read about it.
    • Scale the cluster. Change replicas on the worker pool, re-apply, and watch nodes roll in. Then bump version to a newer compatible release and watch a rolling upgrade — that’s a one-line change doing something that used to be a project.
    • Write a security policy as YAML and watch NSX distributed firewall rules appear.
    • Put it all in Git. Once your infrastructure is files, a GitOps engine can reconcile it. That’s the subject of my next post.

    If you’re a vSphere admin who has been eyeing all this with mild suspicion, I’d encourage you to run through the steps above once in a lab. The concepts are less foreign than the vocabulary suggests — a namespace is a resource pool with a network and a quota, a VM is still a VM, and a load balancer service is the thing you’d have raised a ticket for. The difference is that now it’s a file you own.

    Questions or something that didn’t work in your environment? Leave a comment — I read all of them, and environment-specific quirks are exactly what makes these posts better.

  • Creating NSX distributed firewall rules through the VCFA 9.1 CCI API

    Creating NSX distributed firewall rules through the VCFA 9.1 CCI API

    On VCF Automation 9.1, the Cloud Consumption Interface turns an NSX distributed firewall policy into what is, mechanically, just a Kubernetes object — you POST a manifest and NSX enforces it. That means firewall-as-code without ever logging into NSX Manager. This is the complete, tested toolkit: every Postman request, a browser helper, and two scripts that push a whole rule set from a CSV. It also includes the three walls I hit — a 404, a PowerShell quirk, and a dependency error — because you’ll hit them too, and the errors don’t say what they mean.

    Who this is for

    VCF architects and platform engineers automating east-west segmentation on VCFA 9.1. If you’re building a real GitOps pipeline, the idiomatic tool is vcf-cli plus kubectl apply against these same resources. What follows is the REST-level view — the exact request shapes, clicked through in Postman — useful when you want to understand what’s on the wire, or bulk-load rules from a spreadsheet a human maintains.

    What the CCI actually exposes

    The All-Apps organization in VCFA 9.1 fronts a Kubernetes-style API at /cci/kubernetes. NSX VPC objects appear on it as Custom Resource Definitions in the vpc.nsx.vmware.com/v1alpha1 group. Distributed firewall policy is the FirewallPolicy kind; security groups are NetworkSecurityGroup; named services are NetworkService. You address them with ordinary REST verbs and a bearer token, and the platform realizes your intent as genuine NSX vDefend policy.

    firewallpolicies
    The DFW policy resource (cluster-scoped)
    networksecuritygroups
    Groups you reference from rules
    status.Realized
    Enforcement signal, inline on every object
    AUTHENTICATE API token → exchange or paste a bearer token Authorization: Bearer … POST FirewallPolicy /cci/kubernetes/apis/ …/v1alpha1/firewallpolicies cluster-scoped · no namespace NSX vDEFEND DFW policy + rules Realized: True PREREQUISITE groups must exist before the policy Token → POST → realized policy (groups first)
    The full path. Everything below fills in each box, plus the errors between them.
    Postman setup

    Create an environment with these variables so the requests below resolve: vcfaHost, vcfaTenant, apiToken (durable, from the portal), accessToken (filled by the token request’s script), and namespace is not needed — this resource is cluster-scoped. Select the environment in the top-right dropdown. For a *.vcf.lab self-signed cert, turn off Settings → SSL certificate verification. If you’re in the signed-out Lightweight client (no environments or scripts), just paste literal values instead of {{variables}}.

    1
    Authentication

    Get a bearer token

    Every request carries an Authorization: Bearer header. Mint an API token once in the VCFA portal (account settings → API Tokens, shown only once), then exchange it for a short-lived access token. The exchange is an OAuth refresh-token grant against the tenant manager:

    POST  https://{{vcfaHost}}/tm/oauth/tenant/{{vcfaTenant}}/token
    # Headers
    Content-Type: application/x-www-form-urlencoded
    Accept:       application/json
    
    # Body → x-www-form-urlencoded
    grant_type      refresh_token
    refresh_token   {{apiToken}}

    Capture the token automatically so every later request can use it — add this on the request’s Scripts → Post-response tab:

    Scripts → Post-response
    const res = pm.response.json();
    pm.environment.set("accessToken", res.access_token);   // some builds: res.token
    Postman POST to the tenant token endpoint with a refresh_token grant, returning access_token in the 200 response
    The token exchange in Postman — an x-www-form-urlencoded POST returns access_token (a Bearer token), valid for expires_in seconds (3600 here).

    Already holding a valid bearer token from the portal or a prior exchange? Skip all of the above and paste it straight into the Authorization header (or the scripts’ -AccessToken parameter). Both scripts below support that.

    400 vs 401

    A 400 on the exchange endpoint usually means you fed it something that isn’t a refresh token (an access token, say). A 401 later, on the actual API calls, means your access token expired — re-run the token request to refresh {{accessToken}}. The durable API token doesn’t expire on the same cadence; the access token does.

    2
    The endpoint

    Discover the resource — and learn it’s cluster-scoped

    Here’s the first wall. POST to what looks like the obvious path with a namespace segment and you get a 404 whose details block is empty — meaning the route itself isn’t served, not that your object is missing. Don’t guess the path; ask the API server what it serves:

    GET  https://{{vcfaHost}}/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1
    # Headers
    Authorization: Bearer {{accessToken}}
    Accept:        application/json

    In the response, find the firewallpolicies entry:

    response · APIResourceList excerpt
    {
      "name": "firewallpolicies",
      "singularName": "firewallpolicy",
      "namespaced": false,
      "kind": "FirewallPolicy",
      "verbs": ["create","delete","get","list","patch","update"]
    }
    Postman discovery GET showing the firewallpolicies resource with namespaced set to false and full create/get/patch/update verbs
    Discovery confirms firewallpolicies is namespaced: false — cluster-scoped, so the request path carries no /namespaces/ segment. Adding one is what produces the empty-details 404. The CCI proxy is already per-tenant; your org scoping rides on the token.
    3
    CRUD in Postman

    Create, read, update, delete

    Create a policy by POSTing to the collection. Trim metadata to just the name — uid, resourceVersion, and creationTimestamp are server-owned, and a non-empty resourceVersion can be rejected on create:

    POST  https://{{vcfaHost}}/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1/firewallpolicies
    # Headers
    Authorization: Bearer {{accessToken}}
    Content-Type:  application/json
    
    # Body → raw · JSON
    {
      "kind": "FirewallPolicy",
      "apiVersion": "vpc.nsx.vmware.com/v1alpha1",
      "metadata": { "name": "policy-web01" },
      "spec": {
        "appliedTo": { "groupNames": ["Any"] },
        "category": "Environment",
        "regionName": "region-west",
        "priority": 500000,
        "stateful": true, "tcpStrict": false, "locked": false,
        "rules": [
          {
            "name": "allow-web-in", "direction": "In", "action": "Allow", "ipProtocol": "IPV4",
            "from": [{ "ipAddress": "10.30.0.0/24" }],
            "to":   [{ "ipAddress": "192.168.10.20" }],
            "appliedTo": { "groupNames": ["Any"] },
            "services": [{ "networkServiceName": "Any" }],
            "disabled": false
          }
        ]
      }
    }

    Read. A crucial quirk: the list response omits each policy’s rules array — you get only policy-level fields and status. To see the rules, GET the policy by name:

    GET  list  ·  by-name
    # List — no rules array, but shows status.ruleCount + Realized per policy
    GET  https://{{vcfaHost}}/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1/firewallpolicies
    
    # By name — includes the full rules array
    GET  https://{{vcfaHost}}/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1/firewallpolicies/policy-web01
    
    # Headers (both)
    Authorization: Bearer {{accessToken}}

    Update is read-modify-write: GET the object, change what you need, PUT the whole thing back echoing the current resourceVersion. A PUT replaces the entire rules array, so send them all — an omitted rule is a deleted rule.

    PUT  https://{{vcfaHost}}/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1/firewallpolicies/policy-web01
    # Headers
    Authorization: Bearer {{accessToken}}
    Content-Type:  application/json
    
    # Body → raw · JSON  (full object; paste from your GET, echo resourceVersion)
    {
      "kind": "FirewallPolicy",
      "apiVersion": "vpc.nsx.vmware.com/v1alpha1",
      "metadata": { "name": "policy-web01", "resourceVersion": "<from the GET>" },
      "spec": {
        "appliedTo": { "groupNames": ["Any"] },
        "category": "Environment", "regionName": "region-west", "priority": 500000,
        "stateful": true, "tcpStrict": false, "locked": false,
        "rules": [ /* ALL rules here — the array is replaced wholesale */ ]
      }
    }
    Postman PUT updating policy-web01, with a 200 response whose status shows Realized False and reason InProgress
    The update PUT in Postman, echoing resourceVersion. The response is a 200 — but status.Realized is still False, reason InProgress: the API accepted the intent, NSX hasn’t finished enforcing it yet. Poll the GET until it flips to True.
    DELETE  https://{{vcfaHost}}/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1/firewallpolicies/policy-web01
    # Headers
    Authorization: Bearer {{accessToken}}
    Realization is the real check

    The API returns 200 when it accepts your intent, not when NSX has enforced it. Every object carries the truth in status.conditions[].type == "Realized", alongside status.ruleCount. After any write, GET the policy and confirm Realized: True and the count you expect. No separate realized-state endpoint needed.

    4
    The schema

    Rule fields, and the IP-or-group union

    Source and destination members are a union type — a member is either a literal address or a group reference. This matters: for simple, literal rules you don’t need to pre-create any groups.

    Member styleExampleUse when
    ipAddress (host){ "ipAddress": "192.168.10.20" }A single literal host
    ipAddress (CIDR){ "ipAddress": "10.30.0.0/24" }A literal subnet
    groupName{ "groupName": "web-servers" }Dynamic or reused membership
    FieldValues
    directionIn, Out, InOut
    actionAllow, Drop (silent), Reject (informs client)
    ipProtocolIPV4, IPV6, IPV4_IPV6
    servicesnetworkServiceName refs — Any or an existing named service (read-only via this API)
    category (policy)Eval order: Ethernet → Emergency → Infrastructure → Environment → Application
    priority (policy)Lower number = evaluated first, within the category band
    5
    Dependencies

    Groups must exist before the policy that references them

    The second wall. If a rule references groupName: "web-servers" and that group doesn’t exist, the create fails with a 400 whose message spells out the exact policy path NSX couldn’t resolve. Groups are a sibling resource — cluster-scoped, full create verbs — so you provision them the same way, first:

    POST  https://{{vcfaHost}}/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1/networksecuritygroups
    # Headers
    Authorization: Bearer {{accessToken}}
    Content-Type:  application/json
    
    # Body → raw · JSON
    {
      "kind": "NetworkSecurityGroup",
      "apiVersion": "vpc.nsx.vmware.com/v1alpha1",
      "metadata": { "name": "web-servers" },
      "spec": {
        "regionName": "region-west",
        "criteria": [ { "ipAddresses": ["10.20.0.0/24"] } ]
      }
    }
    Confirm the group spec on your build

    The criteria shape can be IP-based (above) or condition/tag-based for dynamic membership. Create one and GET it back to confirm the exact schema your 9.1 build expects before scripting a batch. The group and the policy that references it must share the same regionName, or the reference won’t resolve.

    6
    Bulk from CSV

    Drive a whole rule set from a spreadsheet

    Now the payoff. Postman is great for single requests, but a whole rule set wants a script. Keep your rules in a CSV, one row per rule, grouped by a policy column. Multiple members in a cell are separated by ;. Row order is preserved as evaluation order, so a catch-all drop belongs last.

    rules.csv
    policy,name,direction,action,protocol,from,to,service,appliedTo,disabled,category,priority,region
    policy-web01,allow-web-in,In,Allow,IPV4,10.30.0.0/24,192.168.10.20,Any,Any,false,Environment,500000,region-west
    policy-web01,allow-app-out,Out,Allow,IPV4,192.168.10.20,10.30.0.5,Any,Any,false,,,
    policy-web01,allow-dns,Out,Allow,IPV4,192.168.10.20,10.10.10.53;10.10.10.54,Any,Any,false,,,
    policy-web01,default-drop,InOut,Drop,IPV4,0.0.0.0/0,0.0.0.0/0,Any,Any,false,,,
    policy-app02,app-from-web,In,Allow,IPV4,web-servers,app-servers,Any,app-servers,false,Environment,510000,region-west
    policy-app02,app-deny,InOut,Drop,IPV4,0.0.0.0/0,0.0.0.0/0,Any,Any,false,,,
    

    Two scripts consume that CSV identically. Both do the same loop: authenticate (exchange an API token, or use a bearer token you pass in), build a FirewallPolicy body per policy with IP-vs-group auto-detection, upsert (POST if new, PUT with resourceVersion if it exists), then poll Realized and report per policy. Both take --dry-run so you can inspect the JSON before anything hits the server, and --insecure for a self-signed lab cert.

    Python

    create_dfw_rules.py  ·  python3, requires requests
    #!/usr/bin/env python3
    """
    create_dfw_rules.py
    
    Create or update VCF Automation (CCI) distributed firewall policies and rules
    from a CSV file.
    
    Flow per run:
      1. Exchange the VCFA API token for a short-lived access (bearer) token.
      2. Read the CSV and group rows by policy name.
      3. Build a FirewallPolicy body per policy (auto-detecting IP/CIDR vs group name).
      4. Upsert: GET the policy -> POST if absent, PUT (with resourceVersion) if present.
      5. Re-read and report the Realized status and rule count.
    
    Endpoint (cluster-scoped, no namespace):
      POST/PUT  https://<host>/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1/firewallpolicies[/<name>]
    
    Config is read from environment variables (override with flags):
      VCFA_HOST       e.g. auto-a.site-a.vcf.lab
      VCFA_TENANT     your All-Apps org / tenant name
      VCFA_API_TOKEN  the durable API token minted in the VCFA portal
    
    CSV columns (header required):
      policy,name,direction,action,protocol,from,to,service,appliedTo,disabled
    Optional per-policy columns (taken from the first row of each policy group):
      category,priority,region
    Multiple members in one cell are separated by ';'.
    Rule order in the CSV is preserved as evaluation order (first match wins).
    """
    
    import argparse
    import csv
    import json
    import os
    import re
    import sys
    import time
    
    import requests
    
    IP_RE = re.compile(r"^\d{1,3}(\.\d{1,3}){3}(/\d{1,2})?$")
    API_GROUP = "vpc.nsx.vmware.com/v1alpha1"
    
    
    # ---------- small helpers ----------
    
    def log(msg):
        print(msg, flush=True)
    
    
    def die(msg, code=1):
        print(f"ERROR: {msg}", file=sys.stderr, flush=True)
        sys.exit(code)
    
    
    def split_cell(cell):
        return [p.strip() for p in (cell or "").split(";") if p.strip()]
    
    
    def member(cell):
        """Turn a from/to cell into a list of {ipAddress|groupName} members."""
        out = []
        for v in split_cell(cell):
            out.append({"ipAddress": v} if IP_RE.match(v) else {"groupName": v})
        return out
    
    
    def services(cell):
        vals = split_cell(cell) or ["Any"]
        return [{"networkServiceName": s} for s in vals]
    
    
    def group_names(cell):
        vals = split_cell(cell) or ["Any"]
        return {"groupNames": vals}
    
    
    # ---------- token exchange ----------
    
    def get_access_token(session, host, tenant, api_token):
        url = f"https://{host}/tm/oauth/tenant/{tenant}/token"
        resp = session.post(
            url,
            headers={
                "Content-Type": "application/x-www-form-urlencoded",
                "Accept": "application/json",
            },
            data={"grant_type": "refresh_token", "refresh_token": api_token},
        )
        if resp.status_code != 200:
            die(f"token exchange failed [{resp.status_code}]: {resp.text[:300]}")
        body = resp.json()
        token = body.get("access_token") or body.get("token")
        if not token:
            die(f"token exchange returned no access_token/token field: {list(body)}")
        return token
    
    
    # ---------- CSV -> policy bodies ----------
    
    def build_policies(rows, defaults):
        """Group CSV rows by 'policy' and build a FirewallPolicy body for each.
        Returns an ordered dict {policy_name: body}."""
        order = []
        groups = {}
        for r in rows:
            name = (r.get("policy") or "").strip()
            if not name:
                die("a CSV row is missing the 'policy' column value")
            if name not in groups:
                groups[name] = []
                order.append(name)
            groups[name].append(r)
    
        policies = {}
        for pname in order:
            prows = groups[pname]
            head = prows[0]
            rules = []
            for r in prows:
                rules.append({
                    "name": (r.get("name") or "").strip(),
                    "direction": (r.get("direction") or "InOut").strip(),
                    "action": (r.get("action") or "Allow").strip(),
                    "ipProtocol": (r.get("protocol") or "IPV4").strip(),
                    "from": member(r.get("from")),
                    "to": member(r.get("to")),
                    "appliedTo": group_names(r.get("appliedTo")),
                    "services": services(r.get("service")),
                    "disabled": str(r.get("disabled", "")).strip().lower() == "true",
                })
            body = {
                "kind": "FirewallPolicy",
                "apiVersion": API_GROUP,
                "metadata": {"name": pname},
                "spec": {
                    "appliedTo": {"groupNames": ["Any"]},
                    "category": (head.get("category") or defaults["category"]).strip(),
                    "regionName": (head.get("region") or defaults["region"]).strip(),
                    "priority": int(head.get("priority") or defaults["priority"]),
                    "stateful": True,
                    "tcpStrict": defaults["tcp_strict"],
                    "locked": False,
                    "rules": rules,
                },
            }
            policies[pname] = body
        return policies
    
    
    # ---------- API calls ----------
    
    def policy_url(host, name=None):
        base = f"https://{host}/cci/kubernetes/apis/{API_GROUP}/firewallpolicies"
        return f"{base}/{name}" if name else base
    
    
    def get_policy(session, host, token, name):
        resp = session.get(
            policy_url(host, name),
            headers={"Authorization": f"Bearer {token}", "Accept": "application/json"},
        )
        if resp.status_code == 200:
            return resp.json()
        if resp.status_code == 404:
            return None
        die(f"GET {name} failed [{resp.status_code}]: {resp.text[:300]}")
    
    
    def upsert_policy(session, host, token, name, body):
        existing = get_policy(session, host, token, name)
        headers = {
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json",
            "Accept": "application/json",
        }
        if existing is None:
            resp = session.post(policy_url(host), headers=headers, data=json.dumps(body))
            verb = "created"
        else:
            rv = existing.get("metadata", {}).get("resourceVersion")
            if rv is not None:
                body["metadata"]["resourceVersion"] = rv
            resp = session.put(policy_url(host, name), headers=headers, data=json.dumps(body))
            verb = "updated"
        if resp.status_code not in (200, 201):
            die(f"{verb} {name} failed [{resp.status_code}]: {resp.text[:400]}")
        return verb
    
    
    def realized(policy):
        if not policy:
            return None, None
        conds = policy.get("status", {}).get("conditions", [])
        state = next((c.get("status") for c in conds if c.get("type") == "Realized"), None)
        return state, policy.get("status", {}).get("ruleCount")
    
    
    # ---------- main ----------
    
    def main():
        ap = argparse.ArgumentParser(description="Create/update CCI DFW policies from a CSV.")
        ap.add_argument("csv", help="path to the rules CSV")
        ap.add_argument("--host", default=os.environ.get("VCFA_HOST"))
        ap.add_argument("--tenant", default=os.environ.get("VCFA_TENANT"))
        ap.add_argument("--api-token", default=os.environ.get("VCFA_API_TOKEN"))
        ap.add_argument("--category", default="Environment", help="default category (default: Environment)")
        ap.add_argument("--priority", type=int, default=500000, help="default priority (default: 500000)")
        ap.add_argument("--region", default="region-west", help="default regionName (default: region-west)")
        ap.add_argument("--tcp-strict", action="store_true", help="set tcpStrict=true on policies (default false)")
        ap.add_argument("--insecure", action="store_true", help="skip TLS verification (self-signed lab certs)")
        ap.add_argument("--ca-cert", help="path to a CA bundle for TLS verification")
        ap.add_argument("--dry-run", action="store_true", help="print the JSON bodies and exit; no API calls")
        args = ap.parse_args()
    
        with open(args.csv, newline="", encoding="utf-8") as f:
            rows = list(csv.DictReader(f))
        if not rows:
            die("CSV has no data rows")
    
        defaults = {
            "category": args.category,
            "priority": args.priority,
            "region": args.region,
            "tcp_strict": args.tcp_strict,
        }
        policies = build_policies(rows, defaults)
    
        if args.dry_run:
            for name, body in policies.items():
                log(f"# ---- {name} ({len(body['spec']['rules'])} rules) ----")
                log(json.dumps(body, indent=2))
            log(f"\n{len(policies)} policy/policies parsed, "
                f"{sum(len(b['spec']['rules']) for b in policies.values())} rules total. (dry-run)")
            return
    
        for k in ("host", "tenant", "api_token"):
            if not getattr(args, k):
                die(f"missing --{k.replace('_','-')} (or its VCFA_* env var)")
    
        session = requests.Session()
        if args.ca_cert:
            session.verify = args.ca_cert
        elif args.insecure:
            session.verify = False
            requests.packages.urllib3.disable_warnings()
            log("WARNING: TLS verification disabled (--insecure)")
    
        log("Exchanging API token for access token ...")
        token = get_access_token(session, args.host, args.tenant, args.api_token)
    
        ok = True
        for name, body in policies.items():
            n = len(body["spec"]["rules"])
            log(f"\n> {name}: {n} rule(s)")
            verb = upsert_policy(session, args.host, token, name, body)
            # brief re-read to confirm realization
            state, count = None, None
            for _ in range(6):
                state, count = realized(get_policy(session, args.host, token, name))
                if state == "True":
                    break
                time.sleep(2)
            flag = "OK" if state == "True" else "PENDING"
            if state != "True":
                ok = False
            log(f"  {verb}; Realized={state} ruleCount={count}  [{flag}]")
    
        log("\nDone." if ok else "\nDone with warnings (some policies not yet Realized).")
        sys.exit(0 if ok else 2)
    
    
    if __name__ == "__main__":
        main()
    
    run it
    pip install requests
    export VCFA_HOST=auto-a.site-a.vcf.lab
    export VCFA_ACCESS_TOKEN=eyJ...        # or VCFA_TENANT + VCFA_API_TOKEN to auto-exchange
    
    python3 create_dfw_rules.py rules.csv --dry-run     # preview JSON, no calls
    python3 create_dfw_rules.py rules.csv --insecure    # create/update for real

    PowerShell

    Runs on stock Windows PowerShell 5.1 and on PowerShell 7+. One quirk to know about, because it produces a baffling error if you write this kind of code from scratch:

    The PowerShell array-nesting trap

    PowerShell’s ConvertTo-Json and its array handling fight each other. On a function return, a leading unary comma (return ,$out) is needed to stop the array unrolling to a scalar. But on a direct hashtable-value assignment, that same comma double-wraps — producing "groupNames": [["Any"]] instead of ["Any"], and NSX rejects it with “cannot unmarshal array into Go struct field … of type string”. The script below has this right: commas on returns, none on assignments. If you adapt it, keep that distinction.

    Create-DfwRules.ps1
    <#
    .SYNOPSIS
      Create or update VCF Automation (CCI) distributed firewall policies and rules from a CSV.
    
    .DESCRIPTION
      Per run:
        1. Exchange the VCFA API token for a short-lived access (bearer) token.
        2. Import the CSV and group rows by the 'policy' column.
        3. Build a FirewallPolicy body per policy (auto-detecting IP/CIDR vs group name).
        4. Upsert: GET the policy -> POST if absent, PUT (with resourceVersion) if present.
        5. Re-read and report Realized status + rule count.
    
      Endpoint (cluster-scoped, no namespace):
        https://<host>/cci/kubernetes/apis/vpc.nsx.vmware.com/v1alpha1/firewallpolicies[/<name>]
    
      Runs on Windows PowerShell 5.1 (built in) and PowerShell 7+.
    
    .PARAMETER Csv
      Path to the rules CSV.
    
    .EXAMPLE
      $env:VCFA_HOST='auto-a.site-a.vcf.lab'; $env:VCFA_TENANT='myorg'; $env:VCFA_API_TOKEN='...'
      .\Create-DfwRules.ps1 -Csv .\rules.csv -DryRun
      .\Create-DfwRules.ps1 -Csv .\rules.csv -Insecure
    
    .NOTES
      CSV columns (header required):
        policy,name,direction,action,protocol,from,to,service,appliedTo,disabled
      Optional per-policy columns (read from the first row of each policy group):
        category,priority,region
      Separate multiple members in one cell with ';'. Row order = evaluation order.
    #>
    
    [CmdletBinding()]
    param(
        [Parameter(Mandatory = $true)][string]$Csv,
        [string]$VcfaHost   = $env:VCFA_HOST,
        [string]$Tenant     = $env:VCFA_TENANT,
        [string]$ApiToken   = $env:VCFA_API_TOKEN,
        [string]$AccessToken = $env:VCFA_ACCESS_TOKEN,
        [string]$Category   = 'Environment',
        [int]$Priority      = 500000,
        [string]$Region     = 'region-west',
        [switch]$TcpStrict,
        [switch]$Insecure,
        [string]$CaCert,
        [switch]$DryRun
    )
    
    $ErrorActionPreference = 'Stop'
    $ApiGroup = 'vpc.nsx.vmware.com/v1alpha1'
    $IpRegex  = '^\d{1,3}(\.\d{1,3}){3}(/\d{1,2})?$'
    
    function Fail($msg) { Write-Error $msg; exit 1 }
    
    # ---- member / cell helpers ----
    function Split-Cell($cell) {
        if ([string]::IsNullOrWhiteSpace($cell)) { return @() }
        return ($cell -split ';' | ForEach-Object { $_.Trim() } | Where-Object { $_ })
    }
    function Convert-Members($cell) {
        $out = @()
        foreach ($v in (Split-Cell $cell)) {
            if ($v -match $IpRegex) { $out += [ordered]@{ ipAddress = $v } }
            else                    { $out += [ordered]@{ groupName = $v } }
        }
        return ,$out    # comma keeps it an array even with one element
    }
    function Convert-Services($cell) {
        $vals = Split-Cell $cell; if ($vals.Count -eq 0) { $vals = @('Any') }
        return ,@($vals | ForEach-Object { [ordered]@{ networkServiceName = $_ } })
    }
    function Convert-GroupNames($cell) {
        $vals = Split-Cell $cell; if ($vals.Count -eq 0) { $vals = @('Any') }
        return [ordered]@{ groupNames = @($vals) }
    }
    
    # ---- build one policy body from its rows ----
    function Build-Policy($name, $rows) {
        $head  = $rows[0]
        $rules = @()
        foreach ($r in $rows) {
            $rules += [ordered]@{
                name       = "$($r.name)".Trim()
                direction  = if ($r.direction) { "$($r.direction)".Trim() } else { 'InOut' }
                action     = if ($r.action)    { "$($r.action)".Trim() }    else { 'Allow' }
                ipProtocol = if ($r.protocol)  { "$($r.protocol)".Trim() }  else { 'IPV4' }
                from       = Convert-Members $r.from
                to         = Convert-Members $r.to
                appliedTo  = Convert-GroupNames $r.appliedTo
                services   = Convert-Services $r.service
                disabled   = ("$($r.disabled)".Trim().ToLower() -eq 'true')
            }
        }
        $cat  = if ($head.PSObject.Properties['category'] -and $head.category) { "$($head.category)".Trim() } else { $Category }
        $reg  = if ($head.PSObject.Properties['region']   -and $head.region)   { "$($head.region)".Trim() }   else { $Region }
        $prio = if ($head.PSObject.Properties['priority'] -and $head.priority) { [int]$head.priority }         else { $Priority }
    
        return [ordered]@{
            kind       = 'FirewallPolicy'
            apiVersion = $ApiGroup
            metadata   = [ordered]@{ name = $name }
            spec       = [ordered]@{
                appliedTo  = [ordered]@{ groupNames = @('Any') }
                category   = $cat
                regionName = $reg
                priority   = $prio
                stateful   = $true
                tcpStrict  = [bool]$TcpStrict
                locked     = $false
                rules      = $rules
            }
        }
    }
    
    # ---- HTTP: build common args for cert handling / version ----
    function Get-IrmArgs {
        $a = @{}
        if ($CaCert)        { $a['Certificate'] = $null }   # placeholder; CA bundle handled below
        if ($PSVersionTable.PSVersion.Major -ge 6 -and $Insecure) { $a['SkipCertificateCheck'] = $true }
        return $a
    }
    
    function Get-AccessToken {
        $uri  = "https://$VcfaHost/tm/oauth/tenant/$Tenant/token"
        $body = "grant_type=refresh_token&refresh_token=" + [uri]::EscapeDataString($ApiToken)
        $args = Get-IrmArgs
        try {
            $resp = Invoke-RestMethod -Method Post -Uri $uri -Body $body `
                -ContentType 'application/x-www-form-urlencoded' -Headers @{ Accept = 'application/json' } @args
        } catch {
            Fail "token exchange failed: $($_.Exception.Message)"
        }
        $tok = $resp.access_token; if (-not $tok) { $tok = $resp.token }
        if (-not $tok) { Fail "token response had no access_token/token field" }
        return $tok
    }
    
    function Get-PolicyUrl($name) {
        $base = "https://$VcfaHost/cci/kubernetes/apis/$ApiGroup/firewallpolicies"
        if ($name) { return "$base/$name" } else { return $base }
    }
    
    function Get-Policy($token, $name) {
        $args = Get-IrmArgs
        try {
            return Invoke-RestMethod -Method Get -Uri (Get-PolicyUrl $name) `
                -Headers @{ Authorization = "Bearer $token"; Accept = 'application/json' } @args
        } catch {
            $code = $null
            if ($_.Exception.Response) { $code = [int]$_.Exception.Response.StatusCode }
            if ($code -eq 404) { return $null }
            Fail "GET $name failed: $($_.Exception.Message)"
        }
    }
    
    function Set-Policy($token, $name, $body) {
        $existing = Get-Policy $token $name
        $headers  = @{ Authorization = "Bearer $token"; Accept = 'application/json' }
        $args     = Get-IrmArgs
        if ($null -eq $existing) {
            $json = $body | ConvertTo-Json -Depth 12
            Invoke-RestMethod -Method Post -Uri (Get-PolicyUrl) -Headers $headers `
                -ContentType 'application/json' -Body $json @args | Out-Null
            return 'created'
        } else {
            if ($existing.metadata.resourceVersion) {
                $body.metadata['resourceVersion'] = $existing.metadata.resourceVersion
            }
            $json = $body | ConvertTo-Json -Depth 12
            Invoke-RestMethod -Method Put -Uri (Get-PolicyUrl $name) -Headers $headers `
                -ContentType 'application/json' -Body $json @args | Out-Null
            return 'updated'
        }
    }
    
    function Get-Realized($policy) {
        if ($null -eq $policy) { return @($null, $null) }
        $state = ($policy.status.conditions | Where-Object { $_.type -eq 'Realized' } | Select-Object -First 1).status
        return @($state, $policy.status.ruleCount)
    }
    
    # ================= main =================
    
    if (-not (Test-Path $Csv)) { Fail "CSV not found: $Csv" }
    $rows = Import-Csv -Path $Csv
    if (-not $rows) { Fail "CSV has no data rows" }
    if (-not ($rows | Get-Member -Name policy -MemberType NoteProperty)) {
        Fail "CSV is missing the required 'policy' column"
    }
    
    # preserve first-seen policy order
    $order = @(); $seen = @{}
    foreach ($r in $rows) {
        $p = "$($r.policy)".Trim()
        if (-not $p) { Fail "a CSV row has an empty 'policy' value" }
        if (-not $seen.ContainsKey($p)) { $seen[$p] = $true; $order += $p }
    }
    
    $policies = [ordered]@{}
    foreach ($p in $order) {
        $prows = @($rows | Where-Object { "$($_.policy)".Trim() -eq $p })
        $policies[$p] = Build-Policy $p $prows
    }
    
    if ($DryRun) {
        foreach ($p in $order) {
            $b = $policies[$p]
            Write-Host "# ---- $p ($($b.spec.rules.Count) rules) ----"
            $b | ConvertTo-Json -Depth 12
        }
        $total = ($order | ForEach-Object { $policies[$_].spec.rules.Count } | Measure-Object -Sum).Sum
        Write-Host "`n$($order.Count) policy/policies parsed, $total rules total. (dry-run)"
        exit 0
    }
    
    if (-not $VcfaHost) { Fail "missing -VcfaHost (or VCFA_HOST env var)" }
    if (-not $AccessToken) {
        foreach ($n in @('Tenant','ApiToken')) {
            if (-not (Get-Variable -Name $n -ValueOnly)) {
                Fail "missing -$n (or its VCFA_* env var). Alternatively pass -AccessToken to skip the exchange."
            }
        }
    }
    
    # TLS handling for Windows PowerShell 5.1 (no -SkipCertificateCheck there)
    if ($PSVersionTable.PSVersion.Major -lt 6) {
        [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocol]::Tls12
        if ($Insecure) {
            Write-Host "WARNING: TLS verification disabled (-Insecure)"
            Add-Type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCerts {
        public static void Set() {
            ServicePointManager.ServerCertificateValidationCallback =
                delegate { return true; };
        }
    }
    "@ -ErrorAction SilentlyContinue
            [TrustAllCerts]::Set()
        }
    } elseif ($Insecure) {
        Write-Host "WARNING: TLS verification disabled (-Insecure)"
    }
    
    if ($AccessToken) {
        Write-Host "Using provided bearer token (skipping token exchange)."
        $token = $AccessToken
    } else {
        Write-Host "Exchanging API token for access token ..."
        $token = Get-AccessToken
    }
    
    $ok = $true
    foreach ($p in $order) {
        $body  = $policies[$p]
        $count = $body.spec.rules.Count
        Write-Host "`n> $p : $count rule(s)"
        $verb = Set-Policy $token $p $body
    
        $state = $null; $rc = $null
        for ($i = 0; $i -lt 6; $i++) {
            $res = Get-Realized (Get-Policy $token $p)
            $state = $res[0]; $rc = $res[1]
            if ($state -eq 'True') { break }
            Start-Sleep -Seconds 2
        }
        $flag = if ($state -eq 'True') { 'OK' } else { 'PENDING' }
        if ($state -ne 'True') { $ok = $false }
        Write-Host "  $verb; Realized=$state ruleCount=$rc  [$flag]"
    }
    
    Write-Host $(if ($ok) { "`nDone." } else { "`nDone with warnings (some policies not yet Realized)." })
    exit $(if ($ok) { 0 } else { 2 })
    
    run it
    $env:VCFA_HOST='auto-a.site-a.vcf.lab'
    .\Create-DfwRules.ps1 -Csv .\rules.csv -DryRun                              # preview
    .\Create-DfwRules.ps1 -Csv .\rules.csv -AccessToken 'eyJ...' -Insecure     # bearer token
    # or auto-exchange:
    $env:VCFA_TENANT='myorg'; $env:VCFA_API_TOKEN='...'
    .\Create-DfwRules.ps1 -Csv .\rules.csv -Insecure
    Pass the token without “Bearer”

    Give -AccessToken just the raw eyJ… value — the script adds the Bearer prefix itself. Paste Bearer eyJ… and you’ll double it up and get a 401.

    Where this earns its keep: VCD-to-VCFA firewall migration

    If you’re moving tenants off vCloud Director onto VCF Automation, the distributed firewall is one of the least glamorous and most error-prone parts of the cut-over — dozens or hundreds of per-tenant rules that have to land exactly right, because a missed allow is an outage and a missed deny is an exposure. Hand-recreating them in a new console doesn’t scale and doesn’t audit. This CSV-driven path turns it into a reviewable data problem: export the source DFW rules from VCD, normalize them into the same rules.csv shape (source, destination, service, action, direction), and let the script load them into VCFA — idempotently, so you can rerun it as many times as the migration-validation cycle needs. The CSV becomes the migration artifact itself: security and network teams review a spreadsheet instead of clicking through screens, it drops into source control as the record of what moved, and --dry-run gives you a diff before anything touches the target region. IP- and CIDR-based rules — the bulk of most VCD rule sets — map straight across as ipAddress members; group-based ones become NetworkSecurityGroup objects you create first, from the same kind of CSV. And because every policy reports Realized inline, you get an automated post-migration check: load the set, confirm every policy flipped to True, then cut traffic over. It won’t do the semantic mapping for you — VCD sections and applied-to scoping don’t line up one-to-one with VCFA’s category bands, and that judgement still needs an architect — but it removes the mechanical re-entry that makes firewall migration slow and risky, and leaves you reviewing intent in a CSV instead of transcribing rules by hand.

    The other two walls, in one place

    The first wall — the namespaced-path 404 — is covered above, in discovery. The two that bite later both throw a 400 whose message points away from the real cause:

    SymptomReal causeFix
    400 “cannot unmarshal array … type string”PowerShell double-wrapped an array ([["Any"]])No leading comma on hashtable-value assignments
    400 “Policy object path … does not exist”A rule references a group that isn’t created yetPOST the NetworkSecurityGroup first, same region
    The takeaway

    Once the token flow and the cluster-scoped path click, DFW on VCFA 9.1 is just declarative Kubernetes resources you can POST from anything — Postman, Python, PowerShell, or a full kubectl/Terraform pipeline when you’re ready. The scripts here take a spreadsheet a human can maintain and turn it into realized, enforced firewall policy, with the realized status checked on the way out. Start with --dry-run, verify one policy, then let it run the set.

    Lab-tested on VCF Automation 9.1 (CCI, vpc.nsx.vmware.com/v1alpha1). Verify resource schemas against your own build with a GET before batching — API surfaces shift between releases.

  • Memory Tiering Meets VKS: Denser Kubernetes Without Touching a Single Manifest

    Memory Tiering Meets VKS: Denser Kubernetes Without Touching a Single Manifest

    Every so often a platform capability arrives that your workloads benefit from without a single line of configuration. Memory Tiering under vSphere Kubernetes Service (VKS) is exactly that: enable it once at the cluster level, and every Kubernetes node — and every pod on it — inherits smarter memory economics automatically. No drivers, no manifest changes, no separate policy for containers. Broadcom has now published testing that confirms it works for VKS just as well as it does for VMs. Here is what the feature actually does, why the Kubernetes inheritance is the interesting part, and how to put it to work.

    TL;DR for the busy architect:
    • Memory Tiering lets an ESXi host combine DRAM with an NVMe-backed second tier into one memory pool — hot pages stay in DRAM, cold pages move to NVMe, and pages promote back automatically when touched.
    • Because VKS worker nodes are VMs on that same hypervisor, Kubernetes inherits the capability transparently — nothing above the hypervisor needs to know it exists.
    • The payoff: more worker nodes per host, a smaller DRAM footprint for your Kubernetes estate, and one capacity model that covers VMs and containers together.
    • Broadcom’s validation shows VKS workloads on tiered hosts performing in line with VM workloads — so the sizing guidance you already use for VMs extends to Kubernetes with confidence.
    • It is a workload-class decision, not a global switch — the “where it fits” section below matters as much as the feature.

    The quiet memory problem every Kubernetes platform team knows

    Kubernetes schedules pods on requests, and requests are set defensively. The app team asked for 4 GiB “to be safe,” the Helm default was generous, and nobody wants to own the OOMKill. Multiply that across hundreds of pods and every production cluster carries a wide gap between the memory workloads reserve and the memory they actually touch. Platform teams fight this with rightsizing dashboards and VPA recommendations — worthy work that never quite ends, because it treats the symptom: the reservation. The reserved-but-cold memory itself still sits in expensive DRAM.

    What if the infrastructure could simply handle cold memory differently from hot memory — without Kubernetes, the node OS, or the application having to participate?

    What Memory Tiering actually does

    Memory Tiering in VCF 9 lets an ESXi host present a single memory pool built from two tiers: fast DRAM and an NVMe device configured as a memory extension. The hypervisor watches page access patterns and manages placement automatically:

    • Hot pages stay in DRAM. Actively-touched memory lives where latency matters.
    • Cool pages move to the NVMe tier. Reserved-but-idle memory stops occupying your most expensive resource.
    • Touched pages promote back. If a cooled page becomes active again, it returns to DRAM — the process is continuous and automatic.

    The crucial design point is where this happens: below the guest OS, below the Kubernetes node VM, below the container runtime. The workload sees one contiguous memory space. There is no driver to install, no kernel parameter to tune, and no application change required. And the economics follow naturally — enterprise NVMe capacity costs a fraction of DRAM per gigabyte, and NVMe bays are plentiful where DIMM slots are finite.

    The VKS part: inheritance for free

    Here is the architectural elegance that makes this a Kubernetes story. A VKS worker node is a VM running on the same vSphere infrastructure as everything else. So when you enable Memory Tiering on a VCF cluster, your Kubernetes nodes inherit it the moment they land there — exactly as your VMs do. The Kubernetes scheduler keeps making its decisions against each node’s advertised capacity, exactly as before. ESXi, underneath, manages page temperature. Kubernetes manages reservations; the hypervisor manages reality. Each layer does the job it is best at, and neither needs to know about the other.

    Contrast that with what this would take anywhere else: per-node OS tuning, kernel-version dependencies, careful coordination with the container runtime, and a Kubernetes-specific memory policy to maintain forever. Under VKS it is one setting, applied at the infrastructure layer, covering your entire fleet — VMs and clusters alike.

    The fair question has always been whether the container layer changes the story — Kubernetes workloads are dense, churny, and allocation-happy. Broadcom put exactly that question to the test in their July 2026 post Memory Tiering for VMs and VKS, running database benchmarks on tiered hosts with half the DRAM of the baseline. The finding that matters: VKS workloads performed in line with the equivalent VM workloads, staying close to the full-DRAM baseline within the published sizing guidance. The Kubernetes layer neither breaks nor degrades the feature. Their numbers and test parameters are in the post, with deeper sizing detail in the Memory Tiering performance whitepaper.

    The picture

    K8s node on fixed memory VKS worker node (VM on tiered host) All memory in DRAM, hot or cold Hot working set actually touched Requested but cold occupying premium DRAM anyway Only lever: chase rightsizing forever DRAM tier — hot pages Hot working sets kept in DRAM by ESXi NVMe tier — cold pages Cold / rarely-touched pages promoted back to DRAM on access Lever: infrastructure absorbs the slack same K8s, smarter physics

    How it helps: four practical wins

    1. More worker nodes per host, without buying DRAM. Memory — not CPU — is the binding constraint on most Kubernetes host density. Tiering raises the addressable memory per host using NVMe capacity, which means more (or larger) worker VMs on the same hardware. Your cold-page-rich container estate is precisely the workload profile this was built for: hundreds of defensively-sized pods generate exactly the reserved-but-idle memory that tiers gracefully.

    2. A smaller DRAM footprint for the same Kubernetes estate. Flip the same coin: instead of packing more onto big hosts, run the estate you have on hosts with less DRAM. That matters most where DIMM slots are the constraint — edge sites, dense chassis, or refresh cycles where high-capacity DIMMs dominate the bill of materials. Because VCF licensing is per-core, capacity gained through memory density adds nothing to the software line — a point worth one sentence in any business case, even a feature-focused one.

    3. One capacity model for VMs and containers. This is the quietly valuable one for platform teams. Broadcom’s validation means the sizing and tiering guidance you use for VM workloads extends to VKS workloads — one set of recommendations, one configuration, one capacity plan for a mixed fleet. No Kubernetes-specific memory policies to author, document, and maintain alongside your VM standards.

    4. Zero change to the things you least want to touch. No application code, no container images, no Kubernetes manifests, no node OS tuning. For a platform team, features that require app-team cooperation move at the speed of your slowest app team; features that live below the stack move at the speed of a maintenance window. This one lives below the stack.

    Where it fits — and where it doesn’t

    An honest feature guide names the boundaries. Memory Tiering is a workload-class decision:

    1. Hot working sets don’t tier. The benefit comes from cold pages. In-memory databases, Redis, Kafka page cache, JVM heaps under sustained load — workloads that touch most of what they allocate — keep their pages in DRAM, and tiering buys little on those nodes. Plan node pools accordingly rather than expecting uniform gains everywhere.
    2. Latency-critical pods deserve DRAM guarantees. A page served from NVMe is fast by storage standards and slow by memory standards. Keep tail-latency-sensitive services on untiered or conservatively tiered hosts, and steer them there with the taints and node selectors you already use — your existing Kubernetes scheduling machinery is exactly the right control surface.
    3. Respect the published guardrail. The official guidance keeps active memory comfortably within DRAM capacity — the working set, not the addressable total, is your design boundary. Start with conservative tier ratios, observe real page-temperature behavior, then extend.
    4. It complements rightsizing; it doesn’t replace it. Tiering does not change what the Kubernetes scheduler sees — which is the point — but that also means inflated requests still strand schedulable capacity. Keep the rightsizing hygiene; let tiering absorb what hygiene can’t reach.

    A practical adoption path

    Step Action Output
    1Baseline active vs. consumed memory across VKS clusters in VCF OperationsYour cold-memory opportunity, quantified
    2Classify node pools: tiering-friendly (stateless services, batch, CI, dev/test) vs. DRAM-guaranteed (data services, latency-critical)A node-pool map, enforced with taints and labels
    3Enable tiering on one tiering-friendly host group at a conservative ratioReal page-temperature data from your workloads
    4Validate application SLOs and node stability through a full business cycleGo/no-go evidence per workload class
    5Extend host group by host group, re-baselining as you goA fleet-wide rollout grounded in your own data

    The bottom line

    The strongest features are the ones your workloads don’t have to know about. Memory Tiering under VKS is a hypervisor capability that Kubernetes inherits simply by running where it runs — hot pages in DRAM, cold pages on NVMe, everything transparent above the hypervisor line. For platform teams it means denser hosts or leaner DRAM configurations, one capacity model across VMs and containers, and not a single manifest changed. VKS already removes the operational overhead of running Kubernetes beside your VM estate; Memory Tiering quietly removes a good part of the DRAM overhead of running it at scale.

    Start with the baseline in VCF Operations — the gap between active and consumed memory on your clusters is the feature’s opportunity in your environment, measured. Everything after that is a controlled rollout.

    Are you running Memory Tiering under VKS yet — or planning to? I would love to hear which workload classes you are tiering first, and what your active-vs-consumed gap looks like in practice. And if there is interest, a tiering-readiness calculator in the style of my other VCF tools might be the natural follow-up.

  • From Org VDC to Namespace: The Architect’s Field Guide to VCD → VCFA Migration

    From Org VDC to Namespace: The Architect’s Field Guide to VCD → VCFA Migration

    In July 2025 I wrote about navigating the shift from VMware Cloud Director to VCF Automation — what stays familiar, what changes, and why VCFA is not a rebranding exercise. That post answered the “what.” With VCF 9.1 now generally available and shipping an official VCD-to-VCFA Migration Tool, it is time to answer the harder question: how do you actually plan this migration?

    This guide is written for cloud service providers and enterprise architects running Cloud Director today. It covers the two migration paths, a full construct-by-construct mapping, the networking decisions that will make or break your timeline, and a readiness checklist you can take into your planning workshop.

    TL;DR for the busy architect:
    • VCFA 9.1 ships an official in-place Migration Tool for VCD — plan the migration from the official documentation, and treat this guide as the design layer on top of it.
    • This is a resource-model migration, not an upgrade: Org VDCs become Supervisor namespaces inside NSX VPCs, and that changes networking, security, and automation design.
    • Most providers will run a hybrid strategy — Migration Tool for standard tenants, side-by-side rebuild for complex ones, retirement for the long tail.
    • The three timeline killers are networking disposition (extend vs. re-IP), automation porting (especially legacy XML API callers), and billing continuity against VCF Operations.
    • Do the inventory and tenant classification now; both cost nothing and inform every downstream decision.
    Scope note: This post deliberately stays at the architecture and planning layer — construct mapping, network design, tenant strategy. For the Migration Tool itself, its step-by-step procedures, and version prerequisites, always work from the official VCFA 9.1 migration documentation. Think of this guide as the homework you do before opening the runbook, so you arrive with a design instead of a question mark.

    Why this is a migration, not an upgrade

    Let’s be honest about what Cloud Director got right, because it explains why this transition deserves respect. VCD gave providers a battle-tested multi-tenancy model, a mature API that an entire ecosystem built against, and constructs — Org VDCs, Edge Gateways, vApps, catalogs — that thousands of operators know by muscle memory. Fifteen years of production hardening is not something you wave away with a version number.

    VCF Automation keeps the best of that lineage — the provider-facing Tenant Manager is built on the VCD codebase, so the provider experience will feel familiar — but the underlying resource model is fundamentally different. VCFA delivers infrastructure through vSphere Supervisor namespaces and NSX VPCs, not through Org VDCs carved out of Provider VDCs. That means workloads, networking, and tenancy boundaries all land on new primitives. Your VMs move; your mental model migrates.

    The payoff for making that jump is real: a single platform that serves VMs, Kubernetes (VKS), and AI workloads to the same tenants under the same quota and governance model, with self-service networking (vDefend firewalling, Avi load balancing, tenant-managed NAT/VPN) that VCD needed extensions and bolt-ons to approximate. VCF 9.1’s tenancy improvements are explicitly aimed at providers running hundreds of isolated tenants on shared infrastructure.

    The two migration paths

    There are exactly two credible strategies, and most providers will end up using both across their estate.

    Path 1: In-place migration with the Migration Tool

    New in VCFA 9.1, the Migration Tool performs an in-place migration of VCD workloads and configuration into VCF Automation. The critical planning facts:

    • Your VCD environment must first be updated to the specific VCD version the tool requires — verify the current prerequisite in the official migration documentation before scheduling anything.
    • The target is a deployed and configured VCF 9.1 landing zone — VCFA does not exist as a standalone product, so the fleet (SDDC Manager, VCF Operations, NSX, Supervisor-enabled workload domains) must be in place first.
    • “In-place” refers to the workloads: the goal is migrating tenants and their VMs with minimal downtime, not a forklift.

    Path 2: Side-by-side build and move

    Stand up VCF 9.1 with VCFA greenfield, design your organizations and regions natively, and move workloads tenant by tenant. VM import into VCFA namespaces is a documented pattern — including VLAN-backed connectivity options (Distributed VLAN Connections attached to a tenant’s Transit Gateway) that let imported VMs keep their existing L2 networks during transition.

    Reality check: Side-by-side costs more hardware and more calendar time, but it converts migration risk into design freedom — you are not importing fifteen years of VCD configuration decisions into a brand-new platform. In-place is faster and preserves tenant continuity, but you inherit your own legacy. Neither answer is universally right; the decision framework below is where most workshops should start.

    Construct mapping: VCD to VCFA

    First, the picture. The left stack is what your operations team knows by muscle memory; the right stack is where each piece lands:

    VMware Cloud Director VCF Automation 9.1 Organization tenancy boundary Organization All-Apps or VM-Apps Provider VDC capacity pool Region mapped to vSphere Supervisors Org VDC allocation models Region quota + Namespaces VM classes · storage classes · VPC Edge Gateway (T1) provider-managed edge Transit Gateway / VPC GW tenant self-service NAT · VPN · FW Provider Gateway (T0) north-south uplink External Connection multiple per org in 9.1 IP Spaces public IP management External IP Blocks multi-CIDR · Infoblox IPAM Catalogs / vApp Templates tenant image library Content Libraries bound per namespace Container Service Extension bolt-on Kubernetes VKS (native) same quota & tenancy model

    Now the detail. This is the table to keep open during every design session. “Closest equivalent” does not mean “identical behavior” — read the notes.

    VMware Cloud Director VCF Automation 9.1 What actually changes
    Organization Organization (All-Apps or VM-Apps) Concept survives intact. New decision: All-Apps orgs (Supervisor-backed, VMs + containers) vs. VM-Apps orgs (VM-only). Choose per tenant based on their roadmap — this is hard to change later.
    Provider VDC Region (mapped to vSphere Supervisors) Regions map VCFA to Supervisors. 9.1 adds multi-Supervisor region quotas, so one tenant’s quota can span Supervisors within a region.
    Org VDC Region quota + namespaces The biggest conceptual shift. Capacity is granted as quota against a region; tenants consume it by creating namespaces (Supervisor namespaces with storage classes, VM classes, and an NSX VPC).
    Org VDC network VPC subnet Routed/isolated Org VDC networks become subnets inside the tenant’s NSX VPC. NAT’d topologies map naturally; direct-connect topologies need the VLAN extension patterns.
    Edge Gateway (T1) Transit Gateway / VPC gateway Tenants can create multiple NSX Transit Gateways and self-manage NAT, IPsec VPN, and vDefend Gateway Firewall on them — genuinely more self-service than VCD Edge management.
    Provider Gateway (T0) External Connection Renamed and expanded: 9.1 supports multiple external connections per organization for split egress paths.
    IP Spaces External IP Blocks Renamed in 9.1, now with multiple CIDRs per block, include/exclude ranges, Infoblox external IPAM integration, and the option to hide block contents from consuming tenants.
    NSX DFW via VCD vDefend DFW + Gateway Firewall, delegated 9.1 brings vDefend firewalling natively into VCFA with delegation to org admins, out-of-the-box security profiles for VPCs, and RBAC-labeled dynamic security groups. Policy will be rebuilt, not translated — plan a rule rationalization exercise.
    Avi / NSX ALB integration Native self-service Avi Full self-service load balancing with provider-set quotas (service engine and application limits) for both All-Apps and VM-Apps orgs.
    Catalogs Content Libraries Templates and media move to vSphere content libraries surfaced through VCFA. Plan a template hygiene pass — do not migrate a decade of orphaned vApp templates.
    vApp No direct equivalent Multi-VM vApps with fencing, start-order, and lease semantics do not map 1:1. Grouping lands on namespaces and deployments; orchestration behavior must be re-expressed in templates and automation. Validate per tenant during the classification workshop.
    Container Service Extension (CSE) VKS (native) A clean win: bolt-on Kubernetes becomes first-class vSphere Kubernetes Service under the same tenancy and quota model.
    Chargeback / VCD tenant metering VCF Operations Metering, capacity, and logs consolidate into VCF Operations. Validate your billing pipeline against the new data sources early — this is a common late surprise.

    Networking: where the timeline is won or lost

    Everything else in this migration is bookkeeping compared to networking. Three decisions dominate:

    1. VPC-native or VLAN-extended? The clean end state is tenants inside NSX VPCs. But brownfield tenants have VMs on VLAN-backed networks with hardcoded IPs, external firewall rules, and neighboring physical systems. VCFA supports Distributed VLAN Connections and VLAN extension subnets precisely for this — imported VMs keep their L2 while the tenant’s Transit Gateway provides the routed path. Decide per tenant which networks extend and which re-IP into VPC subnets, and record it in your migration runbook.

    For a VLAN-extended tenant, the provider-side sequence looks like this in practice:

    1. Deploy the VNA cluster in NSX. In the VCF 9.1 networking model, Virtual Network Appliances take over the role Edge Nodes played — the tenant’s Distributed Transit Gateway (DTGW) needs them in place before any VLAN-backed connectivity works.
    2. Create a shared Distributed VLAN Connection (VLAN-backed) that the DTGW uses as its uplink path. This is provider-scoped plumbing, created once.
    3. Create a dedicated Distributed VLAN Connection per legacy subnet, using the exact VLAN ID of the existing portgroup the source VMs sit on. This is what preserves L2 adjacency for the imported workloads.
    4. Configure the tenant’s region network connectivity to use the Distributed VLAN Connection option, attaching the DTGW to the shared connection, then assign the dedicated per-subnet connection to the organization (Organization > Networking > Connections).
    5. Import the VMs into the tenant’s namespace. The VMs keep their IPs and VLAN; north-south routing and firewalling now flow through the tenant’s self-managed Transit Gateway.
    Design consequence: every VLAN you extend is technical debt you are choosing to carry into the new platform. That is often the right call — re-IP on the tenant’s schedule, not yours — but cap it. Set a policy up front: VLAN extension is a transition state with an owner and an expiry, not a destination.

    2. IP addressing continuity. Map every VCD IP Space to an External IP Block before moving anything. The 9.1 additions (multiple CIDRs, exclusion ranges, Infoblox integration) make it possible to mirror your existing allocations exactly — use that to keep tenant SNAT/VIP addresses stable so their downstream firewall rules and DNS survive the move.

    3. Security policy rebuild. Treat the vDefend transition as an opportunity, not a chore. Export and rationalize existing DFW rule sets, then rebuild them as delegated vDefend policy with dynamic groups. If you are automating this (and at provider scale, you should be), the VCFA cloud template path with Cloud.SecurityGroup resources lets you express tenant micro-segmentation as code from day one — a topic I will cover in depth in a follow-up post.

    Under the hood: what a tenant actually consumes now

    In VCD, a tenant consumed an Org VDC — an opaque slice of a Provider VDC with allocation-model semantics that only VCD understood. In VCFA, the consumption unit is a Supervisor namespace, and it is worth being precise about what that bundle contains, because it is what your migrated VMs land inside:

    • VM classes — t-shirt-sized CPU/memory shapes the provider curates and the tenant selects from (the successor to VCD sizing policies).
    • Storage classes — vSphere storage policies (vSAN ESA policies included) surfaced as Kubernetes storage classes; this is how storage tiering reaches the tenant.
    • An NSX VPC — the tenant’s private routing domain, with subnets, SNAT, and the Transit Gateway attachment.
    • Content library bindings — the curated image catalog (your old VCD catalogs, post-hygiene).
    • Quota enforcement — counted against the organization’s region quota, which in 9.1 can span multiple Supervisors.

    The practical consequence: a “VM” in VCFA is a declarative object served by the VM Service, not a row in the VCD database. The same VM a tenant deploys through the VCFA catalog can be expressed — and lifecycle-managed — as YAML against the namespace:

    apiVersion: vmoperator.vmware.com/v1alpha3
    kind: VirtualMachine
    metadata:
      name: app-vm-01
      namespace: tenant-a-prod        # the Supervisor namespace
    spec:
      className: guaranteed-medium     # provider-curated VM class
      imageName: ubuntu-22.04-lts      # from the bound content library
      storageClass: vsan-esa-gold      # vSphere storage policy
      network:
        interfaces:
          - name: eth0                 # lands on a VPC subnet
    

    Why this matters for migration planning: VMs, VKS clusters, and (in All-Apps orgs) AI workloads are all served through the same namespace primitive. When you classify a tenant as All-Apps, you are not enabling a feature flag — you are placing them on a consumption plane where their VCD-era VMs and their future Kubernetes estate share quota, networking, and governance. That is the architectural payoff that justifies the migration effort, and it is why tenant classification (checklist item 3) deserves a real workshop rather than a default.

    What your tenants will notice

    Your migration communication plan should be built around this table — it is the difference between tenants experiencing an upgrade and tenants filing tickets:

    Tenant task In VCD In VCFA 9.1 Tenant impact
    Deploy a VMvApp wizard from catalog templateCatalog request or cloud template deployment into a namespaceRetraining needed — vApp semantics (fencing, start order, leases) are gone; deployments are the new grouping unit
    Create a networkOrg VDC network via Edge Gateway (often provider-assisted)Self-service VPC subnet creationGenuine upgrade — less waiting on the provider
    Firewall a workloadEdge/DFW rules through VCD proxy, provider guardrails informalDelegated vDefend policy with dynamic groups, provider guardrails enforcedUpgrade, but rules look different — walk tenants through the rebuild
    Publish a serviceEdge NAT + manual LB or bolt-on ALBSelf-service NAT on Transit Gateway + native Avi LB within quotaUpgrade — full self-service within provider limits
    Get KubernetesCSE (if the provider deployed it)VKS cluster from the same portal, same quotaMajor upgrade and the headline of your tenant comms
    Automate deploymentsVCD API scripting, task pollingDeclarative YAML / cloud templates / Supervisor APITheir scripts break; give them a porting guide and examples

    The API transition: where your automation goes

    If you run a service provider practice, your VCD automation — onboarding pipelines, billing extracts, tenant portals — is the part of this migration nobody budgets properly. The mapping:

    Automation surface (VCD) VCFA 9.1 equivalent Porting effort
    Legacy API (/api, XML) None — end of the line Anything still on the XML API gets rewritten, full stop. If you have not inventoried these callers, start there.
    CloudAPI (/cloudapi/1.0.0, JSON) Tenant Manager APIs (VCD-derived) Low-to-moderate for provider-side operations — Tenant Manager inherits the VCD codebase, so org/quota/network provisioning calls will feel structurally familiar. Validate endpoint-by-endpoint; familiar is not identical.
    Tenant-side consumption automation Supervisor Kubernetes APIs + VCFA cloud templates Moderate — but a genuine upgrade. Declarative YAML against the namespace replaces imperative task-polling against VCD, and cloud templates give tenants versionable, reviewable infrastructure definitions.
    Blocking tasks / extensibility framework Blocking tasks on Tenant Manager workflows + VCFA extensibility Better than feared: blocking tasks carry over conceptually in VCFA, tied to provider (Tenant Manager) workflows — your CMDB and metadata-tagging hooks have a landing spot. Tenant-facing UI plugins and custom extensions still need rebuilding; map each to carry-over, rebuild, or retire (checklist item 8).
    Terraform provider for VCD Terraform/IaC against VCFA + Supervisor Moderate — state files do not transfer between providers; plan a re-import or parallel-manage strategy per tenant.

    One concrete example of the tenant-side upgrade. In VCD, giving a tenant firewall self-service meant DFW sections scoped through the VCD proxy and a lot of provider hand-holding. In VCFA, security intent lives directly in the cloud template alongside the workload it protects:

    resources:
      web_sg:
        type: Cloud.SecurityGroup
        properties:
          name: web-tier-sg
          securityGroupType: new       # provisions the group at deploy time
      web_vm:
        type: Cloud.vSphere.Machine
        properties:
          image: ubuntu-22.04-lts
          flavor: guaranteed-medium
          networks:
            - network: '${resource.vpc_net.id}'
              securityGroups:
                - '${resource.web_sg.id}'   # micro-seg attached as code
    

    The deployment provisions the security group, attaches the workload, and — with 9.1’s delegated vDefend model — keeps the whole thing inside guardrails the provider defines once. Tenant micro-segmentation becomes a code review, not a ticket queue. This deserves its own deep dive, and it will get one in the follow-up post.

    Migration readiness checklist

    # Item Why it gates the migration
    1VCF 9.1 landing zone deployed (fleet, NSX, Supervisor-enabled domains)VCFA has no standalone mode; the platform must exist before tenants can land. Size for VCFA’s appliance footprint (24 vCPU) plus VCF Management Services.
    2VCD inventory and version auditThe Migration Tool requires a specific minimum VCD version; know your upgrade distance. Inventory orgs, VDCs, edges, IP allocations, catalogs, and extensions now.
    3Tenant classification: All-Apps vs. VM-AppsOrg type shapes everything downstream — Supervisor placement, quotas, network design. Survey tenant container/AI roadmaps before deciding.
    4Region and Supervisor designRegions replace Provider VDCs as your capacity boundaries. Design them for the target state, including multi-Supervisor quota where tenants need headroom.
    5External IP Block plan mirroring current IP SpacesIP continuity is the difference between a maintenance window and a tenant escalation.
    6Per-tenant network disposition (extend vs. re-IP)Drives the migration order and the amount of VLAN-extension plumbing (VNA clusters, distributed connections) you deploy.
    7DFW rule export and rationalizationRules rebuild as vDefend policy; a rationalized rule set migrates in days, an unrationalized one in months.
    8Extension and ecosystem dependency mapVCD plugins, blocking-task integrations, and portal customizations have no automatic path. Identify each and decide: replace, rebuild via VCFA APIs, or retire.
    8aAPI caller inventory (legacy XML API vs. CloudAPI)Legacy XML API callers must be rewritten; CloudAPI callers port with moderate effort to Tenant Manager APIs. You cannot scope the automation workstream without this split.
    9Billing/metering validation against VCF OperationsRevenue-affecting and always discovered too late. Run parallel metering during pilot tenants.
    10Pilot tenant selectionPick one friendly, low-complexity tenant per network pattern (VPC-native, VLAN-extended, VPN-heavy) and migrate them first.

    Decision framework: which path for which tenant

    Tenant profile Recommended path Rationale
    Standard IaaS tenant, NAT’d networks, no exotic extensionsIn-place (Migration Tool)The tool exists for exactly this profile; minimal downtime, configuration carried over.
    Tenant with heavy VCD API/extension integrationSide-by-sideTheir tooling must be rebuilt against VCFA APIs regardless; a clean landing zone avoids migrating assumptions that no longer hold.
    Tenant asking for Kubernetes/AI servicesSide-by-side into an All-Apps orgDesign the namespace, VKS, and GPU consumption model natively rather than retrofitting after import.
    Tenant with direct-connect/VLAN-backed networks and hardcoded IPsIn-place with VLAN extensionDistributed VLAN Connections preserve L2 continuity; re-IP later on the tenant’s schedule, not yours.
    Dormant or minimal tenantsNeither — consolidate or retireEvery migration is a chance to shrink the estate. Do not pay migration cost for tenants who should be offboarded.
    Bottom line: Treat this as a portfolio decision, not a platform decision. Most providers will run the Migration Tool for the long tail of standard tenants, hand-craft side-by-side moves for their strategic and complex tenants, and quietly retire the rest. The VCD-familiar Tenant Manager means your operations team’s experience transfers; the new resource model means your designs should not.

    Putting it on a calendar: the phased wave plan

    Durations below assume a mid-size provider (50–200 tenants) with a dedicated migration team. Scale accordingly — but keep the phase boundaries, because each phase gates the next:

    Phase Typical duration Key activities Exit criteria
    0 — Foundation4–8 weeksDeploy VCF 9.1 landing zone; region/Supervisor design; External IP Block plan; VNA/DTGW plumbing for VLAN extension; stage the prerequisite VCD update and work through the official Migration Tool documentationLanding zone health-checked; IP plan signed off; migration runbook drafted
    1 — Pilot3–4 weeksMigrate one friendly tenant per network pattern (VPC-native, VLAN-extended, VPN-heavy); run parallel metering; draft tenant comms from real findingsAll three patterns validated end-to-end; billing extract matches VCD baseline
    2 — Standard waves2–6 monthsMigration Tool waves for standard tenants, grouped by network pattern and maintenance windows; automation porting runs in parallel~80% of tenant count migrated; VCD write-freeze for migrated orgs
    3 — Complex rebuilds1–3 months (overlaps Phase 2)Side-by-side moves for extension-heavy and strategic tenants; All-Apps onboarding for Kubernetes/AI adopters; blocking-task logic rebuiltNo tenant left with a “we’ll figure it out later” flag
    4 — Decommission4–6 weeksVCD read-only period; final billing reconciliation; certificate/DNS cleanup; capacity reclaimed into VCFVCD powered off — and nobody noticed

    Five ways this migration goes wrong

    Patterns worth naming, because every one of them is avoidable and every one of them happens:

    1. Migrating the mess. Fifteen years of VCD accumulates orphaned vApp templates, dead orgs, and firewall rules nobody can explain. Running the Migration Tool over an uncleaned estate turns technical debt into migrated technical debt on a platform where it is now unfamiliar debt. Hygiene first.
    2. Classifying every tenant as All-Apps “to be safe.” Org type drives Supervisor placement and consumption design. Defaulting everyone to All-Apps burdens VM-only tenants with a model they will not use and complicates your capacity math. Classify on evidence, not hedging.
    3. Treating VLAN extension as the destination. It is a bridge. Providers who extend everything “temporarily” find themselves running a permanent hybrid L2 estate with double the operational surface. Expiry dates, owners, tracked in the same register as the migration itself.
    4. Discovering the XML API callers in Phase 2. The billing script someone wrote in 2016 against the legacy API will fail loudly mid-wave. The API caller inventory (checklist 8a) is boring, which is why it gets skipped, which is why it becomes the outage.
    5. Underinvesting in tenant communication. The tenant experience genuinely improves — but only tenants who were told what changed will experience it as improvement. Budget real effort for the porting guide, the vApp-to-deployment explanation, and the “your new self-service powers” message. Your NPS through this migration is a comms outcome, not a technical one.

    Frequently asked questions

    Does this mean Cloud Director is end-of-life?
    Check the official Broadcom product lifecycle matrix for authoritative dates rather than blog speculation (including mine). What is unambiguous from the 9.1 release is strategic direction: the Migration Tool exists, the VCSP tenancy investment is landing in VCFA, and the Tenant Manager carries the VCD codebase forward inside VCF. Plan on the trajectory, not the deadline.

    Can I run VCD and VCFA side by side during migration?
    Yes — and you will, for months. That is precisely why the phased plan includes parallel metering and a VCD write-freeze discipline for migrated orgs. The risk is not coexistence; it is coexistence without an exit date.

    Do my tenants’ IP addresses survive?
    They can, if you do the work: mirror IP Spaces into External IP Blocks before moving anything, and use VLAN extension where re-IP is not yet feasible. IP continuity is a design outcome, not a tool feature.

    What happens to my vApps?
    The VMs inside them migrate; the vApp construct itself (fencing, start order, leases) does not have a 1:1 successor in VCFA. Grouping lands on namespaces and deployments, and orchestration behavior is re-expressed in cloud templates. Audit your production vApp patterns during the inventory phase and validate each against the official migration documentation — this is the single biggest tenant-facing conceptual change, so put it at the top of your comms plan.

    How does object-level access control change?
    Significantly, and it needs design work before migration. VCD scoped object access at the vApp and catalog level; VCFA scopes it at the Project level — a new construct where org admins group users and namespaces, and the same user can hold different roles in different projects. Translate your existing vApp-sharing and catalog-permission patterns into a project design per tenant as part of the classification workshop, so access lands correctly on day one instead of being retrofitted after tickets.

    What to do next

    1. Run the inventory and classification exercises (checklist items 2 and 3) — they cost nothing and inform everything.
    2. Work through the official VCFA 9.1 migration documentation and stage the prerequisite VCD update, with your inventory in hand.
    3. Stand up the VCF 9.1 landing zone and pilot one tenant per network pattern before committing to a wave plan.

    In the next post, I will go deeper on the security layer of this story: rebuilding tenant micro-segmentation as code with VCFA cloud templates and Cloud.SecurityGroup — turning the DFW rationalization exercise from this guide into delegated, self-service lateral security.

    Have you started planning a VCD-to-VCFA migration? I would love to hear which construct mappings are causing you the most friction — drop a comment below.

  • VCF 9.1 Makes VKS Harder to Ignore

    VKS VKS VCF 9.1

    VKS on VCF 9.1 What Actually Changed & Why It Matters

    A Comic Book Story in Seven Chapters

    Issue #01 · May 2026 · The VCF 9.1 Saga

    ⚡ Cast of Characters ⚡

    Captain VKS
    vSphere Kubernetes Service 3.6
    The hero. Born from vSphere, forged in CNCF conformance. Now powered up with VCF 9.1 abilities.
    The Architect
    Platform Engineer
    Our protagonist. Runs multi-domain VCF estates. Needs Kubernetes at enterprise scale without the circus.
    Cluster Creep
    The Villain of Sprawl
    Feeds on operational toil, slow provisioning, and fragmented toolchains. Grows stronger with every manual step.
    FINOPS
    The Oracle
    VCF Operations 9.1
    Sees all. Knows cost. Tracks every namespace. Speaks in metrics and FinOps.
    🥊 The Challengers 🥊
    $
    The Cloud Twins
    The Hyperscaler Duo
    They move fast and always whisper: “Just move to our cloud.” They charge per hour and never let go.
    The Red Baron
    The Opinionated Platform
    Arrives in full armor. Brings his own runtime, registry, mesh, and opinions about everything. Enterprise prices included.
    The Wrangler
    The Multi-Cluster Cowboy
    Rides across any ranch — any cloud, any edge, any distro. Freedom is his creed. But who’s managing the cattle?
    $ kubeadm
    Bare Knuckle
    The DIY Brawler
    No platform. No hand-holding. Bare metal, kubeadm, and grit. Cheap up front. Costs you in blood and 3 AM pages.
    Chapter 01The 37-Minute Nightmare
    37 MIN !! ?!
    The data center. 6:42 AM. The Architect stares at a provisioning timer that refuses to move. Cluster Creep watches from the shadows, feeding on frustration.
    The Architect 37 minutes to spin up a dev cluster. Thirty. Seven. Minutes. The hyperscaler team next door gets theirs in ten. The CTO is asking questions.
    Cluster Creep Yesss… and that’s just the deployment. Wait until you see the upgrade windows. I’ve got 45 minutes of downtime planned for each cluster. You have 200 clusters. Do the math. 😈
    That’s 150 hours of maintenance windows per upgrade cycle… across the fleet…
    VCF 9.1 DROPS MAY 5, 2026
    May 5, 2026. Broadcom releases VCF 9.1. And everything changes.
    Captain VKS Miss me? I brought Fast Deploy. Let me show you the new numbers.
    MetricVCF 9.0VCF 9.1
    Cluster Deploy Time37 min11 min (↓69%)
    Cluster Upgrade Time45 min15 min (↓67%)
    Max Clusters / Supervisor~100500
    Node Pool PlacementManualDRS Intelligent
    Chapter 02The Challengers Step Forward
    V S
    Word of VCF 9.1 spreads. Four challengers emerge from the fog, each claiming the throne of enterprise Kubernetes. The Architect has heard their pitches before.
    The Cloud Twins Adorable upgrade, Captain. But we’ve been doing sub-10-minute clusters for years. Managed control plane. Global regions. Auto-scaling node groups. Why fight gravity? Just come to the cloud.
    Captain VKS Sure — and your managed control plane costs how much per cluster per month? Multiply that by 500 clusters. Now add the egress fees. Now add the data sovereignty audit your CISO just mandated. I run on hardware you already own.
    The Red Baron How charming. You finally got CNI choice? I ship with my own SDN, my own service mesh, my own registry, my own CI/CD pipelines, and a full developer portal. I am the platform. You’re still assembling one.
    Captain VKS You are the platform. That’s the problem. Your opinions become my constraints. Your lifecycle becomes my upgrade treadmill. Your per-core subscription becomes my CFO’s nightmare. I give choice. You give mandates.
    The Wrangler Y’all are so cute with your single-vendor stacks. I run on any infrastructure. True multi-cluster freedom. No lock-in. Ever.
    Captain VKS Freedom is great until your team is maintaining six different infrastructure backends. I give you 500 clusters on one Supervisor with one operational model. You give them options and a prayer.
    Bare Knuckle I don’t need a platform. kubeadm, a Makefile, and raw skill. Zero licensing. Zero overhead. Pure Kubernetes.
    Captain VKS I respect the craft. But who patches your nodes at 2 AM? Who handles etcd backups? Who runs certificate rotation? Your “zero cost” platform costs three full-time engineers.
    The Architect I’ve evaluated all of you. Here’s my problem: I already run VCF. My VMs, NSX networking, vSAN storage, and security policies are all here. I need Kubernetes that joins my platform — not one that replaces it or ignores it.
    The best Kubernetes platform is the one that doesn’t make me build a second operations team…
    Chapter 03Fast Deploy — 11 Minutes or Bust
    Captain VKS explains what changed under the hood. Fast Deploy isn’t a marketing stunt — it’s an architectural rework of the provisioning pipeline.
    Captain VKS Here’s what actually happened. We parallelized the node bootstrapping sequence, pre-staged container images into a local content library, and eliminated redundant API round-trips during cluster init. 11 minutes, from API call to workload-ready.
    The Architect What about upgrades? That’s where we bleed. Every cluster upgrade is a maintenance window, and my team juggles 200+ clusters.
    Captain VKS 45 minutes down to 15. Pre-staged images, parallel node drain-and-replace, and Multiple Clusters per Zone means you keep workloads running on Zone A while upgrading Zone B.
    ⚡ IMPACT METER ⚡
    Provisioning Speed Gain
    69%
    Upgrade Speed Gain
    67%
    Scale Ceiling Increase
    5× (500 clusters)
    The Cloud Twins 11 minutes… fine, that’s competitive. But can you match our global availability zones?
    Captain VKS I don’t need 60 regions. My Architect’s data stays in his sovereign data center, on his hardware, under his compliance umbrella. Your 60 regions are 60 places his CISO has to audit.
    Chapter 04DRS Strikes Back — Intelligent Node Pool Placement
    GPU HOST AI ML NVMe HOST DB CACHE COMPUTE HOST WEB API DRS SCHEDULER
    VCF 9.1 introduces Intelligent Node Pool Placement. This isn’t basic affinity rules — it’s DRS-level scheduling applied to Kubernetes node pools.
    Captain VKS GPU pods → GPU hosts. NVMe workloads → NVMe nodes. DRS algorithm decides placement — not your YAML-wrestling platform team.
    The Red Baron I have Topology Manager, NUMA-aware scheduling, and a full operator ecosystem. Infrastructure-aware placement is table stakes for me.
    Captain VKS You schedule within the cluster. I schedule the cluster itself. DRS sees the whole estate. Your scheduler sees one namespace.
    The Oracle With VKS Cost Showback in VCF Operations 9.1, I can tell you exactly what each namespace, each cluster, each team is costing you. FinOps FOCUS-compliant.
    The Oracle I also expose an API for your RAG pipelines and MCP frameworks — your AIOps engine can query cost data directly.
    Per-NS
    Cost Attribution
    FOCUS
    FinOps Compliant
    Real-Time
    Pricing Estimates
    Show + Charge
    Back Capability
    Chapter 05Container-as-a-Service & The CNI Revolution
    CNI-A CNI-B CNI-C VKS
    VCF 9.1 introduces a simplified Container Service — deploy containers without deep Kubernetes expertise. Meanwhile, VKS 3.6 opens up CNI choice for the first time.
    Captain VKS First: Container-as-a-Service. Your app teams get a self-service surface. Click, deploy, done. No Supervisor clusters or ClusterClass YAML.
    Captain VKS Second: CNI freedom. VKS 3.6 deprecated ClusterBootstrap. Pick your CNI through the Addon Framework using AddonConfig CRDs. Antrea default, but the door is open.
    The Wrangler Oh, you’re just now letting people choose their CNI? Welcome to 2022, Captain.
    Captain VKS You let them choose. I let them choose with validated blueprints, lifecycle support, and a single vendor to call at 3 AM. Choice without support is just risk with extra steps.
    The Architect And the Ingress story? The popular open-source Ingress controller is being retired…
    Captain VKS Avi Load Balancer — natively integrated. Centralized control plane, distributed data plane, full observability. Plus vDefend gives you zero-trust lateral security for every pod.
    Chapter 06The Arena — Where Platforms Are Measured
    🛡️ ☁️ 🎩 🤠 🥊
    The Architect pulls up the scoreboard. No hype. No marketing. Just the dimensions that matter when you’re running Kubernetes in a regulated enterprise with 500+ VMs already on VCF.
    ⚔️ HEAD TO HEAD ⚔️
    Dimension 🛡️ Captain VKS ☁️ Cloud Twins 🎩 Red Baron 🤠 Wrangler 🥊 Bare Knuckle
    Data SovereigntyYour DCTheir DCYour DCDependsYour DC
    VM + K8s Unified OpsNativeSeparateSeparateSeparateSeparate
    Infra-Aware SchedulingDRS-LevelNode GroupsTopology MgrManualDIY
    Cluster Scale Ceiling500 / SupervisorUnlimited*Per InfraPer InfraPer Team
    Integrated FinOpsFOCUS NativeCost Explorer3rd Party3rd PartySpreadsheet
    Network SecurityvDefend + NSXVPC / SGBuilt-in SDNBYOBYO
    Licensing ModelPer-Core VCFPer-Cluster/HrPer-Core SubOpen SourceFree
    Day 2 ToilLowLowMediumMediumHigh
    AI / GPU ConformanceCNCF AI CertGPU PoolsOperatorsBYOBYO
    The Cloud TwinsWe still win on global reach and elastic scale.
    The Red BaronAnd I still own the developer experience story. Integrated CI/CD, GitOps, developer portal — out of the box.
    Captain VKS Fair. I’m not claiming I win everywhere. But for organizations already running VCF — I’m the only Kubernetes that doesn’t create a second operational island. VMs and containers. One platform. One team. One pane.
    The Architect That’s the point everyone misses. I don’t need the “best” Kubernetes in a vacuum. I need the best Kubernetes for my stack. And my stack is VCF.
    Chapter 07The Numbers Don’t Lie
    💥 THE FINAL SHOWDOWN 💥
    Broadcom surveyed 44 VCF 9 customers in March 2026. Here’s what they found — and why the challengers are looking over their shoulders.
    51%
    Less Infra Mgmt Time
    46%
    Less Monitoring Time
    47%
    Less Capacity Needed
    39%
    Faster MTTR/MTTI
    Cluster Creep No… NO! My sprawl… my complexity… my beautiful 37-minute deploy times… NOOOOO!
    ⚡ DEFEATED ⚡
    The challengers watch from the sidelines. They’re not defeated — but they know the game just changed.
    The Cloud TwinsWe’ll be back. Hybrid is where we’re heading too. See you at the edge…
    The Red BaronImpressive numbers. But developer experience is the next battlefield. Don’t get comfortable.
    The WranglerNot every ranch runs on one brand of fence. I’ll see you at the multi-cloud rodeo.
    Bare KnuckleSome of us still prefer the raw fight. But… 11 minutes is hard to argue with.
    The Architect VCF 9.1 gives me 11-minute deploys, 15-minute upgrades, 500 clusters per Supervisor, intelligent DRS-based node placement, native FinOps cost tracking, self-service CaaS, open CNI choice, native Avi ingress, and zero-trust pod security. All on the same VCF stack I’m already running.
    Captain VKS And I’m CNCF Kubernetes AI Conformant. The challengers are strong — I respect each of them. But none of them can do what I do: run Kubernetes as a native citizen of your existing VMware estate.
    VCF 9.1 doesn’t just iterate on VKS — it redefines the operational ceiling. Fast Deploy eliminates the provisioning tax. DRS-based placement removes manual scheduling toil. FinOps cost showback closes the last visibility gap. And with 500 clusters per Supervisor, VKS is the platform-scale Kubernetes runtime that VCF architects have been waiting for.

    The challengers each bring real strengths — managed simplicity, opinionated platforms, multi-cloud freedom, zero-cost entry. This isn’t a story where the hero has no flaws. But for the Architect running a VCF estate with VMs, containers, and AI workloads under one roof — the calculus is clear.

    The question is no longer “can VKS compete?” — it’s “what’s your excuse for not running it?”
    📚 Sources & References
  • VCF 9.1 – Top 20 Highlights for Cloud Service Providers

    VCF 9.1 – Top 20 Highlights for Cloud Service Providers

    Multi-tenancy, self-service, networking isolation, storage economics, fleet operations — curated for the CSP lens.

    VCF 9.1 is arguably the most CSP-significant VCF release in recent memory. The networking story alone — edge-free distributed connectivity, VPC isolation policies, EVPN/VXLAN peering — rewrites the playbook for multi-tenant service delivery. But the improvements span every layer: storage economics, Kubernetes density, fleet operations, and cyber recovery. Here are the 20 features that matter most if you’re running — or planning to run — a VMware-powered cloud service.

    Multi-Tenancy & Self-Service
    01 / 20

    VCD → VCF Automation Migration Tool

    This is the feature VCD-based CSPs have been waiting for. VCF 9.1 introduces a native migration path from VMware Cloud Director to VCF Automation. VMs are imported from OrgVDC resource pools directly into vSphere Namespaces. Supervisors, Clusters, Regions, Projects, and Namespaces are auto-created and mapped to existing VCD constructs. Network boundaries of OrgVDC are migrated to NSX VPC — preserving tenant isolation through the transition.

    Why CSPs Care

    Unblocks the single biggest migration concern for VCD-based providers. Automated construct mapping dramatically reduces migration effort, tenant downtime, and the professional services cost of transitioning to the VCF Automation operating model.

    02 / 20

    Self-Service Namespace Creation with Guardrails

    Organization admins can now delegate vSphere Namespace creation to Project Admins on a self-service basis. The governance layer is granular: admins define which Regions, Namespace Classes, Connectivity Profiles, Subnets, Infrastructure Policies, VPCs, and Service Engine Groups are available to each project. Tenants consume within those boundaries without filing tickets.

    Why CSPs Care

    Every namespace creation ticket that disappears from a CSP’s queue is margin improvement. Self-service with admin-defined guardrails is the operational model CSPs need — tenant autonomy without infrastructure risk.

    03 / 20

    Upfront Pricing Estimates & Tenant Notifications

    Tenants now see real-time pricing estimates before deploying catalog items, VMs, and VKS clusters. Consumption reports, infrastructure alerts, and critical operation notifications are surfaced directly in the VCF Automation UI. Providers configure which alerts and reports are visible to tenants.

    Why CSPs Care

    Transparent showback/chargeback is fundamental to CSP economics. When tenants see cost before they click “deploy,” billing disputes drop, resource waste decreases, and self-service confidence goes up.

    04 / 20

    Project-Scoped Content Libraries

    A new form of content library scoped to specific projects within an organization. Admins can restrict VM image availability so that only the users and resources of a given project can access particular images. Canonical Ubuntu images are now available as validated, subscribed content — provider-controlled.

    Why CSPs Care

    Image governance per tenant project. CSPs curate approved OS images without cross-tenant leakage — essential for regulated tenants and for CSPs offering tiered service catalogs.

    Networking & Tenant Isolation
    05 / 20

    VPC Connectivity Policies — Community, Promiscuous, Isolated

    VCF 9.1 introduces connectivity policies that control inter-VPC communication within a tenant project — without firewall rules. Community: VPCs in the same community talk to each other. Promiscuous: talks to any VPC. Isolated: only communicates with promiscuous VPCs. These can be mixed within a project for precise segmentation.

    Why CSPs Care

    Multi-tier tenant networking (dev/staging/prod isolation, shared-services patterns) handled by policy rather than per-rule firewalls. Reduces CSP networking configuration overhead per tenant from hours to minutes.

    06 / 20

    Transit Gateway Advanced Connectivity

    CTGW is now decoupled from Tier-0. VCF 9.1 supports HA mode per CTGW, multiple CTGWs and DTGWs per project, and multiple external connections per CTGW. For outbound traffic, tenants get full control over which Tier-0 is used, where SNAT is applied, and which External IP block is consumed.

    Why CSPs Care

    Per-project external connectivity with independent Tier-0 selection eliminates the shared gateway bottleneck. CSPs can model complex tenant topologies — multi-ISP, multi-region, dedicated uplinks — on shared infrastructure.

    07 / 20

    Distributed Transit Gateway with EVPN/VXLAN

    Peer directly with the physical fabric using industry-standard EVPN/VXLAN. This decouples the control and data plane for north-south traffic — VMs get direct N/S connectivity without traffic tromboning through Edge appliances. No edge lifecycle, no edge provisioning, no edge scaling headaches.

    Why CSPs Care

    Edge VM sprawl is one of the top operational pain points at CSP scale. DTGW with EVPN/VXLAN eliminates it entirely for N/S traffic — better latency, fewer failure domains, dramatically simpler operations.

    08 / 20

    Virtual Network Appliances (VNA) — Edge-Free Network Services

    A dedicated VNA Cluster now runs network services for Distributed External Connections: External IP (1:1 NAT), DHCP, NAT (SNAT/DNAT), VPC Outbound NAT (N:1 — new in 9.1), and NSX LB for Supervisor/VKS (new in 9.1) plus Avi VPC LB Plugin. Only NAT and LB traffic is redirected to VNAs — L2/L3 and External IP traffic remains fully distributed.

    Why CSPs Care

    Network services without deploying and managing Edge VMs per tenant. The distributed data-path keeps per-tenant traffic efficient while VNAs handle only the stateful services that need them.

    09 / 20

    TGW Span + Infoblox IPAM Integration

    Transit Gateway Span constrains a TGW and its subnets to selected vCenter clusters — controlling where subnets are available, where workloads can be placed, and aligning DTGW spans with external connection VLANs. Separately, Infoblox integration discovers and maps Network Containers to external IP blocks, provisions subnets/IPs using Infoblox CIDRs, and auto-registers workload IPs and FQDNs.

    Why CSPs Care

    TGW Span gives CSPs physical network alignment per tenant cluster — critical for VLAN-constrained environments. Infoblox integration provides the single DDI source-of-truth that large CSPs already depend on, now natively integrated with VCF networking.

    Storage & Data Efficiency
    10 / 20

    vSAN ESA Inline Compression (ZSTD) + Global Deduplication GA

    vSAN 9.1 introduces a ZSTD-based inline compression algorithm tuned specifically for vSAN — delivering significantly higher data reduction ratios while balancing CPU utilization. Compression is now always-on. In parallel, vSAN Global Deduplication reaches GA, supporting between 3 and 64 hosts with improved processing efficiency. Crucially, Global Dedup is fully compatible with Data-at-Rest encryption — no negative impact on reduction ratios.

    Why CSPs Care

    Direct $/TB improvement. Better compression + dedup = higher tenant density per physical disk. This is fundamental to CSP storage margin economics, especially for VDI, database, and backup workload profiles.

    11 / 20

    Auto-RAID + Effective Capacity View

    Auto-RAID automatically manages optimal resilience settings per cluster using a single “vSAN ESA Auto RAID Policy” in vCenter — dynamically adjusting as cluster size changes (4-host, 6-host stretched, 2-node, single-host bootstrap). The new “effective capacity” view replaces raw capacity statistics with usable capacity and simplified space-efficiency summaries covering dedup ratio, compression ratio, thin provisioning savings, and snapshot savings.

    Why CSPs Care

    No more manual storage policy tuning across hundreds of tenant clusters. Effective capacity view aligns with how CSPs bill and report storage — usable TB, not raw TB with overhead footnotes.

    12 / 20

    Native S3 Object Storage on vSAN — Technology Preview

    Block, file, and S3-compatible object storage running on the same vSAN cluster. Multi-tenant object storage is provisioned and managed via VCF Automation or vSphere Supervisor. Scalable, resilient architecture courtesy of vSAN ESA. Available as Technology Preview in Patch 01 of VCF 9.1.

    Why CSPs Care

    A new service tier on existing hardware. CSPs can offer S3-compatible object storage to tenants without deploying separate storage infrastructure — opening up developer-oriented and AI/ML data-lake use cases.

    Kubernetes & Containers
    13 / 20

    VKS: 500 Clusters per Supervisor + Fast Deploy

    VKS now supports up to 500 Kubernetes clusters per Supervisor — a 2.6× scale increase over VCF 9.0. VKS 3.6 ships Kubernetes 1.35 (CNCF-certified, 24-month support). Fast Deploy leverages linked-clone (unencrypted VMs) and direct-mode (encrypted VMs) technologies to reduce cluster provisioning time by approximately 70% and upgrades by approximately 75%.

    Why CSPs Care

    Dramatically higher Kubernetes tenant density per control plane instance. Fast Deploy addresses burst scenarios common in VDI and retail — and reduces time-to-revenue for new K8s tenant onboarding from 37 minutes to 11 minutes.

    14 / 20

    Container Service — CaaS Without Kubernetes

    Deploy isolated, secure containers directly on vSphere Pods within vSphere Namespaces — no full Kubernetes cluster required. UI-driven provisioning and lifecycle control. Supports StatefulSets with persistent volumes and multi-container pods. Based on the proven vSphere Pods technology with VM-level isolation.

    Why CSPs Care

    CSPs can offer a lightweight container service tier below full VKS — lower cost, faster deploy, familiar vSphere management. This broadens the addressable tenant market to teams that want containers but don’t need (or want to manage) Kubernetes.

    Operations & Lifecycle
    15 / 20

    Unified Fleet IAM & Management

    VCF 9.1 delivers end-to-end IAM with VCF-level roles across all components — vCenter, NSX, Operations, Automation, Logs, Networks, HCX, and Orchestration — all brokered through VIDB (Identity Broker). Unified password policies with vault integration, bulk certificate management (generate CSRs, renew, import across the fleet), and OAuth/API token access for programmatic automation. Custom VCF roles can be provisioned across vCenter and VCF instances.

    Why CSPs Care

    Single identity plane for the entire VCF estate. CSPs managing multi-instance fleets get consistent RBAC, password governance, and certificate rotation at scale — replacing the fragmented per-instance identity management that doesn’t survive operational audits.

    16 / 20

    Centralized LCM — 4× Parallel Upgrades

    Lifecycle Management is now part of the VCF Services Platform with a unified software depot secured via OAuth token. Optimized precheck workflows and a 4× improvement in parallel cluster upgrade operations — centrally managed from VCF Operations. One place to download and manage binaries, and quickly assess health and upgrade readiness across the fleet.

    Why CSPs Care

    CSPs running hundreds of clusters can upgrade 4× faster in parallel. Single depot and centralized LCM eliminates the maintenance-window sprawl that plagues large CSP environments — turning a weekend-long upgrade cycle into an overnight operation.

    17 / 20

    Flexible Licensing — License Server + Aggregated Usage

    VCF components are automatically licensed via vCenter when configured in connected mode. A dedicated license server offloads license logic from VCF Operations. Multiple licenses can be applied directly to a vCenter and its connected components. Aggregated license usage for ESX 8.x and 9.x. On-prem license appliance available for air-gapped or sovereign environments.

    Why CSPs Care

    CSPs with mixed-version estates (VCF 5.x through 9.x) get aggregated license management across generations. Override licenses support unique CSP scenarios — trial tenants, PoC environments, and tiered service offerings with differentiated entitlements.

    Security & Cyber Recovery
    18 / 20

    On-Premises Cyber Recovery Clean Room

    Full ransomware protection and recovery on customer-owned infrastructure — no cloud dependency. The solution extends vSAN Protection and Recovery to provide on-prem clean room capabilities with push-button vDefend-based network isolation, EDR integration (Carbon Black included by default, CrowdStrike BYOL supported), guided restore point selection, VM analysis and validation in the isolated environment, and orchestrated failback workflows.

    Why CSPs Care

    CSPs can offer “Cyber Recovery as a Service” as a premium tier — fully on-prem, data-sovereign, with clean room isolation that satisfies regulated industries prohibiting cloud-based recovery. The EDR vendor choice (Carbon Black or CrowdStrike) aligns with whatever the tenant already runs.

    19 / 20

    Security Posture Management & Compliance Automation

    Fleet-wide compliance assessments using built-in benchmarks — enable benchmarks, assign to policies, clone and modify rules to suit requirements. Run assessments on-demand, view and filter results, export to PDF/CSV, and perform one-click remediation to infrastructure objects. Confidential Computing visibility through the SecOps dashboard (AMD SEV-SNP, Intel TDX). VCF-wide audit trails with standardized log architecture for security forensics.

    Why CSPs Care

    Automated compliance reporting for regulated tenants (FIPS 140-3, STIG, custom benchmarks). One-click remediation across the fleet reduces CSP audit preparation from weeks to hours. The audit trail becomes a sellable compliance artifact for tenants in financial services and government.

    Edge
    20 / 20

    VCF Edge — 5,000 Hosts, 256 Parallel Upgrades, ZTP + GitOps

    Fleet capacity doubled to 5,000 ESX hosts per instance. Parallel upgrade scale increased 4× from 64 to 256 clusters. Zero Touch Provisioning uses UEFI HTTPS Boot with TPM and Secure Boot support — hosts inherit desired-state image and configuration from the cluster, no TFTP required. Day-0 activation scripts configure vSphere clusters, Supervisor, and FLB. Argo CD-based GitOps provides pull-based workload delivery with drift detection and auto-correction. Flexible 1/2/3+ node topologies with full air-gap support.

    Why CSPs Care

    CSPs serving retail, telco, or industrial edge can scale to thousands of sites with lights-out ZTP and GitOps delivery. 256 parallel upgrades make fleet-wide patching operationally viable — a requirement for edge CSPs where site-by-site maintenance windows are physically impossible.

    The CSP Takeaway

    VCF 9.1 is a platform release, not just a feature release. The networking overhaul (DTGW, VNAs, VPC policies, EVPN/VXLAN) alone justifies the upgrade for any CSP running multi-tenant workloads. Layer on the VCD migration tool, self-service namespaces, storage economics improvements, and fleet-scale operations — and this is the release that brings VCF’s cloud operating model to parity with what CSPs have been building manually around VCD for years.

  • I Built a Tool to Stop YAML Hell During Cloud → VCF 9 VKS Migrations

    I Built a Tool to Stop YAML Hell During Cloud → VCF 9 VKS Migrations

    The Problem:

    The Solution:

    What It Does: 

    🚀 Try it below

    // VMTECHIE.BLOG

    EKS/AKS/OCP → VKS/VCF9

    Upload K8s manifests → analyze cloud deps → transform for VKS on VCF 9 → download migration bundle

    1UPLOAD
    2ANALYSIS
    3BUNDLE
    UPLOAD & CONFIG
    Export: kubectl get all,cm,secret,pvc,ingress,sa,pdb,hpa -n <ns> -o yaml
    SOURCE PLATFORM
    EKS
    AKS
    OCP
    ◫
    DROP YAML FILES
    or click · multi-file · .yaml .yml
    OR PASTE YAML
    Separate docs with ---

    TARGET VKS CONFIG
    HARBOR FQDN
    On-prem Harbor registry
    https://
    HARBOR PROJECT
    /
    vSAN SC NAME
    VELERO BUCKET URL
    VELERO BUCKET NAME
    TARGET NS (opt)
    blank = keep original
    ANALYSIS
    Transformation complete.
    ISSUES & CHANGES
    MIGRATION BUNDLE
    All files ready. Review tabs then download ZIP.

    Step-by-Step Usage:

    # Connect to your source EKS/AKS/OpenShift cluster
    kubectl config use-context my-eks-cluster
    # Export all resources from production namespace
    kubectl get all,configmaps,secrets,pvc,ingress,serviceaccounts,pdb,hpa \
    -n production -o yaml > production-export.yaml
    # Repeat for each namespace you're migrating
    kubectl get all,cm,secret,pvc,ingress,sa,pdb,hpa -n staging -o yaml > staging-export.yaml
    # oc get all,cm,secret,pvc,route,sa,pdb,hpa,deploymentconfigs,imagestreams \
    -n production -o yaml > production-export.yaml

    Known Limitations:

    Disclaimer & Privacy: