Tag: VMware Cloud Foundation

  • GitOps as a Service on VCF 9.1.1 (Part 1: Installing Argo CD)

    GitOps as a Service on VCF 9.1.1 (Part 1: Installing Argo CD)

    This is a lab build log, written as I went. Everything here was done once, on one set of versions, in my own lab. Where I am reporting what the documentation says instead of what I saw, I say so.

    I wanted to answer one question: can a tenant on VMware Cloud Foundation get its own Argo CD, sign in with its VCF Automation identity, and then build a virtual machine, pods and a Kubernetes cluster from a single Git repository? Part 1 gets the platform ready and ends with a working Argo CD instance. Part 2 uses it.

    Why bother with GitOps on VCF

    Most platform teams I talk to have the same three people in the room. An application developer who raised a ticket for a VM and a cluster days ago. A platform engineer who found that production no longer matches the runbook because somebody changed it by hand. And someone from security asking who changed what, when, and who approved it.

    GitOps answers all three with one idea: Git holds the desired state, and a controller keeps the platform matching it. A request is a pull request. Drift is detected and corrected. The audit trail is the commit history. Argo CD is the engine most teams pick for this: it is a CNCF graduated project, it has a UI that application teams actually use, and it takes Helm charts, Kustomize and plain YAML without repackaging.

    Three people and their problems without GitOps, and what changes with Git as the source of truth
    The problem GitOps solves, and why Argo CD.

    You can run upstream Argo CD on any Kubernetes cluster. What VCF adds is everything around it:

    • Nothing to build first. The instance runs as vSphere Pods on the Supervisor. There is no management cluster to stand up.
    • An operator runs it. You pick from VMware-built Argo CD versions; in my lab 3.4.4 and 3.3.12 were the recommended ones.
    • Sign-in is already there. The create wizard sets VCF Automation as the identity provider.
    • One instance per tenant. Each tenant creates its own Argo CD in its own namespace from the org portal.
    • Targets are attached, not hand-registered. You attach namespaces and VKS clusters from a Scope tab, or with a manifest in Git.
    • A VM is a manifest too. The same API that takes a Deployment takes a VirtualMachine and a Cluster. That is the subject of Part 2.
    Argo CD assembled yourself compared with the Argo CD Service on VCF 9.1.1
    What you would otherwise build and keep working yourself.

    Two routes to Argo CD on VCF 9.1.1

    Supervisor Service, from vCenterVCF Service, through VCF Automation
    StatusSupported for productionTech Preview in 9.1.1
    InstallvCenter, Supervisor ServicesUpload in Partner Services, install from VCF Services
    Create an instanceApply an ArgoCD resource in a namespaceSelf-service from the org portal
    Sign-inLocal admin, or OIDC you configureVCF Automation, configured for you
    Attach targetsManagedEntity resourcesScope tab, or a ManagedEntity in Git

    I followed the right-hand column. I did not exercise the left-hand one in this lab; its rows come from the VCF blog announcing the 9.1.1 GitOps service and the product documentation. Underneath, the VCF Service installed an ordinary Supervisor package on my Supervisor, argocd-service.vsphere.vmware.com.

    The lab

    ComponentVersion or name
    VCF and VCF Automation9.1.1
    SupervisorKubernetes v1.34.9, single control-plane node
    NetworkingNSX VPC with the NSX load balancer
    Regional HarborVCF Service 9.1.1.0.25714559, Harbor 2.15.2
    Argo CD Service1.2.0, build 25642124
    Argo CD instance3.4.4
    TenantOrganization org2, project default-project, namespace org2-ns1-mrrtd

    Step 1: Regional Harbor comes first

    Before anything else, check whether your Supervisor has the depot service:

    $ kubectl -n kube-system get svc depot
    Error from server (NotFound): services "depot" not found

    That was my starting point. Regional Harbor is what creates this service, and Supervisor Services pull their images through it. Broadcom KB 447329 describes the failure you get if you install a service without it. Harbor is installed from the provider portal under Service Management → VCF Services.

    VCF Services page with Harbor listed as not installed
    Harbor under VCF Services, not yet installed.

    The wizard has three pages: accept the certificates, choose the region, and keep the default properties. The review page is worth reading because it explains the dependency: besides Harbor itself, the install creates overlays that publish Harbor to every Supervisor in the region and create the depot endpoint.

    Harbor review page listing the resources to be created
    Eleven resources, including the overlays that wire each Supervisor to this Harbor.

    In my lab it took about ten minutes to reach Healthy. One pod, harbor-jobservice, crash-looped twice before settling; it starts before the components it depends on. Afterwards the depot resolves:

    $ kubectl -n kube-system get endpoints depot
    depot   10.150.4.10:443
    Harbor service showing Healthy with all components green
    Harbor Healthy.

    Step 2: get the Argo CD Service package

    I expected the Argo CD Service to appear in the catalog once Harbor was in, because my lab uses an online depot. It did not.

    VCF Services page with no Argo CD card
    Harbor was listed. Argo CD was not.

    The documented flow is to download the package with the VCF Download Tool and upload it yourself:

    1. Generate a Software Depot ID with vcf-download-tool.
    2. Register it on the Broadcom portal to get an activation code. The activation code replaced the old download token in 9.1 (KB 443647).
    3. Download the component VCF_SERVICE_ARGOCD for version 9.1.1.
    4. In the provider portal, go to Service Management → Partner Services → Upload.

    The download tool talks to the Broadcom depot directly with the activation code. It does not go through the VCF Operations depot, which confused me at first.

    Partner Services upload dialog reading the Argo CD package
    Uploading the package. It is about 1 GB, so keep the tab open.
    ArgoCD Service card under VCF Services
    After the upload, the card appears under VCF Services.

    Step 3: install it, and think about two defaults

    The install wizard is the same shape as Harbor’s. Two defaults deserve a decision instead of a click:

    • Scope. The default deploys the service to every region, including ones you add later.
    • keepCRDs: false. An uninstall removes the custom resource definitions, and with them every Argo CD instance your tenants created. Outside a lab I would set this to true.
    Argo CD Service review page listing eight resources
    Eight resources: the operator on the Supervisor, and the pieces that wire it into VCF Automation.

    Those eight resources fall into three groups. The operator runs on the Supervisor. A UI plugin and a global role add the GitOps pages to tenant org portals. And a ConfigurationHandler bridges the two by handing VCF Automation connection details to the Supervisor side. Remember that last one.

    The build that failed

    My first attempt used build 25568464. Seven resources went Healthy. The ConfigurationHandler did not:

    minorErrorCode: BAD_REQUEST
    RDE_RESTRICTED_FIELDS [ $.secrets ]
    ConfigurationHandler showing Unhealthy
    Seven Healthy, one not.

    The operator itself was fine. What was broken was the integration: no org-portal self-service and no single sign-on. Worse, the uninstall then stalled, because the delete path tried to call an entity that had never been created.

    What I did, and why you should not copy it blindly. I cleared the stuck uninstall by removing orphaned finalizers on three VCF Automation objects, after backing each one up. These are VCF Automation internals and this is not a documented procedure. In a lab that is a reasonable risk. Anywhere else, open a support case first. The exact steps follow so you can recognise the situation, and so support can see what was done.

    Why the uninstall was stuck

    The ConfigurationHandler had a deletionTimestamp and one finalizer, services.vcfa.broadcom.com/vcfa-service. Before the controller releases that finalizer it calls a delete behaviour in VCF Automation. But the entity was never created, because the original create had failed, so the delete call went out with an empty ID:

    .../cloudapi/1.0.0/entities//behaviors/...      400 Bad Request
    behaviorFailureCount: 101
    delete backoff cooldown active, requeue after 10m0s

    Note the entities//. The controller was trying to delete something that did not exist, and it would have retried forever.

    The cleanup, step by step

    Everything below was run as root on the VCF Automation node, against its own Kubernetes API, in the prelude namespace. The names are UUIDs from my lab; yours will differ. I took the handler’s name from its YAML in the provider portal. That YAML also names the service account it uses, in callbackServiceAccountName.

    CH=de35349b-a9c1-5ead-85a1-e40fa7fba23f     # the stuck ConfigurationHandler
    SA=8df98ef4-ea51-56be-96f1-a515ac6f75d5     # its callback service account (and Secret of the same name)

    1. The ConfigurationHandler. Back it up, then remove its finalizer.

    kubectl -n prelude get configurationhandler $CH -o yaml > ch-backup.yaml
    
    kubectl -n prelude patch configurationhandler $CH \
      --type=merge -p '{"metadata":{"finalizers":[]}}'

    The handler disappeared at once. I had also restarted the vcfa-service-manager deployment just before this. I did not wait to see whether the restart alone would help, and I would not expect it to: the fault was in the data, not in a stuck process.

    2. The callback service account. The cleanup then stopped on the next object. These objects are named by UUID, so searching for “argocd” finds nothing; use the label.

    kubectl -n prelude get vcfaserviceaccount,secret \
      -l services.vcfa.broadcom.com/refName=argocd-service.vsphere.vmware.com
    
    kubectl -n prelude get vcfaserviceaccount $SA \
      -o jsonpath='deletionTimestamp={.metadata.deletionTimestamp}{"\n"}finalizers={.metadata.finalizers}{"\n"}'
    
    deletionTimestamp=2026-11-11T09:34:08Z
    finalizers=["services.vcfa.broadcom.com/vcfa-service",
                "services.vcfa.broadcom.com/configurationhandler.6866f8ff-..."]

    Two finalizers, and they are not equal. The second belongs to the handler I had just removed, so nothing would ever clear it. The first is the controller’s own, and it should stay, so that VCF Automation still deletes the account properly on its side. I removed only the orphan:

    kubectl -n prelude get vcfaserviceaccount $SA -o yaml > vsa-backup.yaml
    
    # confirm which index holds the configurationhandler finalizer before removing it
    kubectl -n prelude get vcfaserviceaccount $SA -o jsonpath='{.metadata.finalizers[1]}{"\n"}'
    
    kubectl -n prelude patch vcfaserviceaccount $SA \
      --type=json -p='[{"op":"remove","path":"/metadata/finalizers/1"}]'

    3. The Secret. The controller picked the cleanup up again and reported what it was waiting for:

    requeue after 10s
    Unhealthy
    waiting for controller-created secret to be fully deleted

    The Secret with the same UUID was held by one more handler-owned finalizer:

    kubectl -n prelude get secret $SA \
      -o jsonpath='deletionTimestamp={.metadata.deletionTimestamp}{"\n"}finalizers={.metadata.finalizers}{"\n"}'
    
    finalizers=["addons.vcf.vmware.com/configuration-handler"]
    
    kubectl -n prelude get secret $SA -o yaml > secret-backup.yaml
    
    kubectl -n prelude patch secret $SA \
      --type=json -p='[{"op":"remove","path":"/metadata/finalizers/0"}]'

    The Secret backup contains credentials. Keep it only as long as you need it for a support case.

    4. Check that nothing is left. A moment later both objects were gone. The service account going away by itself is the good sign: the controller released its own finalizer.

    # on the VCF Automation node: should return nothing
    kubectl -n prelude get vcfaserviceaccount,secret | grep 8df98ef4
    
    # on the Supervisor: should return nothing
    kubectl get pkgi -A | grep -i argo
    kubectl get ns | grep -i argo
    kubectl get crd | grep -i argocd

    In the provider portal, check Administration → Access Control for leftover Argo CD service accounts or roles, and that the service card is back to Not Installed. Leftovers there are what make a reinstall fail with “already exists” (KB 453499). In my lab everything was gone, and the reinstall in the next step went through. Keep the three backup files and a note of the order of events; that is what a support case will ask for.

    The second attempt used build 25642124 and went Healthy within minutes.

    All eight resources Healthy on build 25642124
    Build 25642124: all eight Healthy.

    I cannot yet say the first build is bad. Both builds were uploaded to VCF Automation at the time, and I have not repeated the install of 25568464 on its own. Until I do, my advice is simply to upload one build, and make it 25642124.

    On the Supervisor, the result looks like this:

    $ kubectl get pkgi -A | grep argo
    svc-argocd-service...   1.2.0-25642124   Reconcile succeeded
    
    $ kubectl get crd | grep argocd
    argocds · argocdversions · managedentities · entitymanagementpolicies

    Step 4: a tenant creates its own instance

    Everything from here is done as a tenant user in the org portal. Under Build & Deploy → Services → GitOps there is now a button to create an Argo CD instance. That page existing at all is the proof that the install wired the tenant side correctly.

    Org portal GitOps page with Create Argo CD Instance
    The GitOps page in the org portal.

    I named the instance org2-argo, chose the project and namespace, and picked version 3.4.4. The wizard warns that project admins and project advanced users become admins of this Argo CD. Take that seriously: project membership is your security boundary.

    OIDC and RBAC page of the create wizard
    VCF Automation is the identity provider by default.

    The RBAC page holds ordinary Argo CD policy. If you have not read one before:

    g, org-admin, role:admin          # g assigns a user or group to a role
    p, role:app-deployer, applications, sync, default/*, allow   # p says what a role may do

    A few minutes after submitting, the instance showed Success, with seven pods in the tenant namespace and a load-balancer address of its own.

    Instance status Success with pods running
    The instance is up.
    One Argo CD instance per tenant
    The provider installs once. Each tenant runs its own Argo CD.

    Step 5: sign in with VCF Automation

    Argo CD login page with a Log in via VCF Automation button
    The button I was hoping for.

    This button is what the failed ConfigurationHandler would have cost me. It worked on the first try. The User Info page shows what Argo CD received from VCF Automation, and those are the values to use in RBAC.

    Argo CD after signing in through VCF Automation
    Signed in. No applications yet.

    One thing I have not settled: my test user is literally named org-admin, so I cannot tell whether the group Argo CD sees comes from the VCF Automation role or from the username. I need to repeat this with a differently named user.

    Step 6: give it somewhere to deploy

    A new instance has no destinations, not even its own namespace. You add them from the Scope tab in the org portal. I attached the namespace org2-ns1-mrrtd.

    Attach Namespace dialog
    Any namespace you can reach in the region, from any project.
    Argo CD Settings, Clusters, showing the attached namespace
    The namespace appears in Argo CD as a destination behind the VCF Automation proxy.

    Look at the URL in that last screenshot. Argo CD does not hold a kubeconfig for the Supervisor. It reaches the namespace through the VCF Automation Kubernetes proxy, using a service account the platform created for the instance. So there are two separate gates: who may use Argo CD, and what Argo CD itself may change.

    Two access paths: signing in to Argo CD, and Argo CD changing things
    An Argo CD admin still cannot make Argo CD create something its service account is not allowed to.

    What I would tell a customer so far

    Three observations from this half. Each was seen once and still needs reproducing.

    1. Install Regional Harbor before any Supervisor Service. The depot service did not exist until Harbor created it.
    2. The online depot did not bring the Argo CD Service. I had to download and upload it.
    3. Build 25568464 failed for me and 25642124 did not. Whether that was the build, or having two builds uploaded, is still open.

    At this point a tenant has a private Argo CD, single sign-on, and one namespace attached. In Part 2 I point it at a Git repository and let it build a virtual machine, vSphere Pods, a VKS cluster and the application inside that cluster, then deploy a twelve-service shop from the Argo CD GUI and fix a real fault along the way.

  • Building a Production VKS Cluster on VCF 9.1: Foundation LB, Harbor, PowerStore NFS and Istio

    Most VKS guides stop at “cluster created”. A production build has more to it: VLANs that must be proven before vSphere touches them, a private registry the cluster must trust from its first boot, and shared storage that pods can actually write to.

    This post walks through a production build I did on VCF 9.1 with the Foundation Load Balancer, Harbor, a 7-node VKS cluster, NFS for ReadWriteMany volumes, and Istio, Contour and Headlamp on top. The commands that matter are all here, in the order you need them.

    All names and IP addresses are examples from a made-up network plan, and the terminal screenshots were re-typed with those example values.

    What is in this post

    • Before you start: what to have ready, including the full list of URLs to allow
    • Step 0: prove the network · Step 1: the jump server · Step 2: the Supervisor and load balancer
    • Step 3: Harbor · Step 4: the vSphere Namespace · Step 5: the cluster · Step 6: access
    • Step 7: ReadWriteMany storage · Step 8: Istio and Contour · Step 9: Headlamp · Final checks

    What we are building

    VKS on VCF 9.1 architecture with three VLANs
    Three VLANs: management, Supervisor workload with the load balancer VIPs, and the VKS node network with the NFS server on it.
    ComponentVersion / size
    VMware Cloud Foundation9.1
    SupervisorVDS networking, control plane HA, size Medium
    Load balancerFoundation Load Balancer, One-Arm, Medium, 2 VMs
    vSphere Kubernetes Service3.6, ClusterClass builtin-generic-v3.6.0
    Kubernetes releasev1.35.5
    RegistryHarbor 2.15.2 as a Supervisor Service
    Cluster3 control-plane nodes (12 vCPU / 48 GB) + 4 workers (24 vCPU / 64 GB, 500 GB container disk)
    Shared storageNFSv4.1 export with csi-driver-nfs (a Dell PowerStore in my case; any NFSv4 server works)
    Add-onscert-manager, Istio, Contour (Gateway API CRDs ship with the VKr)
    ConsoleHeadlamp

    All work is done from a Windows jump server with the VCF CLI, kubectl, Helm and crane. No Docker is needed anywhere. Every third-party image is copied into Harbor first, so the cluster pulls from one registry you control.

    Example network plan

    VLANSubnetUsed for
    10010.10.0.0/24Management: vCenter, ESXi, jump server, Supervisor management IPs (10.10.0.24 to .28), FLB management IPs
    20010.20.0.0/24Supervisor workload network, Harbor, FLB data NICs (.200 to .209) and VIPs (.210 to .239)
    30010.30.0.0/26VKS cluster nodes (.10 to .50) and the NFS server interface (.60)

    Inside the cluster: pod CIDR 10.244.0.0/20 (room for 16 nodes) and service CIDR 10.100.0.0/24. Neither can be changed after the cluster exists, so check they don’t overlap anything the pods need to reach.

    Before you start

    Have these in place before Step 2:

    • Three VLANs trunked on every host uplink, with gateways, routes in both directions, and NAT for internet egress.
    • DNS records for the Supervisor API and for Harbor, created once their VIPs are known (the first VIP from the range goes to the Supervisor).
    • Internal firewall rules: load balancer data NICs → nodes (TCP 6443, 80, 443, 30000 to 32767); nodes → Harbor and Supervisor (443, 6443); jump server → VIPs and nodes (443, 6443, 22).
    • An NFS server with an interface on the node VLAN and NFSv4 enabled.
    • administrator@vsphere.local for every vCenter step, and a Broadcom Support Portal login for the downloads.
    • A Windows jump server with internet access, for the CLI, Helm and crane.

    Internet allow list

    This is the complete list of what had to be reachable, by source. Allow by FQDN or SNI, never by IP: every one of these is CDN-backed and the addresses rotate. If the firewall does TLS inspection, exclude these names or make the Supervisor, Harbor and the nodes trust the inspection CA. The CDN names are the ones that get missed: a rule for registry.k8s.io without *.pkg.dev gives you pulls that connect and then hang. To find a CDN host, run curl.exe -sI https://registry.k8s.io/v2/pause/manifests/3.10 and read the location: header.

    FromAllow (TCP 443 unless noted)Why
    vCenterwp-content.broadcom.com, wp-content.broadcom.com.cdn.cloudflare.netKubernetes release content library
    Supervisor control plane and Supervisor workload networkprojects.packages.broadcom.com, packages.broadcom.com, vcf.packages.broadcom.com, plus the CDN behind them: broadcom.jfrog.io, *.jfrog.io, *.elb.us-west-2.amazonaws.com, jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.comSupervisor Services including Harbor’s own images, the VKS service, add-on and package bundles
    VKS cluster nodesThe same Broadcom names as aboveAdd-on images; everything else comes from Harbor
    Jump serversupport.broadcom.com, github.com, objects.githubusercontent.com, raw.githubusercontent.com, get.helm.sh, kubernetes-sigs.github.ioVCF CLI, Helm, crane and the Helm charts
    Jump serverregistry.k8s.io, *.pkg.dev, prod-registry-k8s-io-*.s3.dualstack.*.amazonaws.comCopying the NFS CSI driver and test images into Harbor with crane
    Jump serverghcr.io, pkg-containers.githubusercontent.comCopying the Headlamp image into Harbor
    Jump serverocsp.digicert.com, crl3.digicert.com, crl4.digicert.com on TCP 80Code-signing revocation checks; without them the CLI starts very slowly
    Harbor (only if you use its proxy cache)Docker Hub: registry-1.docker.io, auth.docker.io, production.cloudflare.docker.com · GHCR: ghcr.io, pkg-containers.githubusercontent.com · Quay: quay.io, cdn01.quay.io to cdn03.quay.io · registry.k8s.io with *.pkg.dev · your <name>.azurecr.ioPulling upstream images through Harbor instead of copying them by hand, and the Trivy vulnerability database

    Step 0: prove the network before vSphere uses it

    Before you hand a VLAN to the Supervisor, put a small Linux VM on its port group, once on every host, and check:

    1. the gateway answers,
    2. a VM on another VLAN answers, and the reply comes back,
    3. curl -I https://projects.packages.broadcom.com returns an HTTP status line.

    Test every host, not just one: a VLAN missing from one host’s uplink trunk only breaks traffic between VMs on different hosts, which is hard to spot later. While the test VM is up, also check the allow list above from each VLAN that needs it.

    Step 1: prepare the jump server

    Download the VCF CLI for Windows and its plugin bundle from the Broadcom Support Portal. Unblock the downloaded files first, or Windows scans every CLI call and the tools run very slowly:

    Get-ChildItem C:\tools -Recurse | Unblock-File
    vcf plugin install all --local-source C:\tools\vcf-plugin-bundle

    Trust the vCenter root CA in an administrator PowerShell:

    curl.exe -k -o C:\temp\vc-certs.zip https://vcsa01.corp.example.com/certs/download.zip
    Expand-Archive C:\temp\vc-certs.zip -DestinationPath C:\temp\vc-certs -Force
    Get-ChildItem C:\temp\vc-certs\certs\win\*.crt | ForEach-Object { certutil -addstore -f Root $_.FullName }

    Install Helm and crane (Google’s registry copy tool, which copies images between registries without Docker):

    curl.exe -L -o C:\temp\crane.tar.gz https://github.com/google/go-containerregistry/releases/download/v0.20.3/go-containerregistry_Windows_x86_64.tar.gz
    tar -xzf C:\temp\crane.tar.gz -C C:\tools crane.exe

    Step 2: enable the Supervisor with the Foundation Load Balancer

    In the vSphere Client: Supervisor Management → Get Started, VDS networking. The values that matter:

    Wizard pageValue
    Load balancerFoundation Load Balancer, One-Arm, Medium, 2 VMs
    LB management networkManagement port group, 10.10.0.29 to 10.10.0.32 (a couple of spare IPs beyond the two VMs)
    LB virtual server networkWorkload port group; data NIC IPs 10.20.0.200 to .209; VIP range 10.20.0.210 to .239 (the two ranges must not overlap)
    Management networkStarting IP 10.10.0.24 (uses 5 in a row)
    Workload network10.20.0.40 to .99, service CIDR 10.96.0.0/23
    Control plane sizeMedium (it can grow later, never shrink)

    How the Foundation Load Balancer uses its IPs

    The wizard asks for three ranges, and each has one job. Management IPs (one per LB VM) are how the Supervisor configures the load balancer. Data NIC IPs (one per LB VM) are the source address of all traffic the load balancer sends to backends, including health checks. The VIP range is handed out on demand: every Service of type LoadBalancer, on the Supervisor or inside a cluster, takes the next free VIP and gives it back when deleted. Size it for one Supervisor API, one per Supervisor Service, one per cluster and one per app ingress, plus headroom.

    Foundation Load Balancer IP pools and VIP assignment
    The three FLB IP ranges, which NIC each lands on, and how a VIP is assigned to a LoadBalancer Service.

    At run time, clients only ever connect to a VIP. The active LB VM forwards the connection to a control-plane node on 6443 (Kubernetes API) or to a worker NodePort (apps), and rewrites the source to its data NIC IP, so replies always come back through it. Two consequences: inbound firewall rules only need LB data NICs → nodes, and pod logs show the data NIC IP as the client. In the other direction, traffic leaving a pod is NATed to the node’s IP, so outbound firewall rules use the node subnet as the source, never the pod or service CIDRs.

    Foundation Load Balancer traffic flow
    Traffic paths: Kubernetes API (A), applications through a NodePort (B), and replies returning to the LB data NIC (C).

    When the Supervisor is Running, the first VIP (.210) is the API. Log in:

    vcf context create sup01 --endpoint 10.20.0.210 --username administrator@vsphere.local --type k8s
    vcf context use sup01

    Step 3: Harbor as a Supervisor Service

    Without a VCF Operations software depot, use Harbor’s legacy Supervisor Service manifest. Register it under Supervisor Management → Services, then install it with a data-values file. The keys that matter:

    hostname: harbor.corp.example.com
    enableNginxLoadBalancer: true
    enableContourHttpProxy: false
    harborAdminPassword: "<strong password>"
    secretKey: "<exactly 16 characters>"
    core:
    xsrfKey: "<exactly 32 characters>"

    Harbor runs as vSphere Pods on the Supervisor workload network, so that subnet needs internet egress and a working return route for the image pulls. Once the pods are Running, save Harbor’s CA; every cluster needs it to trust the registry:

    $b64 = kubectl get secret harbor-tls -n <svc-harbor-namespace> -o jsonpath="{.data.ca\.crt}"
    [IO.File]::WriteAllBytes("C:\temp\harbor-ca.crt", [Convert]::FromBase64String($b64))
    certutil -addstore -f Root C:\temp\harbor-ca.crt

    In the Harbor UI, create a Public project called k8s for the third-party images that come later, and log crane in: crane auth login harbor.corp.example.com -u admin --password-stdin.

    Step 4: the vSphere Namespace

    • Add a second workload network for the node VLAN (10.30.0.10 to .50) under Supervisor → Configure → Network → Workload Network.
    • Create a subscribed content library for Kubernetes releases (https://wp-content.broadcom.com/v2/latest/lib.json) and synchronize the version you will use, so it is stored locally.
    • Create custom VM classes: cp-12c-48g (12 vCPU / 48 GB) and worker-24c-64g (24 vCPU / 64 GB).
    • Create the namespace apps on the node network (it can only be chosen at creation), then add the storage policy, the VM classes and the content library.

    Step 5: the cluster

    First, the Harbor trust secret. Its name must be <cluster>-user-trusted-ca-secret, and it must exist before the cluster is applied. Deleting a cluster removes its secret too, so recreate it before any re-apply:

    $b1 = [Convert]::ToBase64String([IO.File]::ReadAllBytes("C:\temp\harbor-ca.crt"))
    kubectl create secret generic prod-vks01-user-trusted-ca-secret -n apps --from-literal=harbor-ca=$b1

    Then the cluster. On VKS 3.x, trust goes under the osConfiguration variable; the old top-level trust variable is rejected:

    apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    metadata:
    name: prod-vks01
    namespace: apps
    spec:
    clusterNetwork:
    pods:
    cidrBlocks: ["10.244.0.0/20"]
    services:
    cidrBlocks: ["10.100.0.0/24"]
    serviceDomain: cluster.local
    topology:
    class: builtin-generic-v3.6.0
    classNamespace: vmware-system-vks-public
    version: v1.35.5---vmware.1-vkr.1
    controlPlane:
    replicas: 3
    workers:
    machineDeployments:
    - class: node-pool
    name: np1
    replicas: 4
    variables:
    overrides:
    - name: vmClass
    value: worker-24c-64g
    - name: volumes
    value:
    - name: containerd
    mountPath: /var/lib/containerd
    capacity: 500Gi
    storageClass: vks-storage-policy
    variables:
    - name: vmClass
    value: cp-12c-48g
    - name: storageClass
    value: vks-storage-policy
    - name: osConfiguration
    value:
    trust:
    additionalTrustedCAs:
    - caCert:
    secretRef:
    name: prod-vks01-user-trusted-ca-secret
    key: harbor-ca

    The volumes override gives each worker a dedicated 500 GB disk for container images, so large images never fill the root disk.

    kubectl apply -f C:\temp\prod-vks01.yaml
    kubectl get cluster,machines -n apps

    Two minutes after the apply, check that the cluster’s endpoint matches its load balancer service: kubectl get cluster prod-vks01 -n apps -o jsonpath='{.spec.controlPlaneEndpoint.host}' must equal the EXTERNAL-IP of kubectl get svc prod-vks01 -n apps. If you ever rebuild, give the new cluster a name that has not been used in that namespace before.

    Cluster available
    The cluster with 3/3 control-plane and 4/4 worker machines.

    In about 20 minutes the cluster shows AVAILABLE True. Log in:

    vcf context create prod-vks01 --endpoint 10.20.0.210 --username administrator@vsphere.local --workload-cluster-name prod-vks01 --workload-cluster-namespace apps
    vcf context use prod-vks01:prod-vks01
    kubectl get nodes -o wide
    Nodes ready
    All 7 nodes Ready on v1.35.5.

    Step 6: access without handing out kubeconfigs

    Everyone logs in with their own directory account through the vSphere Namespace, and the namespace role decides what they get. Grant the roles from the namespace’s Summary → Permissions card as administrator@vsphere.local:

    Namespace roleInside the clusterGive to
    Can viewLogin only; rights come from RoleBindings you addApplication teams
    Can editcluster-admin, created automaticallyPlatform admins

    For each application, create its namespace, set a Pod Security level, and bind the team’s group to the built-in edit role there. VKS clusters enforce restricted by default, which refuses most images that run as root; baseline suits most applications:

    kubectl create ns app1
    kubectl label ns app1 pod-security.kubernetes.io/enforce=baseline
    kubectl create rolebinding team-edit -n app1 --clusterrole=edit --group="sso:app-team@corp.example.com"

    Step 7: ReadWriteMany storage on NFS

    An NFS datastore mounted on ESXi gives pods only ReadWriteOnce volumes. For volumes shared across pods, the cluster needs an NFS CSI driver talking to the NFS server directly. Putting the server’s interface on the node VLAN keeps NFS off routers and firewalls entirely.

    On the NFS server, create the export /k8s-rwx with NFSv4 enabled, and give host access to the whole node subnet (10.30.0.0/26) as read/write with root allowed (no root squash). The subnet rather than node IPs, because nodes are replaced on every upgrade; root, because the driver creates a folder per volume as root.

    NFS export settings for Kubernetes
    What the export needs: node subnet access, read/write, root allowed, NFSv4.1 on the node VLAN.

    Test a write from a worker before installing anything. The mount succeeds even when root is squashed; only the write tells you the export is right:

    sudo mkdir -p /mnt/t && sudo mount -t nfs -o vers=4.1 10.30.0.60:/k8s-rwx /mnt/t
    sudo touch /mnt/t/test && sudo chown 1000:1000 /mnt/t/test && ls -ln /mnt/t
    sudo rm /mnt/t/test && sudo umount /mnt/t
    NFS mount test
    An nfs4 mount with vers=4.1 and rw, and a test file written as root.

    Copy the driver images into Harbor with crane (chart 4.13.6 shown):

    $h = "harbor.corp.example.com/k8s/sig-storage"
    crane copy --platform linux/amd64 registry.k8s.io/sig-storage/nfsplugin:v4.13.6 $h/nfsplugin:v4.13.6
    crane copy --platform linux/amd64 registry.k8s.io/sig-storage/csi-provisioner:v6.3.0 $h/csi-provisioner:v6.3.0
    crane copy --platform linux/amd64 registry.k8s.io/sig-storage/csi-resizer:v2.2.1 $h/csi-resizer:v2.2.1
    crane copy --platform linux/amd64 registry.k8s.io/sig-storage/csi-snapshotter:v8.6.0 $h/csi-snapshotter:v8.6.0
    crane copy --platform linux/amd64 registry.k8s.io/sig-storage/livenessprobe:v2.19.0 $h/livenessprobe:v2.19.0
    crane copy --platform linux/amd64 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.17.0 $h/csi-node-driver-registrar:v2.17.0
    crane copy --platform linux/amd64 registry.k8s.io/sig-storage/snapshot-controller:v8.6.0 $h/snapshot-controller:v8.6.0
    Harbor repositories
    The seven driver images in the Harbor k8s project.

    Install the chart pointing at Harbor:

    helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
    helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n kube-system --version 4.13.6 --set image.baseRepo=harbor.corp.example.com/k8s --set image.nfs.repository=harbor.corp.example.com/k8s/sig-storage/nfsplugin
    CSI NFS pods
    The CSI controller (5/5) and one node pod per node (3/3), all pulled from Harbor.

    Then the StorageClass. Retain keeps the data folder on the NFS server when a volume is deleted, which is what you want in production:

    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
    name: nfs-rwx
    provisioner: nfs.csi.k8s.io
    parameters:
    server: 10.30.0.60
    share: /k8s-rwx
    reclaimPolicy: Retain
    volumeBindingMode: Immediate
    allowVolumeExpansion: true
    mountOptions:
    - nfsvers=4.1
    - hard

    Prove it with a two-replica Deployment, forced onto different nodes with pod anti-affinity, both appending to the same file on one ReadWriteMany PVC. If tail shows both pod names, RWX works across nodes. Note that this driver does not enforce a size per volume; everything shares the export’s capacity.

    Step 8: Istio, Contour and Gateway API

    VKS add-ons are installed from the Supervisor namespace context, not from inside the cluster:

    vcf context use sup01:apps
    vcf addon install create cert-manager --cluster-name prod-vks01 -y
    vcf addon install create istio --cluster-name prod-vks01 -y
    vcf addon install list --cluster-name prod-vks01

    There is no need for the gateway-api add-on on VKr v1.35.5: the Gateway API CRDs ship with the Kubernetes release, and Istio’s istio GatewayClass is accepted straight away. Check with kubectl get crd | Select-String gateway.networking before installing it on other releases.

    Gateway API CRDs and GatewayClass
    Gateway API CRDs present from cluster creation, and the istio GatewayClass accepted.

    Contour’s Envoy defaults to NodePort. To get a LoadBalancer VIP, pass a values file, which requires an explicit version:

    envoy:
    service:
    type: LoadBalancer
    vcf addon available list contour
    vcf addon install create contour --cluster-name prod-vks01 --version <version> -f C:\temp\contour-values.yaml -y
    Add-ons ready
    cert-manager, Contour and Istio all READY.

    A fresh Envoy resets connections until at least one Ingress or HTTPProxy exists, so a connection reset on a new ingress VIP is expected. To test Istio end to end, apply a Gateway (class istio, port 80), a small web Deployment and Service, and an HTTPRoute pointing at it. Istio creates the gateway pod and its LoadBalancer service from the Gateway itself; when the Gateway shows PROGRAMMED True, curl its address.

    Step 9: Headlamp as the cluster console

    Headlamp gives the team a browser view of workloads, logs and events without handing out kubeconfigs. Same pattern as the NFS driver: copy the image into Harbor, install with Helm, expose on a VIP. The chart already creates a ClusterRoleBinding called headlamp-admin, so the login account’s binding needs a different name:

    crane copy --platform linux/amd64 ghcr.io/headlamp-k8s/headlamp:v0.46.0 harbor.corp.example.com/k8s/headlamp:v0.46.0
    helm repo add headlamp https://kubernetes-sigs.github.io/headlamp/
    helm install headlamp headlamp/headlamp -n headlamp --create-namespace --version 0.46.0 --set image.registry=harbor.corp.example.com --set image.repository=k8s/headlamp --set image.tag=v0.46.0 --set service.type=LoadBalancer
    kubectl create sa headlamp-login -n headlamp
    kubectl create clusterrolebinding headlamp-login --clusterrole=cluster-admin --serviceaccount=headlamp:headlamp-login
    kubectl create token headlamp-login -n headlamp --duration=24h

    Open http://<EXTERNAL-IP>, choose Token, and paste it. For a production URL, put it behind Contour with a hostname and a certificate, and switch the Service back to ClusterIP.

    Final checks

    Before handing the platform to application teams, confirm all of these:

    • kubectl get cluster -n apps shows AVAILABLE True, and 7 nodes are Ready on the expected IPs.
    • The cluster’s controlPlaneEndpoint equals the EXTERNAL-IP of its Service on the Supervisor.
    • The two-pod RWX test shows both pod names in the shared file.
    • vcf addon install list shows cert-manager, Istio and Contour READY, and Contour’s envoy Service has a VIP.
    • The Istio gateway test returns the test page from its VIP.
    • A platform admin logs in with a directory account and kubectl auth can-i '*' '*' -A prints yes; an application user can deploy in their namespace only.
    • Test namespaces are deleted and their VIPs released. All LoadBalancer Services share the same 30-address pool.

    If you are building something similar and run into something this post doesn’t cover, let me know in the comments.