Tag: Multi-tenancy

  • GitOps as a Service on VCF 9.1.1 (Part 1: Installing Argo CD)

    GitOps as a Service on VCF 9.1.1 (Part 1: Installing Argo CD)

    This is a lab build log, written as I went. Everything here was done once, on one set of versions, in my own lab. Where I am reporting what the documentation says instead of what I saw, I say so.

    I wanted to answer one question: can a tenant on VMware Cloud Foundation get its own Argo CD, sign in with its VCF Automation identity, and then build a virtual machine, pods and a Kubernetes cluster from a single Git repository? Part 1 gets the platform ready and ends with a working Argo CD instance. Part 2 uses it.

    Why bother with GitOps on VCF

    Most platform teams I talk to have the same three people in the room. An application developer who raised a ticket for a VM and a cluster days ago. A platform engineer who found that production no longer matches the runbook because somebody changed it by hand. And someone from security asking who changed what, when, and who approved it.

    GitOps answers all three with one idea: Git holds the desired state, and a controller keeps the platform matching it. A request is a pull request. Drift is detected and corrected. The audit trail is the commit history. Argo CD is the engine most teams pick for this: it is a CNCF graduated project, it has a UI that application teams actually use, and it takes Helm charts, Kustomize and plain YAML without repackaging.

    Three people and their problems without GitOps, and what changes with Git as the source of truth
    The problem GitOps solves, and why Argo CD.

    You can run upstream Argo CD on any Kubernetes cluster. What VCF adds is everything around it:

    • Nothing to build first. The instance runs as vSphere Pods on the Supervisor. There is no management cluster to stand up.
    • An operator runs it. You pick from VMware-built Argo CD versions; in my lab 3.4.4 and 3.3.12 were the recommended ones.
    • Sign-in is already there. The create wizard sets VCF Automation as the identity provider.
    • One instance per tenant. Each tenant creates its own Argo CD in its own namespace from the org portal.
    • Targets are attached, not hand-registered. You attach namespaces and VKS clusters from a Scope tab, or with a manifest in Git.
    • A VM is a manifest too. The same API that takes a Deployment takes a VirtualMachine and a Cluster. That is the subject of Part 2.
    Argo CD assembled yourself compared with the Argo CD Service on VCF 9.1.1
    What you would otherwise build and keep working yourself.

    Two routes to Argo CD on VCF 9.1.1

    Supervisor Service, from vCenterVCF Service, through VCF Automation
    StatusSupported for productionTech Preview in 9.1.1
    InstallvCenter, Supervisor ServicesUpload in Partner Services, install from VCF Services
    Create an instanceApply an ArgoCD resource in a namespaceSelf-service from the org portal
    Sign-inLocal admin, or OIDC you configureVCF Automation, configured for you
    Attach targetsManagedEntity resourcesScope tab, or a ManagedEntity in Git

    I followed the right-hand column. I did not exercise the left-hand one in this lab; its rows come from the VCF blog announcing the 9.1.1 GitOps service and the product documentation. Underneath, the VCF Service installed an ordinary Supervisor package on my Supervisor, argocd-service.vsphere.vmware.com.

    The lab

    ComponentVersion or name
    VCF and VCF Automation9.1.1
    SupervisorKubernetes v1.34.9, single control-plane node
    NetworkingNSX VPC with the NSX load balancer
    Regional HarborVCF Service 9.1.1.0.25714559, Harbor 2.15.2
    Argo CD Service1.2.0, build 25642124
    Argo CD instance3.4.4
    TenantOrganization org2, project default-project, namespace org2-ns1-mrrtd

    Step 1: Regional Harbor comes first

    Before anything else, check whether your Supervisor has the depot service:

    $ kubectl -n kube-system get svc depot
    Error from server (NotFound): services "depot" not found

    That was my starting point. Regional Harbor is what creates this service, and Supervisor Services pull their images through it. Broadcom KB 447329 describes the failure you get if you install a service without it. Harbor is installed from the provider portal under Service Management → VCF Services.

    VCF Services page with Harbor listed as not installed
    Harbor under VCF Services, not yet installed.

    The wizard has three pages: accept the certificates, choose the region, and keep the default properties. The review page is worth reading because it explains the dependency: besides Harbor itself, the install creates overlays that publish Harbor to every Supervisor in the region and create the depot endpoint.

    Harbor review page listing the resources to be created
    Eleven resources, including the overlays that wire each Supervisor to this Harbor.

    In my lab it took about ten minutes to reach Healthy. One pod, harbor-jobservice, crash-looped twice before settling; it starts before the components it depends on. Afterwards the depot resolves:

    $ kubectl -n kube-system get endpoints depot
    depot   10.150.4.10:443
    Harbor service showing Healthy with all components green
    Harbor Healthy.

    Step 2: get the Argo CD Service package

    I expected the Argo CD Service to appear in the catalog once Harbor was in, because my lab uses an online depot. It did not.

    VCF Services page with no Argo CD card
    Harbor was listed. Argo CD was not.

    The documented flow is to download the package with the VCF Download Tool and upload it yourself:

    1. Generate a Software Depot ID with vcf-download-tool.
    2. Register it on the Broadcom portal to get an activation code. The activation code replaced the old download token in 9.1 (KB 443647).
    3. Download the component VCF_SERVICE_ARGOCD for version 9.1.1.
    4. In the provider portal, go to Service Management → Partner Services → Upload.

    The download tool talks to the Broadcom depot directly with the activation code. It does not go through the VCF Operations depot, which confused me at first.

    Partner Services upload dialog reading the Argo CD package
    Uploading the package. It is about 1 GB, so keep the tab open.
    ArgoCD Service card under VCF Services
    After the upload, the card appears under VCF Services.

    Step 3: install it, and think about two defaults

    The install wizard is the same shape as Harbor’s. Two defaults deserve a decision instead of a click:

    • Scope. The default deploys the service to every region, including ones you add later.
    • keepCRDs: false. An uninstall removes the custom resource definitions, and with them every Argo CD instance your tenants created. Outside a lab I would set this to true.
    Argo CD Service review page listing eight resources
    Eight resources: the operator on the Supervisor, and the pieces that wire it into VCF Automation.

    Those eight resources fall into three groups. The operator runs on the Supervisor. A UI plugin and a global role add the GitOps pages to tenant org portals. And a ConfigurationHandler bridges the two by handing VCF Automation connection details to the Supervisor side. Remember that last one.

    The build that failed

    My first attempt used build 25568464. Seven resources went Healthy. The ConfigurationHandler did not:

    minorErrorCode: BAD_REQUEST
    RDE_RESTRICTED_FIELDS [ $.secrets ]
    ConfigurationHandler showing Unhealthy
    Seven Healthy, one not.

    The operator itself was fine. What was broken was the integration: no org-portal self-service and no single sign-on. Worse, the uninstall then stalled, because the delete path tried to call an entity that had never been created.

    What I did, and why you should not copy it blindly. I cleared the stuck uninstall by removing orphaned finalizers on three VCF Automation objects, after backing each one up. These are VCF Automation internals and this is not a documented procedure. In a lab that is a reasonable risk. Anywhere else, open a support case first. The exact steps follow so you can recognise the situation, and so support can see what was done.

    Why the uninstall was stuck

    The ConfigurationHandler had a deletionTimestamp and one finalizer, services.vcfa.broadcom.com/vcfa-service. Before the controller releases that finalizer it calls a delete behaviour in VCF Automation. But the entity was never created, because the original create had failed, so the delete call went out with an empty ID:

    .../cloudapi/1.0.0/entities//behaviors/...      400 Bad Request
    behaviorFailureCount: 101
    delete backoff cooldown active, requeue after 10m0s

    Note the entities//. The controller was trying to delete something that did not exist, and it would have retried forever.

    The cleanup, step by step

    Everything below was run as root on the VCF Automation node, against its own Kubernetes API, in the prelude namespace. The names are UUIDs from my lab; yours will differ. I took the handler’s name from its YAML in the provider portal. That YAML also names the service account it uses, in callbackServiceAccountName.

    CH=de35349b-a9c1-5ead-85a1-e40fa7fba23f     # the stuck ConfigurationHandler
    SA=8df98ef4-ea51-56be-96f1-a515ac6f75d5     # its callback service account (and Secret of the same name)

    1. The ConfigurationHandler. Back it up, then remove its finalizer.

    kubectl -n prelude get configurationhandler $CH -o yaml > ch-backup.yaml
    
    kubectl -n prelude patch configurationhandler $CH \
      --type=merge -p '{"metadata":{"finalizers":[]}}'

    The handler disappeared at once. I had also restarted the vcfa-service-manager deployment just before this. I did not wait to see whether the restart alone would help, and I would not expect it to: the fault was in the data, not in a stuck process.

    2. The callback service account. The cleanup then stopped on the next object. These objects are named by UUID, so searching for “argocd” finds nothing; use the label.

    kubectl -n prelude get vcfaserviceaccount,secret \
      -l services.vcfa.broadcom.com/refName=argocd-service.vsphere.vmware.com
    
    kubectl -n prelude get vcfaserviceaccount $SA \
      -o jsonpath='deletionTimestamp={.metadata.deletionTimestamp}{"\n"}finalizers={.metadata.finalizers}{"\n"}'
    
    deletionTimestamp=2026-11-11T09:34:08Z
    finalizers=["services.vcfa.broadcom.com/vcfa-service",
                "services.vcfa.broadcom.com/configurationhandler.6866f8ff-..."]

    Two finalizers, and they are not equal. The second belongs to the handler I had just removed, so nothing would ever clear it. The first is the controller’s own, and it should stay, so that VCF Automation still deletes the account properly on its side. I removed only the orphan:

    kubectl -n prelude get vcfaserviceaccount $SA -o yaml > vsa-backup.yaml
    
    # confirm which index holds the configurationhandler finalizer before removing it
    kubectl -n prelude get vcfaserviceaccount $SA -o jsonpath='{.metadata.finalizers[1]}{"\n"}'
    
    kubectl -n prelude patch vcfaserviceaccount $SA \
      --type=json -p='[{"op":"remove","path":"/metadata/finalizers/1"}]'

    3. The Secret. The controller picked the cleanup up again and reported what it was waiting for:

    requeue after 10s
    Unhealthy
    waiting for controller-created secret to be fully deleted

    The Secret with the same UUID was held by one more handler-owned finalizer:

    kubectl -n prelude get secret $SA \
      -o jsonpath='deletionTimestamp={.metadata.deletionTimestamp}{"\n"}finalizers={.metadata.finalizers}{"\n"}'
    
    finalizers=["addons.vcf.vmware.com/configuration-handler"]
    
    kubectl -n prelude get secret $SA -o yaml > secret-backup.yaml
    
    kubectl -n prelude patch secret $SA \
      --type=json -p='[{"op":"remove","path":"/metadata/finalizers/0"}]'

    The Secret backup contains credentials. Keep it only as long as you need it for a support case.

    4. Check that nothing is left. A moment later both objects were gone. The service account going away by itself is the good sign: the controller released its own finalizer.

    # on the VCF Automation node: should return nothing
    kubectl -n prelude get vcfaserviceaccount,secret | grep 8df98ef4
    
    # on the Supervisor: should return nothing
    kubectl get pkgi -A | grep -i argo
    kubectl get ns | grep -i argo
    kubectl get crd | grep -i argocd

    In the provider portal, check Administration → Access Control for leftover Argo CD service accounts or roles, and that the service card is back to Not Installed. Leftovers there are what make a reinstall fail with “already exists” (KB 453499). In my lab everything was gone, and the reinstall in the next step went through. Keep the three backup files and a note of the order of events; that is what a support case will ask for.

    The second attempt used build 25642124 and went Healthy within minutes.

    All eight resources Healthy on build 25642124
    Build 25642124: all eight Healthy.

    I cannot yet say the first build is bad. Both builds were uploaded to VCF Automation at the time, and I have not repeated the install of 25568464 on its own. Until I do, my advice is simply to upload one build, and make it 25642124.

    On the Supervisor, the result looks like this:

    $ kubectl get pkgi -A | grep argo
    svc-argocd-service...   1.2.0-25642124   Reconcile succeeded
    
    $ kubectl get crd | grep argocd
    argocds · argocdversions · managedentities · entitymanagementpolicies

    Step 4: a tenant creates its own instance

    Everything from here is done as a tenant user in the org portal. Under Build & Deploy → Services → GitOps there is now a button to create an Argo CD instance. That page existing at all is the proof that the install wired the tenant side correctly.

    Org portal GitOps page with Create Argo CD Instance
    The GitOps page in the org portal.

    I named the instance org2-argo, chose the project and namespace, and picked version 3.4.4. The wizard warns that project admins and project advanced users become admins of this Argo CD. Take that seriously: project membership is your security boundary.

    OIDC and RBAC page of the create wizard
    VCF Automation is the identity provider by default.

    The RBAC page holds ordinary Argo CD policy. If you have not read one before:

    g, org-admin, role:admin          # g assigns a user or group to a role
    p, role:app-deployer, applications, sync, default/*, allow   # p says what a role may do

    A few minutes after submitting, the instance showed Success, with seven pods in the tenant namespace and a load-balancer address of its own.

    Instance status Success with pods running
    The instance is up.
    One Argo CD instance per tenant
    The provider installs once. Each tenant runs its own Argo CD.

    Step 5: sign in with VCF Automation

    Argo CD login page with a Log in via VCF Automation button
    The button I was hoping for.

    This button is what the failed ConfigurationHandler would have cost me. It worked on the first try. The User Info page shows what Argo CD received from VCF Automation, and those are the values to use in RBAC.

    Argo CD after signing in through VCF Automation
    Signed in. No applications yet.

    One thing I have not settled: my test user is literally named org-admin, so I cannot tell whether the group Argo CD sees comes from the VCF Automation role or from the username. I need to repeat this with a differently named user.

    Step 6: give it somewhere to deploy

    A new instance has no destinations, not even its own namespace. You add them from the Scope tab in the org portal. I attached the namespace org2-ns1-mrrtd.

    Attach Namespace dialog
    Any namespace you can reach in the region, from any project.
    Argo CD Settings, Clusters, showing the attached namespace
    The namespace appears in Argo CD as a destination behind the VCF Automation proxy.

    Look at the URL in that last screenshot. Argo CD does not hold a kubeconfig for the Supervisor. It reaches the namespace through the VCF Automation Kubernetes proxy, using a service account the platform created for the instance. So there are two separate gates: who may use Argo CD, and what Argo CD itself may change.

    Two access paths: signing in to Argo CD, and Argo CD changing things
    An Argo CD admin still cannot make Argo CD create something its service account is not allowed to.

    What I would tell a customer so far

    Three observations from this half. Each was seen once and still needs reproducing.

    1. Install Regional Harbor before any Supervisor Service. The depot service did not exist until Harbor created it.
    2. The online depot did not bring the Argo CD Service. I had to download and upload it.
    3. Build 25568464 failed for me and 25642124 did not. Whether that was the build, or having two builds uploaded, is still open.

    At this point a tenant has a private Argo CD, single sign-on, and one namespace attached. In Part 2 I point it at a Git repository and let it build a virtual machine, vSphere Pods, a VKS cluster and the application inside that cluster, then deploy a twelve-service shop from the Argo CD GUI and fix a real fault along the way.