This is a lab build log, written as I went. Everything here was done once, on one set of versions, in my own lab. Where I am reporting what the documentation says instead of what I saw, I say so.
I wanted to answer one question: can a tenant on VMware Cloud Foundation get its own Argo CD, sign in with its VCF Automation identity, and then build a virtual machine, pods and a Kubernetes cluster from a single Git repository? Part 1 gets the platform ready and ends with a working Argo CD instance. Part 2 uses it.
Why bother with GitOps on VCF
Most platform teams I talk to have the same three people in the room. An application developer who raised a ticket for a VM and a cluster days ago. A platform engineer who found that production no longer matches the runbook because somebody changed it by hand. And someone from security asking who changed what, when, and who approved it.
GitOps answers all three with one idea: Git holds the desired state, and a controller keeps the platform matching it. A request is a pull request. Drift is detected and corrected. The audit trail is the commit history. Argo CD is the engine most teams pick for this: it is a CNCF graduated project, it has a UI that application teams actually use, and it takes Helm charts, Kustomize and plain YAML without repackaging.
You can run upstream Argo CD on any Kubernetes cluster. What VCF adds is everything around it:
- Nothing to build first. The instance runs as vSphere Pods on the Supervisor. There is no management cluster to stand up.
- An operator runs it. You pick from VMware-built Argo CD versions; in my lab 3.4.4 and 3.3.12 were the recommended ones.
- Sign-in is already there. The create wizard sets VCF Automation as the identity provider.
- One instance per tenant. Each tenant creates its own Argo CD in its own namespace from the org portal.
- Targets are attached, not hand-registered. You attach namespaces and VKS clusters from a Scope tab, or with a manifest in Git.
- A VM is a manifest too. The same API that takes a Deployment takes a
VirtualMachineand aCluster. That is the subject of Part 2.
Two routes to Argo CD on VCF 9.1.1
| Supervisor Service, from vCenter | VCF Service, through VCF Automation | |
|---|---|---|
| Status | Supported for production | Tech Preview in 9.1.1 |
| Install | vCenter, Supervisor Services | Upload in Partner Services, install from VCF Services |
| Create an instance | Apply an ArgoCD resource in a namespace | Self-service from the org portal |
| Sign-in | Local admin, or OIDC you configure | VCF Automation, configured for you |
| Attach targets | ManagedEntity resources | Scope tab, or a ManagedEntity in Git |
I followed the right-hand column. I did not exercise the left-hand one in this lab; its rows come from the VCF blog announcing the 9.1.1 GitOps service and the product documentation. Underneath, the VCF Service installed an ordinary Supervisor package on my Supervisor, argocd-service.vsphere.vmware.com.
The lab
| Component | Version or name |
|---|---|
| VCF and VCF Automation | 9.1.1 |
| Supervisor | Kubernetes v1.34.9, single control-plane node |
| Networking | NSX VPC with the NSX load balancer |
| Regional Harbor | VCF Service 9.1.1.0.25714559, Harbor 2.15.2 |
| Argo CD Service | 1.2.0, build 25642124 |
| Argo CD instance | 3.4.4 |
| Tenant | Organization org2, project default-project, namespace org2-ns1-mrrtd |
Step 1: Regional Harbor comes first
Before anything else, check whether your Supervisor has the depot service:
$ kubectl -n kube-system get svc depot
Error from server (NotFound): services "depot" not found
That was my starting point. Regional Harbor is what creates this service, and Supervisor Services pull their images through it. Broadcom KB 447329 describes the failure you get if you install a service without it. Harbor is installed from the provider portal under Service Management → VCF Services.
The wizard has three pages: accept the certificates, choose the region, and keep the default properties. The review page is worth reading because it explains the dependency: besides Harbor itself, the install creates overlays that publish Harbor to every Supervisor in the region and create the depot endpoint.
In my lab it took about ten minutes to reach Healthy. One pod, harbor-jobservice, crash-looped twice before settling; it starts before the components it depends on. Afterwards the depot resolves:
$ kubectl -n kube-system get endpoints depot
depot 10.150.4.10:443
Step 2: get the Argo CD Service package
I expected the Argo CD Service to appear in the catalog once Harbor was in, because my lab uses an online depot. It did not.
The documented flow is to download the package with the VCF Download Tool and upload it yourself:
- Generate a Software Depot ID with
vcf-download-tool. - Register it on the Broadcom portal to get an activation code. The activation code replaced the old download token in 9.1 (KB 443647).
- Download the component
VCF_SERVICE_ARGOCDfor version 9.1.1. - In the provider portal, go to Service Management → Partner Services → Upload.
The download tool talks to the Broadcom depot directly with the activation code. It does not go through the VCF Operations depot, which confused me at first.
Step 3: install it, and think about two defaults
The install wizard is the same shape as Harbor’s. Two defaults deserve a decision instead of a click:
- Scope. The default deploys the service to every region, including ones you add later.
keepCRDs: false. An uninstall removes the custom resource definitions, and with them every Argo CD instance your tenants created. Outside a lab I would set this totrue.
Those eight resources fall into three groups. The operator runs on the Supervisor. A UI plugin and a global role add the GitOps pages to tenant org portals. And a ConfigurationHandler bridges the two by handing VCF Automation connection details to the Supervisor side. Remember that last one.
The build that failed
My first attempt used build 25568464. Seven resources went Healthy. The ConfigurationHandler did not:
minorErrorCode: BAD_REQUEST
RDE_RESTRICTED_FIELDS [ $.secrets ]
The operator itself was fine. What was broken was the integration: no org-portal self-service and no single sign-on. Worse, the uninstall then stalled, because the delete path tried to call an entity that had never been created.
Why the uninstall was stuck
The ConfigurationHandler had a deletionTimestamp and one finalizer, services.vcfa.broadcom.com/vcfa-service. Before the controller releases that finalizer it calls a delete behaviour in VCF Automation. But the entity was never created, because the original create had failed, so the delete call went out with an empty ID:
.../cloudapi/1.0.0/entities//behaviors/... 400 Bad Request
behaviorFailureCount: 101
delete backoff cooldown active, requeue after 10m0s
Note the entities//. The controller was trying to delete something that did not exist, and it would have retried forever.
The cleanup, step by step
Everything below was run as root on the VCF Automation node, against its own Kubernetes API, in the prelude namespace. The names are UUIDs from my lab; yours will differ. I took the handler’s name from its YAML in the provider portal. That YAML also names the service account it uses, in callbackServiceAccountName.
CH=de35349b-a9c1-5ead-85a1-e40fa7fba23f # the stuck ConfigurationHandler
SA=8df98ef4-ea51-56be-96f1-a515ac6f75d5 # its callback service account (and Secret of the same name)
1. The ConfigurationHandler. Back it up, then remove its finalizer.
kubectl -n prelude get configurationhandler $CH -o yaml > ch-backup.yaml
kubectl -n prelude patch configurationhandler $CH \
--type=merge -p '{"metadata":{"finalizers":[]}}'
The handler disappeared at once. I had also restarted the vcfa-service-manager deployment just before this. I did not wait to see whether the restart alone would help, and I would not expect it to: the fault was in the data, not in a stuck process.
2. The callback service account. The cleanup then stopped on the next object. These objects are named by UUID, so searching for “argocd” finds nothing; use the label.
kubectl -n prelude get vcfaserviceaccount,secret \
-l services.vcfa.broadcom.com/refName=argocd-service.vsphere.vmware.com
kubectl -n prelude get vcfaserviceaccount $SA \
-o jsonpath='deletionTimestamp={.metadata.deletionTimestamp}{"\n"}finalizers={.metadata.finalizers}{"\n"}'
deletionTimestamp=2026-11-11T09:34:08Z
finalizers=["services.vcfa.broadcom.com/vcfa-service",
"services.vcfa.broadcom.com/configurationhandler.6866f8ff-..."]
Two finalizers, and they are not equal. The second belongs to the handler I had just removed, so nothing would ever clear it. The first is the controller’s own, and it should stay, so that VCF Automation still deletes the account properly on its side. I removed only the orphan:
kubectl -n prelude get vcfaserviceaccount $SA -o yaml > vsa-backup.yaml
# confirm which index holds the configurationhandler finalizer before removing it
kubectl -n prelude get vcfaserviceaccount $SA -o jsonpath='{.metadata.finalizers[1]}{"\n"}'
kubectl -n prelude patch vcfaserviceaccount $SA \
--type=json -p='[{"op":"remove","path":"/metadata/finalizers/1"}]'
3. The Secret. The controller picked the cleanup up again and reported what it was waiting for:
requeue after 10s
Unhealthy
waiting for controller-created secret to be fully deleted
The Secret with the same UUID was held by one more handler-owned finalizer:
kubectl -n prelude get secret $SA \
-o jsonpath='deletionTimestamp={.metadata.deletionTimestamp}{"\n"}finalizers={.metadata.finalizers}{"\n"}'
finalizers=["addons.vcf.vmware.com/configuration-handler"]
kubectl -n prelude get secret $SA -o yaml > secret-backup.yaml
kubectl -n prelude patch secret $SA \
--type=json -p='[{"op":"remove","path":"/metadata/finalizers/0"}]'
The Secret backup contains credentials. Keep it only as long as you need it for a support case.
4. Check that nothing is left. A moment later both objects were gone. The service account going away by itself is the good sign: the controller released its own finalizer.
# on the VCF Automation node: should return nothing
kubectl -n prelude get vcfaserviceaccount,secret | grep 8df98ef4
# on the Supervisor: should return nothing
kubectl get pkgi -A | grep -i argo
kubectl get ns | grep -i argo
kubectl get crd | grep -i argocd
In the provider portal, check Administration → Access Control for leftover Argo CD service accounts or roles, and that the service card is back to Not Installed. Leftovers there are what make a reinstall fail with “already exists” (KB 453499). In my lab everything was gone, and the reinstall in the next step went through. Keep the three backup files and a note of the order of events; that is what a support case will ask for.
The second attempt used build 25642124 and went Healthy within minutes.
I cannot yet say the first build is bad. Both builds were uploaded to VCF Automation at the time, and I have not repeated the install of 25568464 on its own. Until I do, my advice is simply to upload one build, and make it 25642124.
On the Supervisor, the result looks like this:
$ kubectl get pkgi -A | grep argo
svc-argocd-service... 1.2.0-25642124 Reconcile succeeded
$ kubectl get crd | grep argocd
argocds · argocdversions · managedentities · entitymanagementpolicies
Step 4: a tenant creates its own instance
Everything from here is done as a tenant user in the org portal. Under Build & Deploy → Services → GitOps there is now a button to create an Argo CD instance. That page existing at all is the proof that the install wired the tenant side correctly.
I named the instance org2-argo, chose the project and namespace, and picked version 3.4.4. The wizard warns that project admins and project advanced users become admins of this Argo CD. Take that seriously: project membership is your security boundary.
The RBAC page holds ordinary Argo CD policy. If you have not read one before:
g, org-admin, role:admin # g assigns a user or group to a role
p, role:app-deployer, applications, sync, default/*, allow # p says what a role may do
A few minutes after submitting, the instance showed Success, with seven pods in the tenant namespace and a load-balancer address of its own.
Step 5: sign in with VCF Automation
This button is what the failed ConfigurationHandler would have cost me. It worked on the first try. The User Info page shows what Argo CD received from VCF Automation, and those are the values to use in RBAC.
One thing I have not settled: my test user is literally named org-admin, so I cannot tell whether the group Argo CD sees comes from the VCF Automation role or from the username. I need to repeat this with a differently named user.
Step 6: give it somewhere to deploy
A new instance has no destinations, not even its own namespace. You add them from the Scope tab in the org portal. I attached the namespace org2-ns1-mrrtd.
Look at the URL in that last screenshot. Argo CD does not hold a kubeconfig for the Supervisor. It reaches the namespace through the VCF Automation Kubernetes proxy, using a service account the platform created for the instance. So there are two separate gates: who may use Argo CD, and what Argo CD itself may change.
What I would tell a customer so far
Three observations from this half. Each was seen once and still needs reproducing.
- Install Regional Harbor before any Supervisor Service. The depot service did not exist until Harbor created it.
- The online depot did not bring the Argo CD Service. I had to download and upload it.
- Build 25568464 failed for me and 25642124 did not. Whether that was the build, or having two builds uploaded, is still open.
At this point a tenant has a private Argo CD, single sign-on, and one namespace attached. In Part 2 I point it at a Git repository and let it build a virtual machine, vSphere Pods, a VKS cluster and the application inside that cluster, then deploy a twelve-service shop from the Argo CD GUI and fix a real fault along the way.

