GitOps as a Service on VCF 9.1.1 (Part 2: One Repo, Three Runtimes)

In Part 1 I installed Regional Harbor and the Argo CD Service on VCF 9.1.1, and created an Argo CD instance with VCF Automation sign-in. Part 2 puts it to work.

The idea is simple to state. A DevOps user on VCF writes Kubernetes manifests, pushes them to Git, and gets a virtual machine, some pods, a whole Kubernetes cluster and an application running inside that cluster, without opening vCenter or NSX Manager and without holding a kubeconfig for the new cluster.

A VM, vSphere Pods, a Kubernetes cluster and an app, each as a Kubernetes manifest on one API
Four kinds of thing, one API, one Git workflow.

This post walks through how it fits together, the details worth getting right, and then a second application deployed and tuned entirely from the Argo CD GUI.

The application and where each tier runs

I used Yelb, a small restaurant-voting app that splits cleanly into tiers:

TierRuns asCreated from
Database (Postgres)A VM Service virtual machine, Ubuntu 24.04, built by cloud-initkind: VirtualMachine
App server and RedisvSphere Pods on the Supervisorkind: Deployment
UIPods inside a new VKS cluster, yelb-vkskind: Cluster, then kind: Deployment

The repository is public: github.com/avnish80/vcf-gitops-yelb.

apps/                     five pointers for Argo CD
  10-yelb-db.yaml
  20-yelb-appserver.yaml
  30-yelb-vks-cluster.yaml
  40-yelb-registration.yaml
  50-yelb-ui-appset.yaml
infra/
  db-vm/                  VirtualMachine + cloud-init
  appserver/              Deployments + LoadBalancer Service
  vks-cluster/            Cluster
  registration/           ManagedEntity
workloads/
  yelb-ui/                Deployment + LoadBalancer Service

The only command run by hand is kubectl apply -f apps/. That creates four Argo CD Applications and one ApplicationSet. No infrastructure exists at that point; Argo CD pulls the repo and the VCF controllers build the rest.

Argo CD showing the Yelb applications just created
Five pointers applied. Argo CD does the rest.

The manifests are small

This is the whole virtual machine:

apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachine
metadata: { name: yelb-db }
spec:
  className: best-effort-small
  image:
    kind: ClusterVirtualMachineImage
    name: vmi-5ab37e711e9121b34
  storageClass: vsan-default-storage-policy
  bootstrap:
    cloudInit:
      rawCloudConfig: { name: yelb-db-cloudinit, key: user-data }

And this is the whole cluster, trimmed:

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata: { name: yelb-vks }
spec:
  topology:
    classRef: { name: builtin-generic-v3.7.0, namespace: vmware-system-vks-public }
    version: v1.36.2+vmware.2
    controlPlane: { replicas: 1 }
    workers:
      machineDeployments:
        - { class: node-pool, name: np-1, replicas: 1 }

Three of these kinds come from VCF: VirtualMachine, Cluster with the VKS ClusterClass, and ManagedEntity. That is the point: the platform’s own resources are Kubernetes objects, so they fit the same workflow as a Deployment. The VM class, image and storage policy names are the ones your platform team has assigned to the namespace.

The part worth showing live: a cluster that registers itself

Git never contains the new cluster’s address or credentials. Two small pieces connect it. A ManagedEntity points at the Cluster and asks for a label:

kind: ManagedEntity
spec:
  clusterName: yelb-vks
  targetRef:
    apiGroup: cluster.x-k8s.io
    kind: Cluster
    name: yelb-vks
    namespace: org2-ns1-mrrtd
    project: default-project
  secretLabels:
    app: yelb

The Argo CD Service sees it, registers the cluster in Argo CD, and puts app: yelb on the cluster secret. An ordinary upstream ApplicationSet then deploys the UI to every cluster carrying that label:

kind: ApplicationSet
spec:
  generators:
    - clusters:
        selector:
          matchLabels: { app: yelb }
  template:
    spec:
      source: { path: workloads/yelb-ui }
      destination: { server: '{{server}}', namespace: yelb }
Five steps from a Cluster in Git to an app deployed in it
Nobody attached the cluster. A manifest did.

Seven things to get right

The repository already has these built in. If you adapt it to your own environment, these are the details that matter.

  1. Assign the storage policy to the namespace. A policy that exists on the Supervisor is not enough. The VM and the Cluster are admitted only with a storage class that is assigned to the namespace they are created in.
  2. Give vSphere Pods an explicit CPU request. A vSphere Pod’s CPU request becomes a resource-pool reservation, and with no request set it takes the value of the limit. requests: { cpu: "0" } keeps small pods from reserving capacity they do not need.
  3. After changing a manifest, start a fresh sync. An automated sync that is retrying stays on the commit it started with. Terminate the running operation and sync again to pick up the new commit.
  4. Connect the tiers with hostAliases. Argo CD 3.x leaves EndpointSlices alone by default, so they are not a good bridge between runtimes. A hostAliases entry in the Deployment is plain, visible and managed like everything else.
  5. Reach the database on the VM’s own address. The app server in a vSphere Pod talks to Postgres on the VM’s IP. The address lives in Git, so update it if you rebuild the VM.
  6. Check the ManagedEntity once the cluster is ready. If it was created before the cluster finished building, touch it with an annotation and it reconciles within a minute.
  7. Match the database’s authentication to the app’s client. Yelb’s database client predates SCRAM, the Postgres 16 default, so the cloud-init sets md5 for this app.

The last one shows the nicest property of the whole setup. To apply a new cloud-init, commit the change and run kubectl delete vm yelb-db. Argo CD notices the VM is missing and creates a new one from Git, with the new cloud-init. Rebuilding a server becomes a one-line operation.

The result

All Yelb applications Healthy and Synced
Everything Healthy and Synced.
The Yelb UI served from the VKS cluster
The UI in VKS, calling the app server in a vSphere Pod, reading restaurants from Postgres on the VM.
Traffic path from browser through two NSX load balancers to the VM
Who talks to whom.

Day 2: more capacity is a commit

Next I wanted a bigger worker for a larger application. That was a few lines added to cluster.yaml and a push:

        - class: node-pool
          name: np-apps
          replicas: 1
          variables:
            overrides:
              - name: vmClass
                value: best-effort-large      # 4 vCPU / 16 GiB

Argo CD synced the change and VKS added the node pool. The Cluster carries a Prune=false annotation, so Git can change it but removing the file would not delete it.

VCF Automation listing three VKS clusters as Available
All three clusters in the namespace Available.

A twelve-service shop from the Argo CD GUI

Not everything needs a repository on day one. I deployed Google’s Online Boutique demo, twelve services, entirely from Argo CD’s New App form: no commit, no kubectl, no Helm CLI.

Repository URL   us-docker.pkg.dev/online-boutique-ci/charts    (type: Helm)
Chart            onlineboutique   0.10.7
Cluster          yelb-vks         (picked by name)
Namespace        boutique         (auto-created)
Overrides        seccompProfile.enable = true
                 frontend.cymbalBranding = true
New App form with Helm source and destination cluster yelb-vks
The cluster that registered itself is simply there to pick.
online-boutique application Synced and Progressing
Synced within seconds; Progressing while images pull.

NSX handed out a new load-balancer address, and the shop was up.

The Online Boutique shop home page
Served from yelb-vks through an NSX load balancer.

Tune it and change it from the same GUI

Two changes followed, both made under Details → Parameters → Edit in Argo CD.

The first was tuning. One of the twelve services, emailservice, needed more CPU than the chart’s default 200m limit to start inside its liveness window. I raised emailService.resources.limits.cpu to 1, and Argo CD replaced that one pod.

The second was a feature flag. Setting frontend.cymbalBranding to false rolled only the frontend and changed the shop’s branding.

Editing Helm parameters in the Argo CD GUI
Editing a Helm parameter in place.
All twelve pods Running
Two edits, two pods replaced. The other ten never restarted.
The shop with Online Boutique branding after the parameter change
After the branding change. The cart kept its item, because the cart lives in Redis.
Order complete page
An order, end to end.
Git or GUI? Yelb lives entirely in Git. The Online Boutique application and its overrides live in Argo CD. The GUI is fast for an experiment; for anything that should last, move the overrides into a values file in a repository.

What this adds up to

One repository built a VM, vSphere Pods and a VKS cluster. The cluster registered itself, and the application arrived without anyone attaching anything. A second application went in from a browser and was tuned from the same place.

For a platform team, that is a private, declarative slice of the private cloud for each team. Requests become pull requests, drift is corrected from Git, and every change has an author and a diff. For the teams using it, virtual machines, pods and clusters all arrive through the one workflow they already know.

The repository is public. Clone it, change the names to your own namespace, image and storage policy, and try it.

Comments

Leave a Reply

More posts