Tag: Kubernetes

If you administer vSphere, you already know how to build a VM. This post shows you how to build the same VM in VMware Cloud Foundation Automation using nothing but a command line — no UI clicks, no cloud template, no prior Kubernetes experience assumed. It takes about less then 10 minutes, and by the end you’ll have a running web server with a load-balanced IP that answers from your browser.

I’m deliberately keeping this simple. There is a longer, deeper reference post for the full API surface, but this one is the on-ramp: the shortest honest path from “I’ve never used the VCF Automation API” to “I just built something with it.”

Why bother, when the UI works fine?

Fair question, and I’d give three honest answers.

Repeatability. A VM you clicked into existence is a VM nobody can rebuild exactly. A VM defined in a YAML file is one you can commit, review, and recreate identically a year later.

It’s where the platform is going. In VCF 9, the consumption layer is a Kubernetes-style API. The UI is a client of that API, not the other way round. Learning the objects underneath means the UI stops being magic.

It’s the foundation for automation. Everything you’d want later — GitOps, pipelines, self-service catalogs — is built on these same objects. This is the first rung.

None of that means you should stop using the UI. I use both. But knowing what’s underneath makes you considerably harder to stump.

Before you start

Everything below is run from PowerShell on Windows. It works identically on Linux and macOS; only the quoting changes.

Step 1 — Log in

First, store your token in a variable. Note the quotes — without them PowerShell tries to execute the token as a command, which produces a confusing error.

$vcfa_token = "your-api-token-here"

Now create a context. This is the CLI’s word for “a saved connection to an environment.”

vcf context create org2-ctgw `
  --endpoint https://auto-a.site-a.vcf.lab `
  --type cci `
  --auth-type basic `
  --tenant-name Org2-CTGW `
  --api-token $vcfa_token `
  --insecure-skip-tls-verify

Substitute your own endpoint and tenant name. The backtick is PowerShell’s line-continuation character — or just put it all on one line. Use --insecure-skip-tls-verify only in a lab; in production supply your CA certificate with --ca-certificate instead.

That output is worth reading carefully, because it explains the whole model. You get an organization context (org2-ctgw) and a namespace context for each namespace you can use (org2-ctgw:org2-ns1-4hm32:default-project).

Step 2 — Create a project

A project is the boundary you’d hand to a team. Save this as project.yml:

apiVersion: project.cci.vmware.com/v1alpha2
kind: Project
metadata:
  name: team-phoenix
spec:
  description: Phoenix dev team

kubectl --context org2-ctgw create -f project.yml
# project.project.cci.vmware.com/team-phoenix created

Step 3 — Create a namespace

The project is just a boundary. A Supervisor Namespace is the thing with actual capacity — CPU, memory, storage, a network — where your VMs will live. Think of it as the resource pool plus the network plus the quota, expressed as one object.

You need four pieces of information, all of which you can read off an existing namespace or get from whoever set up your environment:

apiVersion: infrastructure.cci.vmware.com/v1alpha3
kind: SupervisorNamespace
metadata:
  generateName: phoenix-ns-
  namespace: team-phoenix
spec:
  regionName: region-west
  className: small
  vpcName: default-region-west
  classConfigOverrides:
    storageClasses:
    - name: vsan-default-storage-policy
      limit: 204800Mi
    zones:
    - name: z-wld-a
      cpuLimit: 15000M
      cpuReservation: 0M
      memoryLimit: 15000Mi
      memoryReservation: 0Mi

kubectl --context org2-ctgw create -f namespace.yml
kubectl --context org2-ctgw get supervisornamespaces -n team-phoenix

Wait for STATUS: Created. Then refresh your contexts so the new namespace appears, and switch into it:

vcf context use org2-ctgw
kubectl config get-contexts
kubectl config use-context org2-ctgw:phoenix-ns-kj9g6:team-phoenix

You can confirm the same thing in the UI, which is a reassuring way to check your work while you’re learning:

Step 4 — Create a VM

Now the part that will feel familiar. We need three things: a size, an image, and a storage policy. Ask the namespace what it has:

kubectl get virtualmachineclasses     # the sizes available to you
kubectl get virtualmachineimages      # images from your content library
kubectl get clustervirtualmachineimages | Select-String "ubuntu"

Here’s the VM, together with a cloud-init secret that creates a login and installs nginx. Save both in one file:

apiVersion: v1
kind: Secret
metadata:
  name: web01-bootstrap
stringData:
  user-data: |
    #cloud-config
    users:
    - name: ops
      sudo: ALL=(ALL) NOPASSWD:ALL
      shell: /bin/bash
      ssh_authorized_keys:
      - ssh-ed25519 AAAA...your-key-here
    packages: [nginx]
    runcmd:
    - systemctl enable --now nginx
---
apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachine
metadata:
  name: web01
  labels:
    app: web
spec:
  className: best-effort-small
  imageName: vmi-5dcfa60c7cc579296
  storageClass: vsan-default-storage-policy
  powerState: PoweredOn
  bootstrap:
    cloudInit:
      rawCloudConfig:
        name: web01-bootstrap
        key: user-data

kubectl create -f vm.yml

Watch it come up. The VM will report an IP address once VMware Tools starts reporting:

kubectl get vm web01
kubectl get vm web01 -o jsonpath='{.status.network.primaryIP4}'
# 172.30.0.66

Notice what you didn’t have to specify: no port group, no IP address, no datastore. The VM picked up an address from the namespace’s default network automatically and landed on storage matching the policy you named. That’s the platform doing the placement work, exactly as it does for a Kubernetes pod.

Step 5 — Publish it

A VirtualMachineService puts a load-balanced IP in front of one or more VMs, selected by label. Our VM carries app: web, so the service will find it.

apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachineService
metadata:
  name: web-lb
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP

Step 6 — Confirm it works

kubectl get svc
# web-lb   LoadBalancer   172.29.255.24   40.60.0.4   80/TCP   2m31s

kubectl get endpoints
# web-lb   172.30.0.66:80   2m34s

curl.exe http://40.60.0.4

That’s the full path working. The load balancer address is allocated from the NSX VPC, the endpoint is your VM’s own IP, and nginx — installed by cloud-init at first boot — is serving. Nothing in that sequence involved the UI.

Step 7 — Build a Kubernetes cluster

Here’s the part that surprises people: a Kubernetes cluster is just another object in the same namespace you’ve been working in. Same context, same kubectl create, same mental model. You describe the cluster you want and the platform builds the nodes.

You need to know two things first — and both are places where a sensible-looking manifest fails.

Where the cluster template lives. Clusters are built from a ClusterClass, which is a template your platform team maintains. Crucially, it does not live in your namespace:

kubectl get clusterclasses
# NAME                     VARIABLES READY   AGE
# builtin-generic-v3.1.0   True              164m
# builtin-generic-v3.2.0   True              164m
# builtin-generic-v3.3.0   True              164m

That list is not the whole story. The class you actually want may live in a shared system namespace and won’t appear here at all — you reference it explicitly by name and namespace.

Which Kubernetes version you can use. Ask the platform rather than guessing:

kubectl get kubernetesreleases

You’ll get a long list. Only the rows showing both READY: True and COMPATIBLE: True are usable. Pick one of those.

Now the cluster itself — one control plane node, one worker, deliberately small:

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: phoenix-k8s
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.156.0/20"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
    serviceDomain: cluster.local
  topology:
    classRef:
      name: builtin-generic-v3.6.0
      namespace: vmware-system-vks-public
    version: v1.35.5---vmware.1-vkr.1
    variables:
    - name: vmClass
      value: best-effort-medium
    - name: storageClass
      value: vsan-default-storage-policy
    controlPlane:
      replicas: 1
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-96cb71ec140d192ad
    workers:
      machineDeployments:
      - class: node-pool
        name: phoenix-k8s-np-1
        replicas: 1
        metadata:
          annotations:
            run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-96cb71ec140d192ad

kubectl create -f cluster.yml
kubectl get cluster phoenix-k8s -w

Provisioning takes five to six minutes for this size. PHASE: Provisioned appears early and only means the objects were created — the column to watch is AVAILABLE, which flips to True once the control plane is genuinely up. kubectl get machines gives you the per-node view while you wait.

Step 8 — Get onto the cluster and run something

Once AVAILABLE is True, download the cluster’s kubeconfig from the VCF Automation UI (the cluster’s detail page offers it) and point kubectl at it:

kubectl --kubeconfig .\phoenix-k8s.yaml get nodes
# NAME                        STATUS   ROLES           AGE   VERSION
# phoenix-k8s-...             Ready    control-plane   12m   v1.35.5+vmware.1
# phoenix-k8s-...-np-1-...    Ready    <none>          9m    v1.35.5+vmware.1

You’re now a cluster administrator on an ordinary, conformant Kubernetes cluster. Everything you know about Kubernetes applies from here.

With one important exception, which is worth meeting deliberately rather than by accident.

Your first deployment will probably be rejected — and that’s correct

Try the obvious thing:

kubectl --kubeconfig .\phoenix-k8s.yaml create deployment web --image=nginx --replicas=2

The deployment is created, but no pods ever appear. Looking at the ReplicaSet tells you why:

pods "web-..." is forbidden: violates PodSecurity "restricted:latest":
  allowPrivilegeEscalation != false
  unrestricted capabilities (must set capabilities.drop=["ALL"])
  runAsNonRoot != true
  seccompProfile (must set seccompProfile.type to "RuntimeDefault")

This is Pod Security Admission enforcing the restricted profile, and it’s switched on by default. A stock container image that runs as root is refused at creation — not warned about, refused. For a vSphere admin used to “deploy first, harden later,” this is a genuine change of habit, and a welcome one: the cluster will not run a privileged workload just because you forgot.

The fix is to declare what the workload actually needs. Each field below answers one of the four complaints, and the image is swapped for a non-root variant that listens on 8080 rather than binding port 80 as root:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: nginx
        image: nginxinc/nginx-unprivileged:stable
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]

kubectl --kubeconfig .\phoenix-k8s.yaml apply -f web.yml
kubectl --kubeconfig .\phoenix-k8s.yaml expose deployment web --port=80 --target-port=8080 --type=LoadBalancer
kubectl --kubeconfig .\phoenix-k8s.yaml get svc web

The external IP is allocated by the same NSX load balancer that served your VM in Step 5 — usually within seconds. Confirm there are backends behind it:

kubectl --kubeconfig .\phoenix-k8s.yaml get endpointslices -l kubernetes.io/service-name=web
# ADDRESSTYPE   PORTS   ENDPOINTS
# IPv4          8080    192.168.147.3,192.168.145.12

The five things that will trip you up

These cost me time so they don’t have to cost you any.

1. Use kubectl create, never apply. The API rejects the annotation that apply depends on.
2. Context names are three parts. <org>:<namespace>:<project>, not just the namespace.
3. Know which level you’re at. “The server doesn’t have a resource type” almost always means you’re at the org context asking a workload question, or vice versa.
4. Tokens expire mid-session. A sudden Unauthorized, or a strange complaint about downloading the OpenAPI schema, usually just means your session lapsed. vcf context use <org> re-authenticates.
5. Check the API version against your build. Published examples lag. On the environment used here, namespaces are served at v1alpha3 and VMs at v1alpha5. The most reliable way to learn the current shape of any object is to dump a working one: kubectl get vm <name> -o yaml. Copy the structure, change the values.

That last one is worth internalising as a general habit. Rather than hunting for documentation that matches your exact release, clone something that already works in your own environment. It is always correct, by definition.

Where to go next

You now have the pattern for everything else on this platform. The same two-level model, the same create workflow, and the same “clone a working object” trick apply to Kubernetes clusters, persistent volumes, NSX security policies, and micro-segmentation rules — all of which are just more objects in the same API.

A few natural next steps, in the order I’d take them:

• Delete and rebuild what you just made. Change the VM size in the YAML, recreate, and watch it come back identically. That’s the payoff of declarative infrastructure, and it’s more convincing when you do it than when you read about it.
• Scale the cluster. Change replicas on the worker pool, re-apply, and watch nodes roll in. Then bump version to a newer compatible release and watch a rolling upgrade — that’s a one-line change doing something that used to be a project.
• Write a security policy as YAML and watch NSX distributed firewall rules appear.
• Put it all in Git. Once your infrastructure is files, a GitOps engine can reconcile it. That’s the subject of my next post.

If you’re a vSphere admin who has been eyeing all this with mild suspicion, I’d encourage you to run through the steps above once in a lab. The concepts are less foreign than the vocabulary suggests — a namespace is a resource pool with a network and a quota, a VM is still a VM, and a load balancer service is the thing you’d have raised a ticket for. The difference is that now it’s a file you own.

Questions or something that didn’t work in your environment? Leave a comment — I read all of them, and environment-specific quirks are exactly what makes these posts better.