VMTECHIE.blog

Tag: Kubernetes

  • Health, Forecasts and Pre-Flight for Every VKS Cluster: Introducing vks-fleet

    Health, Forecasts and Pre-Flight for Every VKS Cluster: Introducing vks-fleet

    VKS makes it easy to give every team its own Kubernetes cluster: VCF Automation hands them out, Cluster API keeps them reconciled, and the Supervisor runs them on the vSphere you already operate. As the number of clusters grows, the natural next step is to see the whole fleet at once: which clusters need attention, what is about to run out, and what a change will do before you make it. That is what I built vks-fleet for.

    vks-fleet is an open-source plugin for Headlamp that brings the Supervisor, vCenter, each cluster’s API and its Prometheus together in one view, for operators, read-only admins and tenants alike.

    A personal project. vks-fleet is something I built in my own time. It is not a Broadcom or VMware product, not endorsed or supported by either, and it uses only public APIs. It is Apache 2.0, and the code is on GitHub.

    The vks-fleet dashboard: a fleet score with a 30-day trend, what needs you now, eight blocks and a wall of clusters
    The fleet dashboard, in demo mode: a score, what needs you now, eight blocks that each open their page, and every cluster as a tile.

    Day-2 questions at fleet scale

    VKS gives you a lot out of the box. Cluster API reconciles your clusters, the Supervisor carries the control plane, and VCF Automation hands out self-service. Once there are many clusters, operators start asking fleet-level questions like these:

    • Which of my clusters needs attention right now, and which of those is actually urgent?
    • A node has been “deleting” for a day. Is that a PodDisruptionBudget, a volume that won’t detach, or the ESXi host underneath?
    • Which disk, volume or certificate runs out in the next 30 days?
    • If I upgrade this cluster, or scale this pool down, what breaks, what moves, and does the namespace have room while it rolls?
    • What is open security-wise across every cluster, not just the one I happen to be logged into?

    Every answer is already there, spread across a kubectl session per cluster, vCenter, the Supervisor’s own namespaces and each cluster’s Prometheus. vks-fleet brings them onto one screen, so the same questions take seconds whether you run three clusters or thirty.

    Why a Headlamp plugin

    I started out thinking about forking a Kubernetes UI and went the other way. Headlamp is an open-source Kubernetes web UI with a proper plugin system: it already handles sign-in, kubeconfig contexts, per-cluster views, logs, shells and YAML. A plugin can add pages, read any cluster Headlamp is signed in to, and hand off to Headlamp’s own views when you want the raw objects. That meant I could spend the effort on the fleet layer and let Headlamp do the rest.

    It also keeps the footprint small. The plugin runs in your browser with Headlamp’s sign-ins. There is no server of its own, no database, no agent in your clusters and no telemetry.

    How it is built to work

    The plugin follows the way an operator actually works through a problem:

    ObserveOne dashboard for the fleet, a score with its trend, and what needs you now.
    DetectRanked issues, each with its cause, and the steps that clear the most of them.
    ExplainA cluster’s timeline beside a walk down the layers under a node.
    PredictWhat runs out in 30 days, and what a change will do before you start it.
    ActControlled changes, each with its checks and a server-side dry run.

    Watch it in a minute: a one-minute walk through demo mode (the dashboard, investigating a node, a fix with its dry run, pre-flight, Supervisor health, security and what to do first) is at the top of the project’s README.

    Observe: one screen for the fleet

    The fleet page opens on a dashboard. At the top is a fleet score (the average of each cluster’s best-practice checks) with a 30-day trend, and a short Needs you now list that puts time-bound problems first: a node disk that fills in two days outranks a warning that has been there for a month.

    Below it are eight blocks: clusters, Supervisor, capacity, the next 30 days, lifecycle, security, governance and network. Each is one number with one line of context, and each opens the page that owns the detail. Under them, every cluster is a tile on a wall, coloured by its worst open problem and grouped by org.

    Detect: issues with a cause, not just a list of findings

    The plugin folds raw signals into issues that say what is wrong, why, and what to do. A stuck node, a drain blocked by a PodDisruptionBudget, a crash-looping pod, a certificate close to expiry, quota and memory overcommit, controllers that stopped reconciling, Supervisor services all running on one host, packages that fail to reconcile, backups that stopped: each becomes one line with its fix or the decision it needs.

    On top of that sits Do these first: the recommended steps that clear the most issues, ranked by effect, each opening where it is done. A Simulate view shows the fleet as it would look with the fixes applied, so you can see the payoff before spending the change window.

    The Issues tab: steps to do first, ranked by effect, then every open issue
    Issues: the steps to do first, then every open issue with its fix or the decision it needs.

    Explain: from a pod down to the ESXi host

    This is the part I use most. Investigate puts one cluster’s timeline (changes, alerts, events and anomalies) beside a walk down the layers under a node: pod, node, Machine, VM, ESXi host, namespace and Supervisor. It names the deepest layer in trouble, so you know whether you are looking at a Kubernetes problem, a VM problem or a host problem. When two troubled nodes share a host, it says so. It also drafts the post-mortem from the same timeline.

    Investigate: the cluster's timeline beside the layers under a stuck node, down to the ESXi host
    Investigate: what happened, beside every layer under the node. Here the deepest layer in trouble is the Machine, stuck deleting.

    Predict: the next 30 days, and pre-flight

    Forecasts from each cluster’s Prometheus (node disks, memory, volumes), the Supervisor’s control-plane disk from vCenter, and certificate expiries all land on one 30-day timeline.

    Pre-flight answers what will this change do? before an upgrade, a node pool scale or a VM class change. It works out which nodes are replaced, which pods move, which single-replica workloads take a brief outage, which disruption budgets or pinned pods would block it, whether the namespace has room for the extra nodes while it rolls, and whether the Supervisor itself is healthy enough to start. Then it gives a verdict: go ahead, go ahead with care, or stop.

    Pre-flight for scaling a node pool down: a blocker, the nodes and pods affected, and each check
    Pre-flight for scaling a pool down by one. The verdict is Stop: a node in the pool is already stuck deleting.

    For upgrades across many clusters there is an upgrade planner that arranges clusters in waves (a small canary first, then the rest) with the same checks for each cluster.

    Act: controlled changes

    Upgrades, node pool scaling, node replacement, drain timeouts, package updates, VM power operations and new clusters can all be started from the plugin. Every change shows its checks first and runs a server-side dry run (?dryRun=All), so anything the API or the VKS admission webhooks would refuse is caught before anything changes. Destructive actions ask you to type the cluster’s name.

    For shared installations there is a read by default, elevate to change mode: viewing uses a read-only sign-in, and a change needs a time-limited elevation with a reason, recorded on the object it changed.

    Supervisor health

    Clusters are only as healthy as the Supervisor running them. A Supervisor page shows the control plane, the ESXi hosts and what runs on each, the leader-election leases of the controllers that manage your clusters (a lease that stops renewing means a controller is stuck, even when its pod says Running), Supervisor services, and the reconcile backlog. Click a host and it shows what that host failing would take down. With an optional, read-only collector it also reads vCenter: configuration status, alarms, utilisation and the control-plane VM’s disks.

    Supervisor health: its score, the ESXi hosts and what runs on each, what vCenter reports, and the controllers
    Supervisor health: the hosts, what runs on each, and what vCenter reports.

    Security: what is open, across every cluster

    The Security overview shows what is open across every cluster you are signed in to: privileged namespaces, privileged and host-level pods, cluster-admin grants, and image vulnerabilities from Trivy Operator, counted once per image and split into your images and the ones VKS ships. Policy results from Kyverno or any engine that writes PolicyReports appear beside them. A cluster without a scanner reads not measured, never zero.

    Security overview: privileged pods and admin grants, critical vulnerabilities, privileged namespaces and accepted risks, then one row per cluster
    Security overview: what is open across the fleet, then one row per cluster.

    Principles behind it

    PrincipleWhat it means in practice
    Public APIs onlyCluster API, VM Operator, NSX VPC, Carvel, Prometheus, and vCenter through a small read-only collector. No private interfaces.
    Nothing installed in your clustersEverything is read through the APIs that are already there. Trivy, Kyverno and Prometheus are used when present, never required.
    Your RBAC decides what you seeOperators see every org, read-only admins see everything and change nothing, tenants see only their own org. It follows the account Headlamp signs in with, not a mode in the UI.
    Honest numbersData that can’t be read is “not measured”, not zero or passing. Every figure says where it came from.
    Dry run before every changeChecks, then a server-side dry run, then confirmation. If a multi-step change fails partway, it says which steps took effect.

    Try it without a lab

    The fastest way to see it is demo mode: a fictional fleet with realistic problems built in, where nothing is ever changed. Install the plugin, open VKS fleet and choose Try the demo. You can scale the demo fleet up to 50 clusters to see how the pages behave at size.

    Headlamp desktop app or Docker: download vks-fleet.tar.gz from the latest release, extract it into Headlamp’s plugins folder, restart Headlamp and open VKS fleet.

    In a cluster, with Helm: the chart installs Headlamp, the plugin, and a job that keeps the Supervisor sign-in fresh. The quick start has the full commands, including trusting vCenter’s CA so certificates are verified; the core of it is:

    VERSION=1.43.0
helm install vks-fleet "https://github.com/avnish80/vks-fleet/releases/download/v$VERSION/vks-fleet-$VERSION.tgz" \
  -n vks-fleet --set plugin.source=download \
  --set refresher.supervisors="<your Supervisor address>" --set tls.caSecret=vks-fleet-ca

    Then connect your Supervisor under Settings → Plugins → vks-fleet. It is also listed on Artifact Hub as a Headlamp plugin.

    Where it stands, honestly

    It is developed and tested against VCF 9.1 with VKS 3.7 on Headlamp 0.45, on a real lab with one Supervisor, two VKS clusters and VCF Automation tenants. Every read and view has been checked there, as have the vCenter collector and changes through the VKS admission webhooks: scaling node pools up and down, pausing and resuming clusters, and a clean Helm install under restricted Pod Security with verified TLS.

    What is not yet confirmed on a live system: deletes through the plugin, drain timeouts on machines that are already deleting, and some links into Headlamp’s own pages. The checks also run from the browser, which suits tens of clusters; demo mode exercises the pages at 50, but a real fleet has so far been tested at two. The README’s validation status keeps the current list.

    If you run VKS, I would genuinely like to hear how it behaves on your fleet, especially with more clusters, more than one Supervisor, or tenants coming in through VCF Automation.

    Get vks-fleet on GitHub Report an issue or idea

    vks-fleet is a personal open-source project under the Apache 2.0 licence. It is not affiliated with, endorsed by, or supported by Broadcom or VMware. Product names are used only to describe what it works with.

  • Your First VM and Kubernetes Cluster in VCF Automation — Without Touching the UI

    Your First VM and Kubernetes Cluster in VCF Automation — Without Touching the UI

    If you administer vSphere, you already know how to build a VM. This post shows you how to build the same VM in VMware Cloud Foundation Automation using nothing but a command line — no UI clicks, no cloud template, no prior Kubernetes experience assumed. It takes about less then 10 minutes, and by the end you’ll have a running web server with a load-balanced IP that answers from your browser.

    I’m deliberately keeping this simple. There is a longer, deeper reference post for the full API surface, but this one is the on-ramp: the shortest honest path from “I’ve never used the VCF Automation API” to “I just built something with it.”

    What we’re going to do
    1. Install the CLI and log in to your organization
    2. Create a project (the tenancy boundary for a team)
    3. Create a Supervisor Namespace (where workloads actually live)
    4. Create a VM with cloud-init that installs nginx
    5. Publish it behind a load balancer and browse to it
    6. Build a Kubernetes cluster in the same namespace, and run a workload on it

    Why bother, when the UI works fine?

    Fair question, and I’d give three honest answers.

    Repeatability. A VM you clicked into existence is a VM nobody can rebuild exactly. A VM defined in a YAML file is one you can commit, review, and recreate identically a year later.

    It’s where the platform is going. In VCF 9, the consumption layer is a Kubernetes-style API. The UI is a client of that API, not the other way round. Learning the objects underneath means the UI stops being magic.

    It’s the foundation for automation. Everything you’d want later — GitOps, pipelines, self-service catalogs — is built on these same objects. This is the first rung.

    None of that means you should stop using the UI. I use both. But knowing what’s underneath makes you considerably harder to stump.

    Before you start

    WhatWhere to get it
    VCF CLI (the vcf command)Broadcom Support portal — it replaces the older CCI kubectl plugin
    kubectlAny recent version; the CLI generates a kubeconfig for it
    An API tokenLog into your VCF Automation org portal → your user menu → API Tokens → Generate. Copy it — it’s shown once.
    Org admin rightsNeeded to create projects. If you only have consumer rights, skip to Step 4 and use an existing namespace.

    Everything below is run from PowerShell on Windows. It works identically on Linux and macOS; only the quoting changes.

    Step 1 — Log in

    First, store your token in a variable. Note the quotes — without them PowerShell tries to execute the token as a command, which produces a confusing error.

    $vcfa_token = "your-api-token-here"

    Now create a context. This is the CLI’s word for “a saved connection to an environment.”

    vcf context create org2-ctgw `
  --endpoint https://auto-a.site-a.vcf.lab `
  --type cci `
  --auth-type basic `
  --tenant-name Org2-CTGW `
  --api-token $vcfa_token `
  --insecure-skip-tls-verify

    Substitute your own endpoint and tenant name. The backtick is PowerShell’s line-continuation character — or just put it all on one line. Use --insecure-skip-tls-verify only in a lab; in production supply your CA certificate with --ca-certificate instead.

    VCF CLI context create command output showing successful login and available contexts
    A successful login. Note that the CLI lists every context you now have access to — one for the organization, plus one for each Supervisor Namespace you can reach.

    That output is worth reading carefully, because it explains the whole model. You get an organization context (org2-ctgw) and a namespace context for each namespace you can use (org2-ctgw:org2-ns1-4hm32:default-project).

    The one concept to take away There are two levels. The organization context is where you manage tenancy — projects, namespaces, who gets what. The namespace context is where you build things — VMs, clusters, networks. Almost every “why can’t I see that?” moment is being in the wrong one. If a command returns “the server doesn’t have a resource type”, you’re at the org level asking a workload question.

    Step 2 — Create a project

    A project is the boundary you’d hand to a team. Save this as project.yml:

    apiVersion: project.cci.vmware.com/v1alpha2
kind: Project
metadata:
  name: team-phoenix
spec:
  description: Phoenix dev team
    kubectl --context org2-ctgw create -f project.yml
# project.project.cci.vmware.com/team-phoenix created
    Use create, not apply This will save you a puzzling ten minutes. kubectl apply stores a tracking annotation on the object, and this API rejects it outright with “Annotation updates are not supported.” Everywhere in this post, use kubectl create.

    Step 3 — Create a namespace

    The project is just a boundary. A Supervisor Namespace is the thing with actual capacity — CPU, memory, storage, a network — where your VMs will live. Think of it as the resource pool plus the network plus the quota, expressed as one object.

    You need four pieces of information, all of which you can read off an existing namespace or get from whoever set up your environment:

    FieldWhat it meansExample
    regionNameWhich region (a vCenter/Supervisor grouping)region-west
    classNameThe T-shirt size defined by your adminsmall
    vpcNameThe NSX VPC providing the networkdefault-region-west
    classConfigOverridesThe zone and storage class, with limitssee below
    apiVersion: infrastructure.cci.vmware.com/v1alpha3
kind: SupervisorNamespace
metadata:
  generateName: phoenix-ns-
  namespace: team-phoenix
spec:
  regionName: region-west
  className: small
  vpcName: default-region-west
  classConfigOverrides:
    storageClasses:
    - name: vsan-default-storage-policy
      limit: 204800Mi
    zones:
    - name: z-wld-a
      cpuLimit: 15000M
      cpuReservation: 0M
      memoryLimit: 15000Mi
      memoryReservation: 0Mi
    kubectl --context org2-ctgw create -f namespace.yml
kubectl --context org2-ctgw get supervisornamespaces -n team-phoenix

    Wait for STATUS: Created. Then refresh your contexts so the new namespace appears, and switch into it:

    vcf context use org2-ctgw
kubectl config get-contexts
kubectl config use-context org2-ctgw:phoenix-ns-kj9g6:team-phoenix
    Two things that catch people here The namespace name is generated with a random suffix (phoenix-ns-kj9g6), so read it from the previous command rather than guessing. And the context name is the full three-part string <org>:<namespace>:<project> — using just the namespace name gives you “no context exists.”

    You can confirm the same thing in the UI, which is a reassuring way to check your work while you’re learning:

    VCF Automation UI showing the newly created namespace phoenix-ns-kj9g6 in project team-phoenix, Active, with CPU, memory and storage quota bars
    The namespace created from the command line, shown in the VCF Automation UI — Active, in the team-phoenix project, with its CPU, memory and storage allocation. The API and the UI are two views of the same object.
    Why the long classConfigOverrides block? When you create a namespace through the portal, it fills these values in for you from the class. Creating one through the API, you have to state them yourself — and the zone entry needs all four CPU and memory fields, not just a name. If you leave them out, the API tells you exactly what’s missing, one field at a time. That’s actually a decent way to learn the object: submit a minimal version and let the error messages guide you.

    Step 4 — Create a VM

    Now the part that will feel familiar. We need three things: a size, an image, and a storage policy. Ask the namespace what it has:

    kubectl get virtualmachineclasses     # the sizes available to you
kubectl get virtualmachineimages      # images from your content library
kubectl get clustervirtualmachineimages | Select-String "ubuntu"
    If virtualmachineimages comes back empty, you’re not stuck. That list is empty when no content library is attached to your namespace class. You can still deploy from a cluster-wide image by referencing its vmi- identifier directly, which is what I do below. Getting a content library attached is the better long-term answer — it gives you friendly names instead of hashes — but it isn’t a blocker today.

    Here’s the VM, together with a cloud-init secret that creates a login and installs nginx. Save both in one file:

    apiVersion: v1
kind: Secret
metadata:
  name: web01-bootstrap
stringData:
  user-data: |
    #cloud-config
    users:
    - name: ops
      sudo: ALL=(ALL) NOPASSWD:ALL
      shell: /bin/bash
      ssh_authorized_keys:
      - ssh-ed25519 AAAA...your-key-here
    packages: [nginx]
    runcmd:
    - systemctl enable --now nginx
---
apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachine
metadata:
  name: web01
  labels:
    app: web
spec:
  className: best-effort-small
  imageName: vmi-5dcfa60c7cc579296
  storageClass: vsan-default-storage-policy
  powerState: PoweredOn
  bootstrap:
    cloudInit:
      rawCloudConfig:
        name: web01-bootstrap
        key: user-data
    kubectl create -f vm.yml
    PowerShell output showing the VirtualMachine manifest and confirmation that the secret and virtual machine were created
    Two objects, one command: the cloud-init secret and the VM itself.

    Watch it come up. The VM will report an IP address once VMware Tools starts reporting:

    kubectl get vm web01
kubectl get vm web01 -o jsonpath='{.status.network.primaryIP4}'
# 172.30.0.66

    Notice what you didn’t have to specify: no port group, no IP address, no datastore. The VM picked up an address from the namespace’s default network automatically and landed on storage matching the policy you named. That’s the platform doing the placement work, exactly as it does for a Kubernetes pod.

    Cloud-init only runs on first boot You cannot add a bootstrap section to a VM that already exists. If you forget it, the VM boots fine but has no user account and no way in — you delete and recreate. Decide your bootstrap before you provision.

    Step 5 — Publish it

    A VirtualMachineService puts a load-balanced IP in front of one or more VMs, selected by label. Our VM carries app: web, so the service will find it.

    apiVersion: vmoperator.vmware.com/v1alpha5
kind: VirtualMachineService
metadata:
  name: web-lb
spec:
  type: LoadBalancer
  selector:
    app: web
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
    PowerShell output showing the VirtualMachineService manifest and the service created as type LoadBalancer
    The load balancer service, created. NSX allocates the external address within seconds.
    Reading the external IP kubectl get vmservice prints only name, type and age — the address isn’t in its default columns. Use kubectl get svc instead, or ask for the field directly with -o jsonpath='{.status.loadBalancer.ingress[0].ip}'.

    Step 6 — Confirm it works

    kubectl get svc
# web-lb   LoadBalancer   172.29.255.24   40.60.0.4   80/TCP   2m31s

kubectl get endpoints
# web-lb   172.30.0.66:80   2m34s

curl.exe http://40.60.0.4
    PowerShell output showing the service endpoints and a curl request returning the nginx welcome page
    The whole chain proven: external IP 40.60.0.4, backend 172.30.0.66:80, and nginx answering.

    That’s the full path working. The load balancer address is allocated from the NSX VPC, the endpoint is your VM’s own IP, and nginx — installed by cloud-init at first boot — is serving. Nothing in that sequence involved the UI.

    Step 7 — Build a Kubernetes cluster

    Here’s the part that surprises people: a Kubernetes cluster is just another object in the same namespace you’ve been working in. Same context, same kubectl create, same mental model. You describe the cluster you want and the platform builds the nodes.

    You need to know two things first — and both are places where a sensible-looking manifest fails.

    Where the cluster template lives. Clusters are built from a ClusterClass, which is a template your platform team maintains. Crucially, it does not live in your namespace:

    kubectl get clusterclasses
# NAME                     VARIABLES READY   AGE
# builtin-generic-v3.1.0   True              164m
# builtin-generic-v3.2.0   True              164m
# builtin-generic-v3.3.0   True              164m

    That list is not the whole story. The class you actually want may live in a shared system namespace and won’t appear here at all — you reference it explicitly by name and namespace.

    Which Kubernetes version you can use. Ask the platform rather than guessing:

    kubectl get kubernetesreleases

    You’ll get a long list. Only the rows showing both READY: True and COMPATIBLE: True are usable. Pick one of those.

    Now the cluster itself — one control plane node, one worker, deliberately small:

    apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: phoenix-k8s
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["192.168.156.0/20"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
    serviceDomain: cluster.local
  topology:
    classRef:
      name: builtin-generic-v3.6.0
      namespace: vmware-system-vks-public
    version: v1.35.5---vmware.1-vkr.1
    variables:
    - name: vmClass
      value: best-effort-medium
    - name: storageClass
      value: vsan-default-storage-policy
    controlPlane:
      replicas: 1
      metadata:
        annotations:
          run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-96cb71ec140d192ad
    workers:
      machineDeployments:
      - class: node-pool
        name: phoenix-k8s-np-1
        replicas: 1
        metadata:
          annotations:
            run.tanzu.vmware.com/resolve-os-image: os-name=photon, content-library=cl-96cb71ec140d192ad
    kubectl create -f cluster.yml
kubectl get cluster phoenix-k8s -w

    Provisioning takes five to six minutes for this size. PHASE: Provisioned appears early and only means the objects were created — the column to watch is AVAILABLE, which flips to True once the control plane is genuinely up. kubectl get machines gives you the per-node view while you wait.

    Three things in that manifest are not obvious 1. classRef needs a namespace. The ClusterClass lives in a shared system namespace, and without it you’ll get a “not found” even though the class plainly exists.
    2. The version string is the long form — v1.35.5---vmware.1-vkr.1, with three dashes, not the +vmware.1 form shown elsewhere.
    3. The resolve-os-image annotation is often required. If your environment has the same node image registered in more than one content library — common in labs that have been upgraded a few times — the resolver refuses to guess and fails with “Multiple OSImages resolved.” Adding content-library= narrows it to one. You’ll find the library ID in your Content Libraries list.
    The shortcut worth knowing Rather than assembling this by hand, start a cluster creation in the VCF Automation UI and look at the YAML it generates. It will show you the right ClusterClass namespace, the right version format, and any annotations your environment needs — all correct for your specific build. Copy that, then edit it. This is far quicker than reverse-engineering from error messages, and it’s how I arrived at the manifest above.

    Step 8 — Get onto the cluster and run something

    Once AVAILABLE is True, download the cluster’s kubeconfig from the VCF Automation UI (the cluster’s detail page offers it) and point kubectl at it:

    kubectl --kubeconfig .\phoenix-k8s.yaml get nodes
# NAME                        STATUS   ROLES           AGE   VERSION
# phoenix-k8s-...             Ready    control-plane   12m   v1.35.5+vmware.1
# phoenix-k8s-...-np-1-...    Ready    <none>          9m    v1.35.5+vmware.1

    You’re now a cluster administrator on an ordinary, conformant Kubernetes cluster. Everything you know about Kubernetes applies from here.

    With one important exception, which is worth meeting deliberately rather than by accident.

    Your first deployment will probably be rejected — and that’s correct

    Try the obvious thing:

    kubectl --kubeconfig .\phoenix-k8s.yaml create deployment web --image=nginx --replicas=2

    The deployment is created, but no pods ever appear. Looking at the ReplicaSet tells you why:

    pods "web-..." is forbidden: violates PodSecurity "restricted:latest":
  allowPrivilegeEscalation != false
  unrestricted capabilities (must set capabilities.drop=["ALL"])
  runAsNonRoot != true
  seccompProfile (must set seccompProfile.type to "RuntimeDefault")

    This is Pod Security Admission enforcing the restricted profile, and it’s switched on by default. A stock container image that runs as root is refused at creation — not warned about, refused. For a vSphere admin used to “deploy first, harden later,” this is a genuine change of habit, and a welcome one: the cluster will not run a privileged workload just because you forgot.

    The fix is to declare what the workload actually needs. Each field below answers one of the four complaints, and the image is swapped for a non-root variant that listens on 8080 rather than binding port 80 as root:

    apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: nginx
        image: nginxinc/nginx-unprivileged:stable
        ports:
        - containerPort: 8080
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop: ["ALL"]
    kubectl --kubeconfig .\phoenix-k8s.yaml apply -f web.yml
kubectl --kubeconfig .\phoenix-k8s.yaml expose deployment web --port=80 --target-port=8080 --type=LoadBalancer
kubectl --kubeconfig .\phoenix-k8s.yaml get svc web

    The external IP is allocated by the same NSX load balancer that served your VM in Step 5 — usually within seconds. Confirm there are backends behind it:

    kubectl --kubeconfig .\phoenix-k8s.yaml get endpointslices -l kubernetes.io/service-name=web
# ADDRESSTYPE   PORTS   ENDPOINTS
# IPv4          8080    192.168.147.3,192.168.145.12
    Note that kubectl apply works here. The “use create, not apply” rule applies to the VCF Automation API only. Inside a VKS cluster you’re talking to a normal Kubernetes API server, and all your usual habits — apply, Helm, Kustomize — work exactly as expected.

    The five things that will trip you up

    These cost me time so they don’t have to cost you any.

    1. Use kubectl create, never apply. The API rejects the annotation that apply depends on.
    2. Context names are three parts. <org>:<namespace>:<project>, not just the namespace.
    3. Know which level you’re at. “The server doesn’t have a resource type” almost always means you’re at the org context asking a workload question, or vice versa.
    4. Tokens expire mid-session. A sudden Unauthorized, or a strange complaint about downloading the OpenAPI schema, usually just means your session lapsed. vcf context use <org> re-authenticates.
    5. Check the API version against your build. Published examples lag. On the environment used here, namespaces are served at v1alpha3 and VMs at v1alpha5. The most reliable way to learn the current shape of any object is to dump a working one: kubectl get vm <name> -o yaml. Copy the structure, change the values.

    That last one is worth internalising as a general habit. Rather than hunting for documentation that matches your exact release, clone something that already works in your own environment. It is always correct, by definition.

    Where to go next

    You now have the pattern for everything else on this platform. The same two-level model, the same create workflow, and the same “clone a working object” trick apply to Kubernetes clusters, persistent volumes, NSX security policies, and micro-segmentation rules — all of which are just more objects in the same API.

    A few natural next steps, in the order I’d take them:

    • Delete and rebuild what you just made. Change the VM size in the YAML, recreate, and watch it come back identically. That’s the payoff of declarative infrastructure, and it’s more convincing when you do it than when you read about it.
    • Scale the cluster. Change replicas on the worker pool, re-apply, and watch nodes roll in. Then bump version to a newer compatible release and watch a rolling upgrade — that’s a one-line change doing something that used to be a project.
    • Write a security policy as YAML and watch NSX distributed firewall rules appear.
    • Put it all in Git. Once your infrastructure is files, a GitOps engine can reconcile it. That’s the subject of my next post.

    If you’re a vSphere admin who has been eyeing all this with mild suspicion, I’d encourage you to run through the steps above once in a lab. The concepts are less foreign than the vocabulary suggests — a namespace is a resource pool with a network and a quota, a VM is still a VM, and a load balancer service is the thing you’d have raised a ticket for. The difference is that now it’s a file you own.

    Questions or something that didn’t work in your environment? Leave a comment — I read all of them, and environment-specific quirks are exactly what makes these posts better.