VCAP — VCF VKS (3V0-24.25)

VKS Architecture & Components#

VKS in VCF 9.0 introduces a simplified control plane based on Supervisor clusters—a native vSphere construct that transforms ESXi hosts into Kubernetes nodes. The Supervisor is not a virtual machine; it IS the vSphere cluster itself running the Kubernetes control plane as system VMs.

Supervisor Cluster Topology

A Supervisor requires:

1 control plane VM (Simple availability - the default when a Supervisor is activated during VCF workload-domain creation, scalable to 3 later) or 3 control plane VMs (High Availability, anti-affinity enforced, required for the Three Management Zone model): Each runs kubelet, API server, etcd shard, controller-manager, scheduler. These are 4-vCPU, 8GB RAM system pods on dedicated resource pool.

Worker ESXi nodes

: Regular cluster members host user workloads. Each gets kubelet agent injected.

vCenter registration

: Cluster must be managed by vCenter; standalone hosts not supported.

Networking stack

: NSX-T (preferred for network policies, GSLB) OR vSphere Distributed Switch + HAProxy/Avi (legacy path).

Persistent storage

: vSAN or external storage with vSphere CSI driver support.

┌─────────────────────────────────────────────────────────┐
│                     VCF Cluster (Supervisor)             │
├──────────────┬──────────────┬──────────────────────────┤
│  Control-0   │  Control-1   │  Control-2               │
│  (4vCPU/8GB) │  (4vCPU/8GB) │  (4vCPU/8GB)            │
│  API Server  │  API Server  │  API Server             │
│  Scheduler   │  Scheduler   │  Scheduler              │
│  etcd        │  etcd        │  etcd                   │
├──────────────┼──────────────┼──────────────────────────┤
│         ESXi-1        │         ESXi-2        │ ESXi-3 │
│       (Worker)        │       (Worker)        │(Worker)│
│  TKG Pods, Apps       │  TKG Pods, Apps       │ Pods   │
└──────────────┴──────────────┴──────────────────────────┘

Namespace Isolation & Resource Classes

Namespaces in Supervisor allow multi-tenancy with enforced quotas:

VM Classes

: Predefined CPU/RAM/storage combinations for pod requests (e.g., guaranteed-small: 2vCPU/4GB, guaranteed-xlarge: 8vCPU/32GB). Pod requests must match a defined class.

Storage Classes

: vSAN storage policies, vSAN file service for ReadWriteMany, external storage via CSI.

Content Library

: Required for TKR (Tanzu Kubernetes Release) image retrieval. Each Supervisor uses a shared library for base images.

RBAC + SSO

: Namespaces bound to vSphere SSO principals; kubectl access via vsphere login plugin.

TKG (Tanzu Kubernetes Grid) Cluster Provisioning

TKG clusters are provisioned on Supervisor using ClusterClass—a CRD-based declarative model:

ClusterClass

: Defines template for node image, control plane topology, worker node pools, CNI (Antrea), kubelet config.

Machine Deployments

: Scale-out worker nodes with autoscaling enabled. Topology labels for zone awareness (topology.kubernetes.io/zone).

Control Plane Scaling

: 1, 3, or 5 control plane VMs; etcd clustering supported at scale.

Kubernetes Version Management

: TKR images pinned (v1.27, v1.28, v1.29 available). Rolling cluster upgrades with node pool coordination.

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: prod-tkg-cluster
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["10.244.0.0/16"]
  topology:
    class: tanzukubernetescluster
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
      - class: node-pool-1
        name: primary-pool
        replicas: 3

Storage Architecture: CSI & vSphere StorageClass

vSphere CSI driver (deployed system-wide in Supervisor) connects Kubernetes to vSAN and external arrays:

vSphere CSI Driver

: Controller plugin (creates PVs), node plugin (attaches volumes). Uses vCenter SOAP API for provisioning.
PersistentVolumeClaims (PVC)

: Pod storage requests resolved via StorageClass → vSAN policy objects or external array.

vSAN ReadWriteMany

: File services enable RWX mode (typical for databases, NFS-backed PVCs).

Volume Snapshots

: Kubernetes VolumeSnapshot CRD maps to vSAN snapshots; snapshots can be cloned to new PVs.

Topology-Aware Provisioning

: Volumes provisioned on same rack/zone as pod for latency optimization.

Networking for Kubernetes

NSX Container Plugin (NCP)
(preferred):

Translates Kubernetes NetworkPolicy to NSX DFW rules automatically.

Pod-to-pod communication via overlay segments; service ingress via Avi ALB LoadBalancer services.

GSLB (Global Service Load Balancing) for multi-cluster services.

Integration with NSX Federation for multi-site clusters.

Antrea CNI
(alternative in non-NSX environments):

Open-source, CNCF project; uses OVS (Open vSwitch) on hypervisor.

Network policies enforced via OVS flows; lower overhead than NSX DFW.

Service LoadBalancer requires external Avi/HAProxy.

Ingress & Services
:

Kubernetes Service Types

: ClusterIP (internal only), NodePort (host port), LoadBalancer (Avi VIP allocation).

Ingress Controller

: ALB Ingress Controller (Avi) for HTTP/HTTPS routing; NSX ALB required for production HA.

Service Mesh

: Istio/Anthos Service Mesh optional for advanced traffic management, mutual TLS.

Key Takeaways

• In VCDX defense, be ready to justify NSX vs. VDS+HAProxy choice. NSX provides superior multi-site support, GSLB, and identity-based firewall. VDS+HAProxy is cost-effective for single-site labs but limits security design at scale.

Supervisor Cluster Setup & Configuration#

Step 1: Enable Workload Management

In vSphere Client → Cluster → Configure → Workload Management:

Select networking stack: NSX-T Tier-0 router OR vSphere VDS.

Assign management network (vSAN, management VMK separate or shared).

Allocate control plane network CIDR and node CIDR (not overlapping with vSAN/management).

Select storage (vSAN or external).

Enable container runtime (containerd default; Docker no longer supported in VCF 9.0).

NSX-T Configuration
:

Require Tier-0 router with connected Tier-1 for Supervisor workload networks.

VLAN transport zone for control plane; overlay transport zone for pod networks.

Edge nodes sized appropriately (4vCPU/8GB for small, 8vCPU/32GB for large clusters).

BGP peering to upstream network for Kubernetes service CIDR advertisement.

#!/bin/bash

NSX CLI: Create Supervisor integration

• nsx-cli
• get platform cluster-managers
• get logical-routers

Verify Tier-0/Tier-1 connectivity

show logical-switches

Step 2: Content Library Setup for TKR Images

TKR (Tanzu Kubernetes Release) images must be imported into a shared vCenter content library:

Content Library Type

: Subscribed library (preferred, auto-syncs minor versions) OR local library (manual upload).

TKR Image Source

: VMware tanzu.io or internal proxy for air-gapped deployments.

Library storage must have sufficient capacity (3-5GB per TKR version × number of supported versions).

Supervisor automatically discovers available TKRs from library for ClusterClass definition.

Step 3: Namespace Creation & Quotas

Each namespace enforces CPU, memory, storage quotas and limits pod proliferation:

apiVersion: v1
kind: Namespace
metadata:
  name: production
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: prod-quota
  namespace: production
spec:
  hard:
    pods: "100"
    requests.cpu: "50"
    requests.memory: "200Gi"
    limits.cpu: "100"
    limits.memory: "400Gi"
    persistentvolumeclaims: "20"

Step 4: RBAC & vSphere SSO Integration

Kubernetes RBAC bound to vSphere SSO users/groups:

kubectl vsphere login

: Single sign-on; kubeconfig auto-generates token bound to SSO session.

Role Bindings

: vSphere SSO group → Kubernetes ClusterRole/Role.

Audit Logging

: Audit Policy bound to vCenter event logging; track API calls, RBAC decisions.

Key Takeaways

• For exam: Workload Management requires NSX-T Tier-0/Tier-1 (preferred) or VDS+external LB. Management network, control plane CIDR, node CIDR must not overlap.
• For exam: Subscribed Content Library auto-syncs TKR versions. Capacity planning: 3-5GB per TKR version × number of versions. Always validate library sync before TKG cluster creation.
• For exam: Namespace quotas enforce multi-tenancy isolation (pods, CPU, memory, PVC limits). RBAC binds vSphere SSO to Kubernetes roles. kubectl vsphere login provides SSO-based kubeconfig.

TKG Cluster Lifecycle: Provisioning to Upgrade#

ClusterClass-Based Provisioning (VCF 9.0 Standard)

ClusterClass replaced TKC (Tanzu Kubernetes Cluster) CRD in VCF 9.0+. Advantages: templatization, consistent upgrades, topology-aware node pools.

ClusterClass Structure
:

TKR Reference

: Links to TKG image in content library (e.g., v1.28.5).

Kubernetes YAML

: kubelet flags, API server audit log, encryption-at-rest keys.

Control Plane Topology

: 1, 3, or 5 replicas; machine health checks; node pool definitions.

Bootstrap Script

: Post-provisioning config (agent installation, security hardening).

Node Pool Management & Autoscaling

Machine Deployment

: Each pool is an independent MachineDeployment object. Replicas define desired count.

Cluster Autoscaler

: Deployed as system pod; scales pools based on pending pod requests and node utilization thresholds.

Pod Disruption Budgets (PDB)

: Ensure graceful shutdown during autoscaler downscaling (min-available replicas).

Zone Topology

: Machine pools labeled with zone (rack, building); scheduler respects topology.kubernetes.io/zone affinity.

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  name: prod-pool-zone-a
spec:
  selector:
    matchLabels:
      nodepool: prod
  template:
    spec:
      clusterName: prod-tkg-cluster
      bootstrap:
        configRef:
          name: prod-pool-zone-a
          kind: KubeadmConfigTemplate
      infrastructureRef:
        name: prod-pool-zone-a
        kind: VSphereVMTemplate
---
apiVersion: policy.podautoscaler.io/v1
kind: ClusterAutoscaler
metadata:
  name: autoscaler-config
spec:
  minNodeGroupSize: 1
  maxNodeGroupSize: 10
  scaleDownEnabled: true
  scaleDownDelay: 10m

Rolling Updates & Zero-Downtime Upgrades

VCF 9.0 supports async patching—apply updates independently to Supervisor and clusters:

In-place Node Updates

: CoreDump disabled during updates to prevent pod eviction delays.

Etcd Backup Before Upgrade

: Velero snapshot of etcd cluster state prior to Kubernetes version bump.

Control Plane Rolling Restart

: One API server updated at a time; clients reconnect automatically.

Worker Pool Strategy

: Surge replicas spun up, old nodes drained gracefully, PDB honors in-flight connections.

Kubernetes Version Lifecycle

VCF 9.0 ships with multiple TKR versions concurrently:

Upstream Kubernetes release cycle: every 4 months; 3 releases supported (current + 2 prior).

Update ClusterClass TKR reference; rolling update triggered for all nodes.

Feature gates controlled via kubelet config in ClusterClass YAML.

Backup with Velero

Velero (Heptio) provides cluster-wide backup:

Velero Agent

: DaemonSet on all nodes; takes snapshots of PVs via CSI driver.

Backup Scope

: Namespaces, CRDs, secrets (encrypted at rest in backup target); exclude system namespaces.

Backup Target

: S3 (MinIO, AWS S3), NFS via Restic.

Restore Strategy

: Restore to same cluster, different cluster, or different namespace.

RPO/RTO

: Hourly backups typical; 15-30 min restore time depending on cluster size.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: prod-daily-backup
spec:
  schedule: "0 2 * * *"  # 2am UTC daily
  includedNamespaces: ["production", "staging"]
  excludedNamespaces: ["kube-system", "kube-public"]
  storageLocation: s3-backup-bucket
  volumeSnapshotLocation: vsphere-csi-snapshots
  ttl: 720h  # 30 days retention

Key Takeaways

• Design consideration: NSX+NCP provides tighter integration, GSLB, and federation support. Antrea is cost-effective for single-site, non-NSX deployments but limits multi-site HA design. In VCDX design doc, justify networking stack choice against SLA, multi-site requirements, and security posture.
• In VCDX design, emphasize multi-tenant isolation strategy: combine namespace quotas, network policies, PSA enforcement, and RBAC. Document assumptions about trust boundaries (are all tenants internal? external SaaS?). Justify encryption-at-rest if regulated data present (PCI-DSS, HIPAA).

Supervisor Architecture Deep Dive#

Three-Node Control Plane: HA & Leader Election via etcd

Supervisor clusters deploy one or three Kubernetes control-plane VMs on three separate ESXi hosts, providing N+1 fault tolerance. These VMs are not user-facing (they're system-managed) and run a minimal Linux OS with kubelet components.

Key Point:

In a High Availability Supervisor (three control-plane nodes), the three control-plane VMs form a single etcd cluster (port 2379-2380). Leader election ensures only one API server actively handles requests; the other two standby. If leader fails, etcd re-elects within 1-3 seconds.

vSphere Pods: Pod Isolation via Kata Containers

By default, pods on Supervisor run on TKG worker nodes (traditional Kubernetes VMs). Enable "vSphere Pods" — a pod class that runs each pod as its own minimal VM (via Kata containers), achieving near-bare-metal isolation. Trade-off: slower startup (1-2 sec vs milliseconds), higher resource footprint, but superior isolation.

TKG-on-Supervisor vs Supervisor-as-K8s

Two deployment patterns exist:

Pattern 1: TKG Clusters on Supervisor (Recommended for Multi-Tenancy)

• Supervisor (3 control planes, system namespace for infra, no user pods directly)
• + TKG Cluster "prod-k8s-1" (user namespace ns-tenant-a, N worker nodes)
• + TKG Cluster "dev-k8s-1" (user namespace ns-tenant-b, M worker nodes)
• Advantage: Namespace isolation, separate etcd per cluster, fault containment
• Risk: Each TKG cluster = 3-node etcd + N workers = larger footprint

Pattern 2: Supervisor-as-K8s (Bare Usage, Simpler, Limited Scale)

• Supervisor (3 control planes + user-created TKG nodes via CAPI directly)
• No TKG cluster CR; users deploy Kubernetes objects directly to Supervisor API
• Supervisor manages TKG node VMs via CAPI MachineDeployment
• Advantage: Single etcd, no nested cluster overhead, minimal infra
• Risk: All workloads in shared namespace, lower isolation

Workload Management Enablement Prerequisites

Enabling Workload Management on a vSphere cluster requires these components present and healthy:

vSAN Cluster (Required)

All nodes in the vSphere cluster must have vSAN enabled (vSAN disk groups configured). Single-node vSAN is not supported for Supervisor. Minimum 3 nodes recommended for HA (Supervisor control planes spread across 3 hosts).

Distributed Virtual Switch (VDS)

All ESXi hosts must be connected to a VDS. VLAN or VXLAN uplink portgroups required for management network and pod network overlay. Standard vSwitch is not supported.

NSX or External Load Balancer

Supervisor API, Kubernetes API, and TKG service LoadBalancer endpoints need a load balancer. Options: NSX native (recommended, integrated DFW), or external LB (HAProxy, F5, Avi). LB must be layer 2 or 3 adjacent to management network.

Content Library (Subscribed)

Supervisor needs a subscribed content library to pull TKG OVA templates (Ubuntu, Photon OS). If offline, manual OVA upload. Check library sync status daily; out-of-date templates cause TKG creation failures.

DNS and NTP

All vSphere components and Supervisor must have synchronized clocks (within 1 sec). Forward/reverse DNS resolution for Supervisor API FQDN must work from all clusters. Forward lookup for vCenter FQDN required from ESXi.

Compute Resources

Minimum 3 ESXi hosts with 8 cores, 32 GB RAM each. Supervisor control-plane VMs consume 4 cores, 12 GB total. Plus TKG node capacity on same cluster (rule of thumb: reserve 20 percent headroom for Supervisor plus upgrades).

vSphere Network Setup

Supervisor Management Network (CIDR for control planes, assigned via network policy), TKG Service CIDR (pods), Service CIDR (ClusterIP ranges). No overlaps. Typical: 10.0.0.0/24 for mgmt, 10.1.0.0/16 for pod, 10.2.0.0/16 for service.

vSphere 8.0+ or vSphere 7 with U3+

Supervisor on vSphere 7 requires specific update levels; vSphere 8.0+ is standard. vCenter must be at same or newer version than ESXi.

Dependency Chain on Enablement Failure:

If content library sync fails, TKG creation blocked. If NSX LB down, new TKG service LoadBalancer assignments hang. If vSAN unhealthy, Supervisor control planes crash-loop. Always pre-flight all 8 requirements before attempting Workload Management enablement.

Supervisor Deployment Types and Control Plane Availability (VCF 9.0)

Simplified Supervisor Model — documented attributes: single control plane VM, single vNIC control plane VM, no load balancer. Benefit: supports running VM Service. Limitation, verbatim: 'Does not support running vSphere Pods and other Supervisor services.' It is real VCF 9.0 terminology, not a nickname.

Control Plane Availability is a separate axis:

├── Simple: ONE control-plane node. The default when a Supervisor is activated as part of VCF workload-domain creation. Scalable to three later. NOT available for the Three-Zone model.
└── High Availability: THREE control-plane nodes. Mandatory for the Three Management Zone model.

So 'a Supervisor always has three control-plane VMs' is FALSE for VCF 9.0 — three is the HA choice, not a universal rule. Related constraint: the Foundation Load Balancer (FLB) is VLAN-networking-only and is not supported with zonal HA.
Source: techdocs VCF 9.0 Design — vSphere Supervisor Deployment Types.

Key Takeaways

• Supervisor Cluster Won't Enable:Check: vSAN health (all hosts healthy), DNS resolves supervisor-api FQDN, NSX LB configured and reachable, Content Library subscribed and synced within 24h, all 8 hosts have 20 percent free disk space.
• TKG Cluster Stuck in Creating:Check CAPI controllers: kubectl get events -A (look for Machine creation errors). Common: IP pool exhausted (increase IP range), VM class too small (min 2 CPU, 4 GB RAM), content library OVA stale (force resync).
• kubectl vsphere login Fails:Check vSphere SSO health, user exists in AD, kubeconfig has correct server FQDN (ping supervisor-api). Proxy issues: if behind proxy, set HTTP_PROXY, HTTPS_PROXY env vars.
• Pod Stuck in Pending:Check Pod events (kubectl describe pod), most common: CSI driver not ready (pvc pending), CNI plugin missing (no IP assigned), PSA blocking (image policy, securityContext). Verify StorageClass exists, CNI DaemonSet healthy.
• Service LoadBalancer Stuck in Pending:NCP agent logs show NSX LB error (quota exceeded, wrong backend pool subnet). Check NSX LB capacity, verify service CIDR routable to NSX.
• Exam: Supervisor control-plane count is ONE (Simple, the default via VCF workload-domain creation) or THREE (High Availability, required for Three Management Zones) — not 'always three'. The Simplified Supervisor Model is 1 CP VM, 1 vNIC, no load balancer, VM Service only (no vSphere Pods, no other Supervisor services).