Academy/VVS/Private AI Ready Infrastructure
This solution targets VCF 5.2

Private AI Ready Infrastructure for VMware Cloud Foundation

VCF 5.2architectvcdxautomationPages 170-236; design decisions appendix pp.226-236 (49 decisions, AIR-* ids)

Design, implementation, and operational guidance for AI GPU-enabled accelerated workload domains running on vSphere with Tanzu. Extends DRI with GPU passthrough/vGPU (NVIDIA A100/L40S/H100), SR-IOV, GPUDirect RDMA, NVLink/NVSwitch support, NVIDIA AI Enterprise (NVAIE) licensing (CLS/DLS), VMware Data Services Manager for database services (PostgreSQL with pgvector, MySQL), and VMware Private AI Foundation with NVIDIA add-on. Supports deep learning VMs, Tanzu Kubernetes Grid clusters running AI containers, and RAG workloads.

Key Components: NSX, SDDC Manager, vCenter, ESXi, Tanzu

External dependencies: VMware Data Services Manager 2.0.2 (part of VMware Private AI Foundation with NVIDIA)

Design Decisions
Implementation
Operations
VCDX Defense
Quiz (15)
Flashcards (15)

49 design decisions. The Quality column is the Academy's assessment for defence practice; the VVS documents do not classify decisions by design quality (D133).

DD-IDDecisionQuality (Academy view)
AIR-COMPUTE-001Select servers with CPUs with a high number of cores.Performance

Decision: Select servers with CPUs with a high number of cores.

Rationale: To optimize computational efficiency and minimize the need for scaling out by adding more nodes, consider scaling up the CPU core count in each server. By choosing CPUs with a high number of cores, you can effectively handle multiple inference threads simultaneously. This approach maximizes hardware utilization and enhances the capacity to manage parallel tasks, leading to improved performance and resource utilization in inference workloads

Implication: High-end CPUs might increase the overall cost of the solution.

Component: AIR

AIR-COMPUTE-002Select a fast-access memory.Performance

Decision: Select a fast-access memory.

Rationale: Minimal latency for data retrieval is crucial for realtime inference applications. Increased latency reduces inference performance and give a poor user experience.

Implication: Re-purposing available servers might not be a feasible option and overall cost of the solution might increase.

Component: AIR

AIR-COMPUTE-003Select CPUs with Advanced Vector Extensions (AVX, AVX2, or AVX-512).Performance

Decision: Select CPUs with Advanced Vector Extensions (AVX, AVX2, or AVX-512).

Rationale: CPUs with support for AVX or AVX2 can improve performance in deep learning tasks by accelerating vector operations.

Implication: Re-purposing available servers might not be a feasible option and overall cost of the solution might increase.

Component: AIR

AIR-TZU-NET-001Set up networking for 100 Gbps or higher if possible.Manageability

Decision: Set up networking for 100 Gbps or higher if possible.

Rationale: 100 Gbps networking provides enough bandwidth and very low latency for inference and fine-tuning use cases backed by vSAN ESA.

Implication: The cost of the solution is increased.

Component: TZU

AIR-TZU-NET-002Add a /28 overlay-backed NSX segment for use by the Supervisor control plane nodes.Manageability

Decision: Add a /28 overlay-backed NSX segment for use by the Supervisor control plane nodes.

Rationale: Supports the Supervisor control plane nodes.

Implication: You must create the overlaybacked NSX segment.

Component: TZU

AIR-TZU-NET-003Use a dedicated /20 subnet for pod networking.Manageability

Decision: Use a dedicated /20 subnet for pod networking.

Rationale: A single /20 subnet is sufficient to meet the design requirement of 2000 pods.

Implication: You must set up a private IP space behind a NAT that you can use in multiple Supervisors.

Component: TZU

AIR-TZU-NET-004Use a dedicated /22 subnet for services.Manageability

Decision: Use a dedicated /22 subnet for services.

Rationale: A single /22 subnet is sufficient to meet the design requirement of 2000 pods.

Implication: Private IP space behind a NAT that you can use in multiple Supervisors.

Component: TZU

AIR-TZU-NET-005Use a dedicated /24 or larger subnet on your corporate network for ingress endpoints.Manageability

Decision: Use a dedicated /24 or larger subnet on your corporate network for ingress endpoints.

Rationale: A /24 subnet is sufficient to meet the design requirement of 2000 pods in most cases.

Implication: This subnet must be routable to the rest of the corporate network. A /24 subnet will be sufficient for most use cases, but you should evaluate your ingress needs before deployment.

Component: TZU

AIR-TZU-NET-006Use a dedicated /24 or larger subnet on your corporate network for egress endpoints.Manageability

Decision: Use a dedicated /24 or larger subnet on your corporate network for egress endpoints.

Rationale: A /24 subnet is sufficient to meet the design requirement of 2000 pods in most cases.

Implication: This subnet must be routable to the rest of the corporate network. A /24 subnet will be sufficient for most use cases, but you should evaluate your egress needs before to deployment.

Component: TZU

AIR-ACCELERATE-001Select GPUs with high memory bandwidthPerformance

Decision: Select GPUs with high memory bandwidth

Rationale: AI workloads require high memory bandwidth to efficiently handle large amounts of data. Look for GPUs with high memory bandwidth specifications.

Implication: • The cost of the solution is increased. • GPU choice might be limited.

Component: AIR

AIR-ACCELERATE-002Select GPUs with large memory capacity.Performance

Decision: Select GPUs with large memory capacity.

Rationale: To handle efficiently LLMs, select GPUs equipped with substantial memory capacities. LLMs containing billions of parameters demand significant GPU memory resources for model fine-tuning and inference.

Implication: • The cost of the solution is increased. • GPU choice might be limited.

Component: AIR

AIR-ACCELERATE-003Evaluate and compare compute performance of the available options of GPUs.Performance

Decision: Evaluate and compare compute performance of the available options of GPUs.

Rationale: Assess the GPU's compute performance based on metrics like CUDA cores (for NVIDIA GPUs) or stream processors (for AMD GPUs). Higher compute performance provide support for faster model training and inference, particularly beneficial for complex AI tasks.

Implication: • The cost of the solution is increased. • GPU choice might be limited.

Component: AIR

AIR-ACCELERATE-004Evaluate cooling and power efficiency of GPUs.Performance

Decision: Evaluate cooling and power efficiency of GPUs.

Rationale: To manage the strain large language models place on GPUs, prioritize systems with efficient cooling and power management to mitigate high power consumption and heat generation.

Implication: You must select server platforms focused on GPU.

Component: AIR

AIR-STORAGE-001Use vSAN ESA with 100 Gbps networking and,if possible, RDMA.SecurityPerformance

Decision: Use vSAN ESA with 100 Gbps networking and,if possible, RDMA.

Rationale: Provides high performance and efficiency. Although the minimum bandwidth for vSAN ESA is 25 Gbps, 100 Gbps and faster provides the best performance in terms of bandwidth and latency for all AI use cases.

Implication: • The cost of the solution is increased. • RDMA increases the design complexity. • The choice of vSAN ReadyNodes is limited to nodes that are approved for use with vSAN ESA.

Component: AIR

AIR-STORAGE-002Use vSAN ESA RAID 5 or RAID 6 erasure coding.Performance

Decision: Use vSAN ESA RAID 5 or RAID 6 erasure coding.

Rationale: Provides performance equal to RAID 1 mirroring.

Implication: None.

Component: AIR

AIR-STORAGE-003Leave data compression enabled for vSAN ESA.Availability

Decision: Leave data compression enabled for vSAN ESA.

Rationale: Enables transmitting data in compressed state across hosts in the cluster. Data compression in vSAN ESA is controllable using storage policies.

Implication: None.

Component: AIR

AIR-SPBM-CFG-001Create a vSphere tag and tag category, and apply the vSphere tag to the vSAN datastore in the…AvailabilityManageability

Decision: Create a vSphere tag and tag category, and apply the vSphere tag to the vSAN datastore in the shared edge and workload vSphere cluster in the VI workload domain.

Rationale: Supervisor activation requires the use of vSphere Storage Based Policy Management (SPBM). To assign the vSAN datastore to the Supervisor, you need to create a vSphere tag and tag category to create an SPBM rule.

Implication: You must perform this operation manually or by using PowerCLI.

Component: SPBM

AIR-SPBM-CFG-002Create a vSphere Storage Policy Based Management (SPBM) policy that specifies the vSphere tag you…Manageability

Decision: Create a vSphere Storage Policy Based Management (SPBM) policy that specifies the vSphere tag you created for the Supervisor.

Rationale: When you create the SPBM policy and define the vSphere tag for the Supervisor, you can then assign that SPBM policy during Supervisor activation.

Implication: You must perform this operation manually or by using PowerCLI.

Component: SPBM

AIR-TZU-CFG-001Activate vSphere with Tanzu on the shared edge and workload vSphere cluster in the VI workload…AvailabilityPerformance

Decision: Activate vSphere with Tanzu on the shared edge and workload vSphere cluster in the VI workload domain.

Rationale: The Supervisor is required to run Kubernetes workloads natively and to deploy Tanzu Kubernetes Grid clusters natively using Tanzu Kubernetes Grid Service.

Implication: Ensure the shared edge and workload vSphere cluster is sized to support the Supervisor control plane, any additional integrated management workloads, and any customer workloads.

Component: TZU

AIR-TZU-CFG-002Deploy the Supervisor with small-size control plane nodes.Performance

Decision: Deploy the Supervisor with small-size control plane nodes.

Rationale: Deploying the control plane nodes as small-size appliances gives you the ability to run up to 2,000 pods within your Supervisor. If your pod count is higher than 2,000 for the Supervisor, you must deploy control plane nodes that can handle that level of scale.

Implication: You must consider the size of the control plane nodes.

Component: TZU

AIR-TZU-CFG-003Use NSX as provider of the software-defined networking for the Supervisor.Manageability

Decision: Use NSX as provider of the software-defined networking for the Supervisor.

Rationale: You can deploy a Supervisor either by using NSX or vSphere networking. VMware Cloud Foundation uses NSX for softwaredefined networking across the SDDC. Deviating for vSphere with Tanzu would increase the operational overhead.

Implication: None.

Component: TZU

AIR-TZU-CFG-004Deploy the NSX Edge cluster with large-size nodes.AvailabilityPerformance

Decision: Deploy the NSX Edge cluster with large-size nodes.

Rationale: Large-size NSX Edge nodes are the smallest size supported to activate a Supervisor.

Implication: You must account for the size of the NSX Edge nodes.

Component: TZU

AIR-TZU-CFG-005Deploy a single-zone Supervisor.Availability

Decision: Deploy a single-zone Supervisor.

Rationale: A three-zone Supervisor requires three separate vSphere clusters.

Implication: No change to existing design or procedures with single-zone Supervisor.

Component: TZU

AIR-TZU-CFG-006Deploy a Tanzu Kubernetes Grid Cluster in the Supervisor.Availability

Decision: Deploy a Tanzu Kubernetes Grid Cluster in the Supervisor.

Rationale: For applications that require upstream Kubernetes compliance, a Tanzu Kubernetes Grid Cluster is required.

Implication: None.

Component: TZU

AIR-TZU-CFG-007For a disconnected environment, configure a local content library for Tanzu Kubernetes releases…Availability

Decision: For a disconnected environment, configure a local content library for Tanzu Kubernetes releases (TKrs) for use in the shared edge and workload vSphere cluster.

Rationale: In a disconnected environment, the Supervisor is unable to pull TKr images from the central public content library maintained by VMware. To deploy a Tanzu Kubernetes Grid on a Supervisor, you must configure a content library in the shared edge and workload vSphere cluster with the required images, downloaded from the public library.

Implication: You must manually configure the content library.

Component: TZU

AIR-TZU-CFG-008Use Antrea as the container network interface (CNI) for your Tanzu Kubernetes Grid clusters.Availability

Decision: Use Antrea as the container network interface (CNI) for your Tanzu Kubernetes Grid clusters.

Rationale: Antrea is the default CNI for Tanzu Kubernetes Grid clusters.

Implication: New Tanzu Kubernetes Grid clusters are deployed with Antrea as the CNI, unless you specify Calico.

Component: TZU

AIR-TZU-CFG-009Deploy Tanzu Kubernetes Grid clusters with a minimum of three control plane nodes.Availability

Decision: Deploy Tanzu Kubernetes Grid clusters with a minimum of three control plane nodes.

Rationale: Deploying three control plane nodes ensures the control plance state of your Tanzu Kubernetes Grid cluster stays if a node failure occurs. Horizontal and vertical scaling of the control plane is supported. See Scale a TKG Cluster on Supervisor Using Kubectl.

Implication: None.

Component: TZU

AIR-TZU-CFG-010For production environments, deploy Tanzu Kubernetes Grid clusters with a minimum of three worker…Availability

Decision: For production environments, deploy Tanzu Kubernetes Grid clusters with a minimum of three worker nodes.

Rationale: Deploying three worker nodes provides a higher level of availability of your workloads deployed to the cluster.

Implication: You must configure your customer workloads to use effectively the additional worker nodes in the cluster for high availability at an application-level.

Component: TZU

AIR-TZU-CFG-011Deploy Tanzu Kubernetes Grid clusters with small-size control plane nodes if your cluster will have…AvailabilityPerformance

Decision: Deploy Tanzu Kubernetes Grid clusters with small-size control plane nodes if your cluster will have less than 10 worker nodes.

Rationale: You must size the control plane of a Tanzu Kubernetes Grid cluster according to the amount of worker nodes and pod density.

Implication: The size of the cluster nodes impacts the scale of a given cluster. If you must add a node to a cluster, consider the use of larger nodes. For AI GPU-enabled workloads, the GPU is the constraining factor for the amount of worker nodes that could be deployed.

Component: TZU

AIR-HRB-CFG-001Deploy Contour as an Ingress Supervisor Service.Manageability

Decision: Deploy Contour as an Ingress Supervisor Service.

Rationale: Harbor requires Contour on the target Supervisor to provide Ingress Service. The Ingress IP address provided by Contour must be resolved to the Harbor FQDN.

Implication: None.

Component: HRB

AIR-HRB-CFG-002Deploy the Harbor Registry as a Supervisor Service.Security

Decision: Deploy the Harbor Registry as a Supervisor Service.

Rationale: Harbor as a Supervisor Service has replaced the integrated registry in previous vSphere versions.

Implication: You must provide the following configuration: • Harbor FQDN • Record and Pointer Record (PTR) for the Harbor Registry IP (this IP is provided by the Contour Ingress Service) • Manage Supervisor Services privilege in vCenter Server.

Component: HRB

AIR-TZU-LCM-001For life cycle management of a GPU-enabled VI workload domain, use a vSphere Lifecycle Manager…PerformanceManageability

Decision: For life cycle management of a GPU-enabled VI workload domain, use a vSphere Lifecycle Manager image with a custom ESXi image that includes the GPU driver and any other core components from the GPU vendor.

Rationale: • Eases maintaining the right host driver versions and daemons. • Introduces consistency across the GPU-enabled hosts.

Implication: You must create the customer vSphere Lifecycle Manager image before you deploy the VI workload domain.

Component: TZU

AIR-TZU-LCM-002Use the vSphere Client for life cycle management of a Supervisor.Manageability

Decision: Use the vSphere Client for life cycle management of a Supervisor.

Rationale: Life cycle management of a Supervisor is not integrated in SDDC Manager.

Implication: You perform deployment, patching, updates, and upgrades of a Supervisor and its components manually.

Component: TZU

AIR-TZU-LCM-003Use kubectl for life cycle management of a Tanzu Kubernetes Grid cluster.AvailabilityManageability

Decision: Use kubectl for life cycle management of a Tanzu Kubernetes Grid cluster.

Rationale: Life cycle management of a Tanzu Kubernetes Grid cluster is not integrated in SDDC Manager.

Implication: You perform deployment, patching, updates, and upgrades of a Tanzu Kubernetes Grid cluster and its components manually.

Component: TZU

AIR-TZU-SEC-001Create a security group in Active Directory for DevOps administrators. Add users who need edit…AvailabilitySecurity

Decision: Create a security group in Active Directory for DevOps administrators. Add users who need edit permissions within a namespace to the group and grant Can Edit permissions to the namespace for that group. If you require different permissions per namespace, create additional groups.

Rationale: Necessary for auditable rolebased access control within the Supervisor and Tanzu Kubernetes Grid clusters.

Implication: You must define and manage security groups, group membership, and security controls in Active Directory.

Component: TZU

AIR-TZU-SEC-002Create a security group in Active Directory for DevOps administrators. Add users who need read-only…AvailabilitySecurity

Decision: Create a security group in Active Directory for DevOps administrators. Add users who need read-only permissions in a namespace to the group, and grant Can View permissions to the namespace for that group. If you require different permissions per namespace, create additional groups.

Rationale: Necessary for auditable rolebased access control within the Supervisor and Tanzu Kubernetes Grid clusters.

Implication: You must define and manage security groups, group membership, and security controls in Active Directory.

Component: TZU

AIR-TZU-SEC-003Replace the default self-signed certificate for the Supervisor management interface with a…Security

Decision: Replace the default self-signed certificate for the Supervisor management interface with a PEM-encoded, CA-signed certificate.

Rationale: Ensures that the communication between administrators and the Supervisor management interface is encrypted by using a trusted certificate.

Implication: You must replace and manager certificates manually, outside certificate management automation of SDDC Manager.

Component: TZU

AIR-TZU-SEC-004Use a SHA-2 or higher algorithm when signing certificates.Security

Decision: Use a SHA-2 or higher algorithm when signing certificates.

Rationale: The SHA-1 algorithm is considered less secure and has been deprecated.

Implication: Not all certificate authorities support SHA-2.

Component: TZU

AIR-NVD-LIC-001For Delegated License Service (DLS) instances, account for extra compute, storage, and network…AvailabilityManageability

Decision: For Delegated License Service (DLS) instances, account for extra compute, storage, and network resources as part of your management domain.

Rationale: DLS is deployed as a virtual appliance with specific hardware requirements. The appliance can also be configured in a high availability setup, independent from vSphere HA.

Implication: • Increased resources for the management stack. • You must perform life cycle management of the DLS instance.

Component: NVD

AIR-NVD-LIC-002For Cloud License Service (CLS) instances, Internet access is required.Manageability

Decision: For Cloud License Service (CLS) instances, Internet access is required.

Rationale: Internet access is required between a licensed client and a CLS instance. Ports 80 and 443 (Egress) must be allowed.

Implication: • Introduces potential security risks. • You must enforce firewall rules, intrusion detection systems, and monitoring.

Component: NVD

AIR-DSM-001Deploy VMware Data Services Manager in the management domain.Availability

Decision: Deploy VMware Data Services Manager in the management domain.

Rationale: A 1:1 relationship between a VMware Data Services Manager appliance and a vCenter Server instance is required. The vCenter Server instances for the VI workload domans in a VMware Cloud Foundation instance run in the management domain.

Implication: You must deploy one VMware Data Services Manager appliance per vCenter Server which impacts the required resources for the management domain and its clusters.

Component: AIR

AIR-DSM-002For production-grade deployments, deploy PostgreSQL databases in HA mode (3 or 5 nodes).Availability

Decision: For production-grade deployments, deploy PostgreSQL databases in HA mode (3 or 5 nodes).

Rationale: High Availability of Vector Databases, increasing the overall availability of the whole system that depends on the DBs.

Implication: Increased resource consumption of the target VI WLD, and increased number used IP Addresses.

Component: AIR

AIR-DSM-003Allocate enough IP addresses for the IP pools of infrastructure policies.AvailabilityManageability

Decision: Allocate enough IP addresses for the IP pools of infrastructure policies.

Rationale: You determine the number of IP addresses reserved for the IP pools according to the requirements and to the high availability topology of the database deployed by using VMware Data Services Manager. For example, a 5node PostgreSQL cluster requires 7 IP addresses - one for each node, one for for kube_VIP, and one for database load balancing).

Implication: You must consider planning and subnet sizing.

Component: AIR

AIR-DSM-004Define VM classes in VMware Data Services Manager that align to your resource requirements.Manageability

Decision: Define VM classes in VMware Data Services Manager that align to your resource requirements.

Rationale: Consider the use case, types of workloads using the databases, amount of data, Transactions per Second (TPS), and other factors, such as target infrastructure overcommitment if applicable. See Data Services Manager Documentation and Data Modernization with VMware Data Services Manager.

Implication: You must consider VMware Data Services Manager planning and design.

Component: AIR

AIR-DSM-005Configure LDAP as Directory Service for VMware Data Services Manager.Security

Decision: Configure LDAP as Directory Service for VMware Data Services Manager.

Rationale: LDAP (TLS available if needed) can be configured as the identity provider to import users and assign roles on VMware Data Services Manager.

Implication: Increased security operation costs. You must allow port access from VMware Data Services Manager to the LDAP identity source: • LDAP - 389 TCP • LDAPS - 636 TCP/UDP

Component: AIR

AIR-DSM-006Configure the S3-compatible object store, for example, MinIO, with TLS.RecoverabilitySecurityManageability

Decision: Configure the S3-compatible object store, for example, MinIO, with TLS.

Rationale: The provider repositories for core VMware Data Services Manager storage, backup, logs and database backups must be enabled with TLS.

Implication: • Security and complexity is increased. • You must manage TLS certificates.

Component: AIR

AIR-DSM-007Create a VMware Tanzu Network account and use it to configure a refresh token in VMware Data…Manageability

Decision: Create a VMware Tanzu Network account and use it to configure a refresh token in VMware Data Services Manager.

Rationale: Database templates and software updates are uploaded to VMware Tanzu Network. In a connected environment, you must configure a Tanzu Network Refresh Token as part of the VMware Data Services Manager setup. In a disconnected environment, you must download the airgapped environment repository and uploaded it manually to the Provider Repository.

Implication: You must perform this operation manually.

Component: AIR

AIR-DSM-008If you plan to run databases managed by VMware Data Services Manager on SAN ESA clusters, create a…AvailabilitySecurityPerformance

Decision: If you plan to run databases managed by VMware Data Services Manager on SAN ESA clusters, create a vSphere SPBM policy that is based on erasure coding.

Rationale: Provides performance that is equivalent to RAID 1 but with no compromises and with better space efficiency. The available erasure coding, RAID 5 or RAID 6, depends on the size of the all-flash vSAN ESA cluster. Erasure Coding 5 RAID 5 erasure coding requires a minimum of 4 ESXi hosts while RADI 6 erasure coding requires a minimum of 6 ESXi hosts.

Implication: • Design complexity, cost, and management overhead of the solution are increased. • You must perform this operation manually or by using PowerCLI.

Component: AIR

AIR-DSM-009Use RAID 5 or RAID 6 erasure coding as the default vSAN storage policy for databases.AvailabilitySecurityPerformance

Decision: Use RAID 5 or RAID 6 erasure coding as the default vSAN storage policy for databases.

Rationale: Eliminates the trade-off of performance and deterministic space efficiency. Set FTT=1 for RAID 5 and FTT=2 for RAID 6 according to the number of hosts in the vSAN ESA cluster and your data availability requirements.

Implication: Design complexity is increased.

Component: AIR

Prerequisites

  • VCF version in Support Matrix (5.1.0-5.2.1)
  • DRI prerequisites (vSphere with Tanzu activated)
  • SR-IOV enabled globally at BIOS/UEFI on all hosts with GPUs
  • GPU-enabled ESXi hosts (vLCM-managed) with compatible NVIDIA GPU (A100, L40S, H100)
  • Supported GPU sharing mode: Time slicing or MIG
  • NSX Edge cluster large-sized
  • AD/DNS/NTP/CA external services (as per IAM + DRI)
  • Workspace ONE Access optional for Service Broker catalog
  • VMware Data Services Manager license (part of VMware Private AI Foundation with NVIDIA)
  • NVIDIA AI Enterprise license (NVAIE) — per-GPU
  • Tanzu Network account for DSM refresh token
  • VMware Aria Automation 8.18.0, deployed manually or with PowerShell (see the Private Cloud Automation for VMware Cloud Foundation validated solution) (VVS 1.0 p.172)

Implementation Procedure

Implementation stages (VVS 1.0 p.173, Table 134):

  1. Plan and prepare the Private AI Ready VMware Cloud Foundation environment: work with the technology team on the physical servers, network and storage; record the details in the VMware Cloud Foundation Planning and Preparation Workbook.
  2. Deploy and configure vSphere with Tanzu on VMware Cloud Foundation: configure vSphere for AI GPU-enabled workloads; deploy a Tanzu Kubernetes Grid cluster on the Supervisor for AI-ready workloads; deploy and configure the NVIDIA Kubernetes Operators.
  3. Add VMware Private AI Foundation with NVIDIA: create self-service catalog items for Service Broker in VMware Aria Automation; deploy and configure VMware Data Services Manager; configure a content library for deep learning VM images; in environments without Internet access, make the deep learning VM images, TKr images and NGC container images available locally; deploy AI workloads and vector databases for RAG workloads from the Automation Service Broker catalog items.

Personas

PersonaResponsibilityComponent role or group
Cloud AdministratorFull administrative access to the vSphere with Tanzu infrastructure: configures and activates Supervisors and namespacesvCenter Server and SDDC Manager: Administrator
DevOps EngineerDeploys vSphere Pods, VMs and Tanzu Kubernetes Grid clusters on namespaces within a SupervisorvSphere namespace: Can edit
Data ScientistDeploys VMs in vSphere with Tanzu through the VM Service and Kubernetes APIAria Automation Service Broker, Tanzu Kubernetes Cluster, vSphere Namespace: Can edit
AuditorRead-only access for security and compliance reviewvSphere namespace: Can view

Source: VVS 1.0 Table 155, Example Personas for Private AI Ready Infrastructure (p.214)

Day-2 Operations Tasks

Operational verification — Supervisor and GPU

After installation or upgrade

Supervisor Running with the Large Edge cluster healthy (AIR-TZU-CFG-001/003); hosts show the NVIDIA vGPU/MIG profiles enumerated (BIOS Global SR-IOV on); a test deep-learning VM from the VM Service class with a GPU attaches successfully.

Operational verification — Data Services Manager

After installation or upgrade

Provision a test PostgreSQL database (HA 3 or 5 nodes for production, AIR-DSM-002) from the infrastructure policy; confirm IP allocation from the IP pools (AIR-DSM-003); delete it.

NVIDIA licensing

Per licence renewal

DLS appliance in the management domain is life-cycled by the operator (AIR-NVD-LIC-001); CLS requires internet egress with firewall and monitoring controls (AIR-NVD-LIC-002); check licence lease status on the GPU VMs.

Certificate management

Before expiry

Replace the Supervisor management-interface certificate with a PEM-encoded CA-signed certificate, SHA-2 or higher (AIR-TZU-SEC-003/004) through the vSphere Client CSR workflow; outside SDDC Manager automation.

Life cycle of the GPU workload domain

Per release

Maintain the vSphere Lifecycle Manager image with the GPU driver and vendor components (AIR-TZU-LCM-001); update the image before host upgrades; Supervisor and TKG clusters update through vSphere with Tanzu and kubectl.

Backup

Scheduled

Velero with the vSphere plug-in for Kubernetes objects; DSM database backups to the configured backup location; Harbor registry images (AIR-HRB-CFG-*).

Capacity

Quarterly

Track GPU utilisation and vSAN ESA capacity (AIR-STORAGE-001); plan IP pools (AIR-DSM-003) and Supervisor control-plane size.

Monitoring Points

  • Monitor the parameters of new or reconfigured components in the VMware Cloud Foundation system; for validated monitoring solutions see the VMware Cloud Foundation Validated Solutions (VVS 1.0 p.224)
  • VMware Cloud Foundation Operations 8.16.1 or later monitors Supervisor clusters, Tanzu Kubernetes clusters and deep learning VMs through its native VMware Cloud Foundation integration (VVS 1.0 p.224)
  • vSphere Client: monitor GPU metrics at cluster level in custom charts (VVS 1.0 p.224)

Troubleshooting

vGPU or MIG profiles do not enumerate on a host
Cause: Global SR-IOV disabled in BIOS or driver missing from the vLCM image (AIR-TZU-LCM-001)
Fix: Enable SR-IOV, remediate the host with the GPU-enabled image.
VM with GPU cannot be placed or vMotioned
Cause: Heterogeneous GPU models in one cluster
Fix: Homogeneous GPUs per cluster; separate clusters per GPU type.
Supervisor activation fails
Cause: NSX Edge nodes smaller than Large
Fix: Redeploy the Edge cluster with Large nodes (AIR-TZU-CFG-003 dependency).
DSM database provisioning stuck waiting for IP
Cause: Infrastructure policy IP pool exhausted (AIR-DSM-003; a 5-node HA PostgreSQL needs 7 IPs)
Fix: Extend the IP pool.
GPU workloads lose licence
Cause: DLS appliance down or CLS egress blocked (AIR-NVD-LIC-001/002)
Fix: Restore the DLS appliance (protect with vSphere HA) or the firewall rule; leases survive for the grace period.

Likely Panelist Questions

Q: Why vSAN ESA with 100 Gbps and RDMA?

AIR-STORAGE-001: AI training and inference I/O profiles need NVMe-class storage throughput; the cost and complexity increase and ReadyNode choice narrows.

Q: DLS or CLS for NVIDIA licensing?

AIR-NVD-LIC-001/002: DLS keeps licensing on-premises at the cost of management-domain resources and life cycle; CLS removes that but needs internet egress with security controls.

Q: Where does Data Services Manager live and how is it sized?

AIR-DSM-001..003: DSM appliance in the management domain, one per vCenter; production PostgreSQL in 3- or 5-node HA; IP pools sized for the HA topology.

Q: Why single-zone Supervisor on NSX?

AIR-TZU-CFG-003/005: NSX is the VCF SDN; a three-zone Supervisor would need three clusters; single-zone keeps the design unchanged.

Failure Scenarios

Global SR-IOV not enabled in BIOS
Impact: vGPU/MIG fail to enumerate; GPU VMs cannot start.
Mitigation: Host build standard and vLCM image (AIR-TZU-LCM-001).
Heterogeneous GPU mix in a cluster
Impact: vMotion and placement failures for GPU VMs.
Mitigation: One GPU model per cluster.
Under-provisioned NSX Edge (not Large)
Impact: Supervisor enablement fails.
Mitigation: Large Edge nodes before activation.
Insufficient IPs in the infrastructure-policy pools
Impact: DSM database provisioning fails (5-node HA needs 7 IPs).
Mitigation: AIR-DSM-003 pool sizing.
DLS licence appliance failure
Impact: GPU VMs lose licences after the grace period.
Mitigation: vSphere HA for the DLS appliance; HA pair where supported (AIR-NVD-LIC-001).

Trade-off Analysis

DSM in the management domain vs a workload domain

Chosen: Management domain (AIR-DSM-001)

Justification: Consistent with the VVS management-plane pattern; one appliance per vCenter adds management-domain resources.

DLS vs CLS licensing

Chosen: DLS (on-premises) where internet egress is restricted; CLS otherwise

Justification: AIR-NVD-LIC-001/002 trade management overhead against egress security risk.

vSAN ESA + RDMA vs standard vSAN

Chosen: ESA with 100 Gbps and RDMA where possible (AIR-STORAGE-001)

Justification: Performance for AI workloads versus cost, complexity and ReadyNode constraints.

Quiz — Private AI Ready Infrastructure

0/15
Q1
Which GPU sharing modes are supported in this solution?
  • Fractional vMotion
  • Time slicing and Multi-Instance GPU (MIG)
  • MIG only
  • Time slicing only
Supported GPU sharing modes: Time slicing and MIG (Multi-Instance GPU).
Q2
What must be enabled in BIOS/UEFI on every GPU host before vGPU/MIG works?
  • Secure Boot
  • Hyper-Threading
  • Global SR-IOV
  • TPM 2.0
Global SR-IOV is required; for MIG, a vGPU is associated with a virtual function at boot time.
Q3
What is the minimum subnet size designed for pod networking in PAI?
  • /22
  • /16
  • /24
  • /20
AIR-TZU-NET-003: /20 subnet for pod networking (supports 2000 pods).
Q4
Which NVIDIA license system deployment is hosted on the NVIDIA Licensing Portal?
  • DLS (Delegated License Service)
  • NVL (NVIDIA Virtual License)
  • NGC (NVIDIA GPU Cloud)
  • CLS (Cloud License Service)
CLS is hosted on the NVIDIA Licensing Portal; DLS is on-prem.
Q5
What is the HA topology required for a production PostgreSQL in DSM per this design?
  • Single node
  • Unlimited
  • 2-node primary/standby
  • 3 or 5 nodes
AIR-DSM-002: deploy PostgreSQL in HA mode (3 or 5 nodes) for production.
Q6
How many IP addresses does a 5-node PostgreSQL DSM cluster require?
  • 10
  • 7
  • 6
  • 5
5 node IPs + 1 kube_VIP + 1 DB LB = 7 IPs (per AIR-DSM-003 example).
Q7
Which GPU architecture example is recommended for middle-range (7-13B parameter) LLMs?
  • NVIDIA V100 (Volta)
  • NVIDIA RTX 4090
  • NVIDIA A100 (Ampere)
  • NVIDIA H100 (Hopper)
A100 (40/80 GB HBM2e) is suited for middle-range LLMs (7-13B) and embedding models per design.
Q8
Which DSM decision requires internet access for package downloads?
  • AIR-DSM-003 (IP pool)
  • AIR-DSM-001 (deploy in mgmt)
  • AIR-DSM-006 (S3 with TLS)
  • AIR-DSM-007 (Tanzu Network refresh token)
AIR-DSM-007: Tanzu Network refresh token required in connected env; air-gapped env requires manual upload of repository.
Q9
What technology combines high-speed GPU-to-GPU communication suitable for generative AI scaling?
  • PCIe only
  • GPUDirect RDMA + NVLink + NVSwitch
  • VMXNET3
  • SR-IOV + vMotion
GPUDirect RDMA (direct GPU-to-GPU via network), NVLink (GPU-GPU bridge), NVSwitch (multi-GPU orchestration).
Q10
Which vSphere feature groups complementary hardware devices connected via PCIe switch/interconnect?
  • Device groups (vSphere 8)
  • DRS VM groups
  • Resource pools
  • VM classes
Device groups in vSphere 8 identify hardware-connected devices as unified entity; recognized by DRS and HA; supported with TKG Service.
Q11
What filesystem/protocol supports ReadWriteMany persistent volumes for TKG model repositories?
  • NFS v3
  • iSCSI
  • vSAN File Service (via CNS file volumes)
  • HDFS
vSphere with Tanzu uses CNS file volumes backed by vSAN File Service for RWX persistent volumes (model repos, archives).
Q12
Limit for file shares per vSAN cluster for DSM/RWX volumes?
  • 50
  • 10
  • 1000
  • 100
Maximum of 100 file shares per vSAN cluster; max file share size = max available vSAN capacity.
Q13
Which feature of vSAN ESA reduces network traffic for AI data transfers?
  • Deduplication
  • Erasure coding
  • Replication
  • Advanced compression at top of stack (enabled by default), data transmitted compressed across network
vSAN ESA advanced compression at top of stack = data compressed before network transfer, optimizing bandwidth and storage.
Q14
Which precision is recommended for LLM inference deployment to optimize resources?
  • FP16 / BF16 / Int8
  • FP4 only
  • FP32 only
  • FP64
Convert from FP32 (training) to FP16/BF16/Int8 (inference) for model size/compute reduction without significant accuracy loss.
Q15
Which persona uses VMware Aria Automation Service Broker with Can Edit for AI workload deployment?
  • Cloud Administrator
  • Auditor
  • DevOps Engineer
  • Data Scientist
Data Scientist persona: Can Edit on Service Broker, TKC, vSphere Namespace — deploys VMs via VM Service and Kubernetes API.

Flashcards — Private AI Ready Infrastructure

Card 1 of 15
Supported NVIDIA GPUs (PAI)
NVIDIA A100, L40S, H100 (Ampere, Ada Lovelace, Hopper architectures).

Labs

Lab 1: Enable GPU Passthrough and Deploy a Deep Learning VM

Configure Global SR-IOV on ESXi host, install NVIDIA vGPU VIB, create vGPU-enabled VM class, and deploy a deep learning VM with PyTorch.

Starting State: VCF 5.2 with VI workload domain; host has NVIDIA A100/L40S/H100 installed; NVAIE license available; content library for deep learning VM images configured; AI Workstation catalog item available in Automation Service Broker; Supervisor activated.

  1. Reboot ESXi host into BIOS/UEFI; enable Global SR-IOV; save and boot
  2. Confirm Global SR-IOV is enabled in the server BIOS/UEFI (vendor tool, for example Dell iDRAC); SR-IOV is required for both MIG and time-slicing vGPU modes (VVS 1.0 p.179)
  3. Install NVIDIA vGPU VIB via vLCM: Import vendor add-on; add to cluster image; remediate host
  4. Verify driver: nvidia-smi in ESXi shell shows GPU
  5. Create PCI passthrough or vGPU profile in vCenter
  6. Create VM class with vGPU device (e.g., A100-40C for 40 GB profile) and ACS Relax advanced param
  7. Deploy the deep learning VM from the AI Workstation catalog item in Automation Service Broker (Consume > Catalog > AI Workstation > Request: GPU-enabled VM class, software bundle, NGC API key) (VVS 1.0 p.209)
  8. Verify the vGPU guest driver installation on the deep learning VM over SSH: tail -f /var/log/nvidia-installer.log (VVS 1.0 p.210)
  9. SSH to VM: nvidia-smi — verify GPU visible
  10. Run PyTorch sample: python -c 'import torch; print(torch.cuda.is_available())' — expect True

Lab 2: Deploy PostgreSQL with pgvector via Data Services Manager for RAG

Configure DSM infrastructure policy, deploy 3-node HA PostgreSQL with pgvector extension, verify connectivity.

Starting State: DSM appliance deployed in management domain; Tanzu Network refresh token configured; S3-compatible bucket with TLS (MinIO) provisioned; LDAP integration complete; Storage Policy and VM Classes defined.

  1. Log in to DSM https://<dsm_fqdn>/login as DSM Admin (LDAP user)
  2. Navigate to Infrastructure Policies > Create; define compute (VM Class), storage (vSAN SPBM), network (allocate IP pool with 7+ IPs per AIR-DSM-003), and VM resources
  3. Databases > Create Database
  4. Engine: Postgres; Version: 15.5+vmware.v2.0.0; Name: paiftest; Replica Mode: Single vSphere Cluster; Topology: 3 (1 Primary, 1 Replica, 1 Monitor)
  5. In the Advanced Settings pane, do not enable the Database Options (pgvector is activated after creation with CREATE EXTENSION vector;) (VVS 1.0 p.216)
  6. Storage Policy = vsan-default; Disk Size = 60 GB
  7. Enable Backups; Backup Location = MinIO S3; Retention = 30 days
  8. Create; monitor until Ready status
  9. Click DB instance; Copy Connection String
  10. Test: psql -h <host> -p 5432 -d paiftest -U pgadmin -W → enter password
  11. In psql: CREATE EXTENSION IF NOT EXISTS vector; verify \dx shows 'vector'

Lab 3: Deploy a RAG Workload in Jupyter Notebook Using pgvector

Deploy a Jupyter Notebook that ingests text documents into a pgvector-backed knowledge base (NASA history books) and queries an LLM via RAG.

Starting State: Deep learning VM running a PyTorch container (Lab 1); optionally the DSM pgvector database from Lab 2; Docker Hub and NVIDIA NGC registry access; 16 vCPU, 128 GB RAM, a GPU with 40 GB+ memory and 100 GB disk available (VVS 1.0 p.216).

  1. If using the DSM database from Lab 2, note its connection string (postgres://pgadmin:<pw>@<host>:5432/paiftest) and split it into DB_USER, DB_PASSWD, DB_NAME and DB_HOST (VVS 1.0 p.217)
  2. Only if not using DSM: in the deep learning VM create a pgvector directory and a docker-compose.yaml for image pgvector/pgvector:pg12, then run sudo docker compose up -d (VVS 1.0 p.217-218)
  3. Only if not using DSM: create a user-defined Docker network (sudo docker network create my-network) and connect the PostgreSQL and PyTorch containers to it (VVS 1.0 p.218)
  4. On your workstation, open an SSH tunnel to the PyTorch deep learning VM: ssh -L 8888:localhost:8888 vmware@<dl_vm_ip>, then browse to http://localhost:8888 for JupyterLab (VVS 1.0 p.219)
  5. In a JupyterLab terminal, clone the reference repository: git clone https://github.com/vmware-private-ai/VMware-generative-ai-reference-architecture.git (VVS 1.0 p.219)
  6. cd to Starter-Packs/Improved_RAG/02-KB-Documents/ and run ./get_NASA_books.sh to download the NASA history books (VVS 1.0 p.220)
  7. Create requirements.txt with the pinned package list from the source (vllm, transformers, llama-index packages, psycopg2-binary) and run pip install -r requirements.txt (VVS 1.0 p.220-221)
  8. In a separate terminal tab, serve the LLM: python -m vllm.entrypoints.openai.api_server --model TheBloke/zephyr-7B-alpha-AWQ --port 8010 --enforce-eager (keep the tab open) (VVS 1.0 p.221)
  9. Open Starter-Packs/Improved_RAG/03-Document_ingestion/Document_ingestion_pipeline.ipynb; under Global Config Setup reduce the VRAM footprint (LLM_MODEL TheBloke/zephyr-7B-alpha-AWQ, EMB_MODEL BAAI/bge-small-en-v1.5, DEVICE cpu, NUM_WORKERS 2) and set DB_HOST and the database values for your pgvector instance (VVS 1.0 p.221-222)
  10. Run all cells; at the final cell verify the LLM answers 'What are the main Hubble telescope discoveries about exoplanets?' using context retrieved from the pgvector store (VVS 1.0 p.222)
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.