Private AI Ready Infrastructure for VMware Cloud Foundation
Design, implementation, and operational guidance for AI GPU-enabled accelerated workload domains running on vSphere with Tanzu. Extends DRI with GPU passthrough/vGPU (NVIDIA A100/L40S/H100), SR-IOV, GPUDirect RDMA, NVLink/NVSwitch support, NVIDIA AI Enterprise (NVAIE) licensing (CLS/DLS), VMware Data Services Manager for database services (PostgreSQL with pgvector, MySQL), and VMware Private AI Foundation with NVIDIA add-on. Supports deep learning VMs, Tanzu Kubernetes Grid clusters running AI containers, and RAG workloads.
Key Components: NSX, SDDC Manager, vCenter, ESXi, Tanzu
External dependencies: VMware Data Services Manager 2.0.2 (part of VMware Private AI Foundation with NVIDIA)
49 design decisions. The Quality column is the Academy's assessment for defence practice; the VVS documents do not classify decisions by design quality (D133).
| DD-ID | Decision | Quality (Academy view) |
|---|---|---|
| AIR-COMPUTE-001 | Select servers with CPUs with a high number of cores. | Performance |
Decision: Select servers with CPUs with a high number of cores. Rationale: To optimize computational efficiency and minimize the need for scaling out by adding more nodes, consider scaling up the CPU core count in each server. By choosing CPUs with a high number of cores, you can effectively handle multiple inference threads simultaneously. This approach maximizes hardware utilization and enhances the capacity to manage parallel tasks, leading to improved performance and resource utilization in inference workloads Implication: High-end CPUs might increase the overall cost of the solution. Component: AIR | ||
| AIR-COMPUTE-002 | Select a fast-access memory. | Performance |
Decision: Select a fast-access memory. Rationale: Minimal latency for data retrieval is crucial for realtime inference applications. Increased latency reduces inference performance and give a poor user experience. Implication: Re-purposing available servers might not be a feasible option and overall cost of the solution might increase. Component: AIR | ||
| AIR-COMPUTE-003 | Select CPUs with Advanced Vector Extensions (AVX, AVX2, or AVX-512). | Performance |
Decision: Select CPUs with Advanced Vector Extensions (AVX, AVX2, or AVX-512). Rationale: CPUs with support for AVX or AVX2 can improve performance in deep learning tasks by accelerating vector operations. Implication: Re-purposing available servers might not be a feasible option and overall cost of the solution might increase. Component: AIR | ||
| AIR-TZU-NET-001 | Set up networking for 100 Gbps or higher if possible. | Manageability |
Decision: Set up networking for 100 Gbps or higher if possible. Rationale: 100 Gbps networking provides enough bandwidth and very low latency for inference and fine-tuning use cases backed by vSAN ESA. Implication: The cost of the solution is increased. Component: TZU | ||
| AIR-TZU-NET-002 | Add a /28 overlay-backed NSX segment for use by the Supervisor control plane nodes. | Manageability |
Decision: Add a /28 overlay-backed NSX segment for use by the Supervisor control plane nodes. Rationale: Supports the Supervisor control plane nodes. Implication: You must create the overlaybacked NSX segment. Component: TZU | ||
| AIR-TZU-NET-003 | Use a dedicated /20 subnet for pod networking. | Manageability |
Decision: Use a dedicated /20 subnet for pod networking. Rationale: A single /20 subnet is sufficient to meet the design requirement of 2000 pods. Implication: You must set up a private IP space behind a NAT that you can use in multiple Supervisors. Component: TZU | ||
| AIR-TZU-NET-004 | Use a dedicated /22 subnet for services. | Manageability |
Decision: Use a dedicated /22 subnet for services. Rationale: A single /22 subnet is sufficient to meet the design requirement of 2000 pods. Implication: Private IP space behind a NAT that you can use in multiple Supervisors. Component: TZU | ||
| AIR-TZU-NET-005 | Use a dedicated /24 or larger subnet on your corporate network for ingress endpoints. | Manageability |
Decision: Use a dedicated /24 or larger subnet on your corporate network for ingress endpoints. Rationale: A /24 subnet is sufficient to meet the design requirement of 2000 pods in most cases. Implication: This subnet must be routable to the rest of the corporate network. A /24 subnet will be sufficient for most use cases, but you should evaluate your ingress needs before deployment. Component: TZU | ||
| AIR-TZU-NET-006 | Use a dedicated /24 or larger subnet on your corporate network for egress endpoints. | Manageability |
Decision: Use a dedicated /24 or larger subnet on your corporate network for egress endpoints. Rationale: A /24 subnet is sufficient to meet the design requirement of 2000 pods in most cases. Implication: This subnet must be routable to the rest of the corporate network. A /24 subnet will be sufficient for most use cases, but you should evaluate your egress needs before to deployment. Component: TZU | ||
| AIR-ACCELERATE-001 | Select GPUs with high memory bandwidth | Performance |
Decision: Select GPUs with high memory bandwidth Rationale: AI workloads require high memory bandwidth to efficiently handle large amounts of data. Look for GPUs with high memory bandwidth specifications. Implication: • The cost of the solution is increased. • GPU choice might be limited. Component: AIR | ||
| AIR-ACCELERATE-002 | Select GPUs with large memory capacity. | Performance |
Decision: Select GPUs with large memory capacity. Rationale: To handle efficiently LLMs, select GPUs equipped with substantial memory capacities. LLMs containing billions of parameters demand significant GPU memory resources for model fine-tuning and inference. Implication: • The cost of the solution is increased. • GPU choice might be limited. Component: AIR | ||
| AIR-ACCELERATE-003 | Evaluate and compare compute performance of the available options of GPUs. | Performance |
Decision: Evaluate and compare compute performance of the available options of GPUs. Rationale: Assess the GPU's compute performance based on metrics like CUDA cores (for NVIDIA GPUs) or stream processors (for AMD GPUs). Higher compute performance provide support for faster model training and inference, particularly beneficial for complex AI tasks. Implication: • The cost of the solution is increased. • GPU choice might be limited. Component: AIR | ||
| AIR-ACCELERATE-004 | Evaluate cooling and power efficiency of GPUs. | Performance |
Decision: Evaluate cooling and power efficiency of GPUs. Rationale: To manage the strain large language models place on GPUs, prioritize systems with efficient cooling and power management to mitigate high power consumption and heat generation. Implication: You must select server platforms focused on GPU. Component: AIR | ||
| AIR-STORAGE-001 | Use vSAN ESA with 100 Gbps networking and,if possible, RDMA. | SecurityPerformance |
Decision: Use vSAN ESA with 100 Gbps networking and,if possible, RDMA. Rationale: Provides high performance and efficiency. Although the minimum bandwidth for vSAN ESA is 25 Gbps, 100 Gbps and faster provides the best performance in terms of bandwidth and latency for all AI use cases. Implication: • The cost of the solution is increased. • RDMA increases the design complexity. • The choice of vSAN ReadyNodes is limited to nodes that are approved for use with vSAN ESA. Component: AIR | ||
| AIR-STORAGE-002 | Use vSAN ESA RAID 5 or RAID 6 erasure coding. | Performance |
Decision: Use vSAN ESA RAID 5 or RAID 6 erasure coding. Rationale: Provides performance equal to RAID 1 mirroring. Implication: None. Component: AIR | ||
| AIR-STORAGE-003 | Leave data compression enabled for vSAN ESA. | Availability |
Decision: Leave data compression enabled for vSAN ESA. Rationale: Enables transmitting data in compressed state across hosts in the cluster. Data compression in vSAN ESA is controllable using storage policies. Implication: None. Component: AIR | ||
| AIR-SPBM-CFG-001 | Create a vSphere tag and tag category, and apply the vSphere tag to the vSAN datastore in the… | AvailabilityManageability |
Decision: Create a vSphere tag and tag category, and apply the vSphere tag to the vSAN datastore in the shared edge and workload vSphere cluster in the VI workload domain. Rationale: Supervisor activation requires the use of vSphere Storage Based Policy Management (SPBM). To assign the vSAN datastore to the Supervisor, you need to create a vSphere tag and tag category to create an SPBM rule. Implication: You must perform this operation manually or by using PowerCLI. Component: SPBM | ||
| AIR-SPBM-CFG-002 | Create a vSphere Storage Policy Based Management (SPBM) policy that specifies the vSphere tag you… | Manageability |
Decision: Create a vSphere Storage Policy Based Management (SPBM) policy that specifies the vSphere tag you created for the Supervisor. Rationale: When you create the SPBM policy and define the vSphere tag for the Supervisor, you can then assign that SPBM policy during Supervisor activation. Implication: You must perform this operation manually or by using PowerCLI. Component: SPBM | ||
| AIR-TZU-CFG-001 | Activate vSphere with Tanzu on the shared edge and workload vSphere cluster in the VI workload… | AvailabilityPerformance |
Decision: Activate vSphere with Tanzu on the shared edge and workload vSphere cluster in the VI workload domain. Rationale: The Supervisor is required to run Kubernetes workloads natively and to deploy Tanzu Kubernetes Grid clusters natively using Tanzu Kubernetes Grid Service. Implication: Ensure the shared edge and workload vSphere cluster is sized to support the Supervisor control plane, any additional integrated management workloads, and any customer workloads. Component: TZU | ||
| AIR-TZU-CFG-002 | Deploy the Supervisor with small-size control plane nodes. | Performance |
Decision: Deploy the Supervisor with small-size control plane nodes. Rationale: Deploying the control plane nodes as small-size appliances gives you the ability to run up to 2,000 pods within your Supervisor. If your pod count is higher than 2,000 for the Supervisor, you must deploy control plane nodes that can handle that level of scale. Implication: You must consider the size of the control plane nodes. Component: TZU | ||
| AIR-TZU-CFG-003 | Use NSX as provider of the software-defined networking for the Supervisor. | Manageability |
Decision: Use NSX as provider of the software-defined networking for the Supervisor. Rationale: You can deploy a Supervisor either by using NSX or vSphere networking. VMware Cloud Foundation uses NSX for softwaredefined networking across the SDDC. Deviating for vSphere with Tanzu would increase the operational overhead. Implication: None. Component: TZU | ||
| AIR-TZU-CFG-004 | Deploy the NSX Edge cluster with large-size nodes. | AvailabilityPerformance |
Decision: Deploy the NSX Edge cluster with large-size nodes. Rationale: Large-size NSX Edge nodes are the smallest size supported to activate a Supervisor. Implication: You must account for the size of the NSX Edge nodes. Component: TZU | ||
| AIR-TZU-CFG-005 | Deploy a single-zone Supervisor. | Availability |
Decision: Deploy a single-zone Supervisor. Rationale: A three-zone Supervisor requires three separate vSphere clusters. Implication: No change to existing design or procedures with single-zone Supervisor. Component: TZU | ||
| AIR-TZU-CFG-006 | Deploy a Tanzu Kubernetes Grid Cluster in the Supervisor. | Availability |
Decision: Deploy a Tanzu Kubernetes Grid Cluster in the Supervisor. Rationale: For applications that require upstream Kubernetes compliance, a Tanzu Kubernetes Grid Cluster is required. Implication: None. Component: TZU | ||
| AIR-TZU-CFG-007 | For a disconnected environment, configure a local content library for Tanzu Kubernetes releases… | Availability |
Decision: For a disconnected environment, configure a local content library for Tanzu Kubernetes releases (TKrs) for use in the shared edge and workload vSphere cluster. Rationale: In a disconnected environment, the Supervisor is unable to pull TKr images from the central public content library maintained by VMware. To deploy a Tanzu Kubernetes Grid on a Supervisor, you must configure a content library in the shared edge and workload vSphere cluster with the required images, downloaded from the public library. Implication: You must manually configure the content library. Component: TZU | ||
| AIR-TZU-CFG-008 | Use Antrea as the container network interface (CNI) for your Tanzu Kubernetes Grid clusters. | Availability |
Decision: Use Antrea as the container network interface (CNI) for your Tanzu Kubernetes Grid clusters. Rationale: Antrea is the default CNI for Tanzu Kubernetes Grid clusters. Implication: New Tanzu Kubernetes Grid clusters are deployed with Antrea as the CNI, unless you specify Calico. Component: TZU | ||
| AIR-TZU-CFG-009 | Deploy Tanzu Kubernetes Grid clusters with a minimum of three control plane nodes. | Availability |
Decision: Deploy Tanzu Kubernetes Grid clusters with a minimum of three control plane nodes. Rationale: Deploying three control plane nodes ensures the control plance state of your Tanzu Kubernetes Grid cluster stays if a node failure occurs. Horizontal and vertical scaling of the control plane is supported. See Scale a TKG Cluster on Supervisor Using Kubectl. Implication: None. Component: TZU | ||
| AIR-TZU-CFG-010 | For production environments, deploy Tanzu Kubernetes Grid clusters with a minimum of three worker… | Availability |
Decision: For production environments, deploy Tanzu Kubernetes Grid clusters with a minimum of three worker nodes. Rationale: Deploying three worker nodes provides a higher level of availability of your workloads deployed to the cluster. Implication: You must configure your customer workloads to use effectively the additional worker nodes in the cluster for high availability at an application-level. Component: TZU | ||
| AIR-TZU-CFG-011 | Deploy Tanzu Kubernetes Grid clusters with small-size control plane nodes if your cluster will have… | AvailabilityPerformance |
Decision: Deploy Tanzu Kubernetes Grid clusters with small-size control plane nodes if your cluster will have less than 10 worker nodes. Rationale: You must size the control plane of a Tanzu Kubernetes Grid cluster according to the amount of worker nodes and pod density. Implication: The size of the cluster nodes impacts the scale of a given cluster. If you must add a node to a cluster, consider the use of larger nodes. For AI GPU-enabled workloads, the GPU is the constraining factor for the amount of worker nodes that could be deployed. Component: TZU | ||
| AIR-HRB-CFG-001 | Deploy Contour as an Ingress Supervisor Service. | Manageability |
Decision: Deploy Contour as an Ingress Supervisor Service. Rationale: Harbor requires Contour on the target Supervisor to provide Ingress Service. The Ingress IP address provided by Contour must be resolved to the Harbor FQDN. Implication: None. Component: HRB | ||
| AIR-HRB-CFG-002 | Deploy the Harbor Registry as a Supervisor Service. | Security |
Decision: Deploy the Harbor Registry as a Supervisor Service. Rationale: Harbor as a Supervisor Service has replaced the integrated registry in previous vSphere versions. Implication: You must provide the following configuration: • Harbor FQDN • Record and Pointer Record (PTR) for the Harbor Registry IP (this IP is provided by the Contour Ingress Service) • Manage Supervisor Services privilege in vCenter Server. Component: HRB | ||
| AIR-TZU-LCM-001 | For life cycle management of a GPU-enabled VI workload domain, use a vSphere Lifecycle Manager… | PerformanceManageability |
Decision: For life cycle management of a GPU-enabled VI workload domain, use a vSphere Lifecycle Manager image with a custom ESXi image that includes the GPU driver and any other core components from the GPU vendor. Rationale: • Eases maintaining the right host driver versions and daemons. • Introduces consistency across the GPU-enabled hosts. Implication: You must create the customer vSphere Lifecycle Manager image before you deploy the VI workload domain. Component: TZU | ||
| AIR-TZU-LCM-002 | Use the vSphere Client for life cycle management of a Supervisor. | Manageability |
Decision: Use the vSphere Client for life cycle management of a Supervisor. Rationale: Life cycle management of a Supervisor is not integrated in SDDC Manager. Implication: You perform deployment, patching, updates, and upgrades of a Supervisor and its components manually. Component: TZU | ||
| AIR-TZU-LCM-003 | Use kubectl for life cycle management of a Tanzu Kubernetes Grid cluster. | AvailabilityManageability |
Decision: Use kubectl for life cycle management of a Tanzu Kubernetes Grid cluster. Rationale: Life cycle management of a Tanzu Kubernetes Grid cluster is not integrated in SDDC Manager. Implication: You perform deployment, patching, updates, and upgrades of a Tanzu Kubernetes Grid cluster and its components manually. Component: TZU | ||
| AIR-TZU-SEC-001 | Create a security group in Active Directory for DevOps administrators. Add users who need edit… | AvailabilitySecurity |
Decision: Create a security group in Active Directory for DevOps administrators. Add users who need edit permissions within a namespace to the group and grant Can Edit permissions to the namespace for that group. If you require different permissions per namespace, create additional groups. Rationale: Necessary for auditable rolebased access control within the Supervisor and Tanzu Kubernetes Grid clusters. Implication: You must define and manage security groups, group membership, and security controls in Active Directory. Component: TZU | ||
| AIR-TZU-SEC-002 | Create a security group in Active Directory for DevOps administrators. Add users who need read-only… | AvailabilitySecurity |
Decision: Create a security group in Active Directory for DevOps administrators. Add users who need read-only permissions in a namespace to the group, and grant Can View permissions to the namespace for that group. If you require different permissions per namespace, create additional groups. Rationale: Necessary for auditable rolebased access control within the Supervisor and Tanzu Kubernetes Grid clusters. Implication: You must define and manage security groups, group membership, and security controls in Active Directory. Component: TZU | ||
| AIR-TZU-SEC-003 | Replace the default self-signed certificate for the Supervisor management interface with a… | Security |
Decision: Replace the default self-signed certificate for the Supervisor management interface with a PEM-encoded, CA-signed certificate. Rationale: Ensures that the communication between administrators and the Supervisor management interface is encrypted by using a trusted certificate. Implication: You must replace and manager certificates manually, outside certificate management automation of SDDC Manager. Component: TZU | ||
| AIR-TZU-SEC-004 | Use a SHA-2 or higher algorithm when signing certificates. | Security |
Decision: Use a SHA-2 or higher algorithm when signing certificates. Rationale: The SHA-1 algorithm is considered less secure and has been deprecated. Implication: Not all certificate authorities support SHA-2. Component: TZU | ||
| AIR-NVD-LIC-001 | For Delegated License Service (DLS) instances, account for extra compute, storage, and network… | AvailabilityManageability |
Decision: For Delegated License Service (DLS) instances, account for extra compute, storage, and network resources as part of your management domain. Rationale: DLS is deployed as a virtual appliance with specific hardware requirements. The appliance can also be configured in a high availability setup, independent from vSphere HA. Implication: • Increased resources for the management stack. • You must perform life cycle management of the DLS instance. Component: NVD | ||
| AIR-NVD-LIC-002 | For Cloud License Service (CLS) instances, Internet access is required. | Manageability |
Decision: For Cloud License Service (CLS) instances, Internet access is required. Rationale: Internet access is required between a licensed client and a CLS instance. Ports 80 and 443 (Egress) must be allowed. Implication: • Introduces potential security risks. • You must enforce firewall rules, intrusion detection systems, and monitoring. Component: NVD | ||
| AIR-DSM-001 | Deploy VMware Data Services Manager in the management domain. | Availability |
Decision: Deploy VMware Data Services Manager in the management domain. Rationale: A 1:1 relationship between a VMware Data Services Manager appliance and a vCenter Server instance is required. The vCenter Server instances for the VI workload domans in a VMware Cloud Foundation instance run in the management domain. Implication: You must deploy one VMware Data Services Manager appliance per vCenter Server which impacts the required resources for the management domain and its clusters. Component: AIR | ||
| AIR-DSM-002 | For production-grade deployments, deploy PostgreSQL databases in HA mode (3 or 5 nodes). | Availability |
Decision: For production-grade deployments, deploy PostgreSQL databases in HA mode (3 or 5 nodes). Rationale: High Availability of Vector Databases, increasing the overall availability of the whole system that depends on the DBs. Implication: Increased resource consumption of the target VI WLD, and increased number used IP Addresses. Component: AIR | ||
| AIR-DSM-003 | Allocate enough IP addresses for the IP pools of infrastructure policies. | AvailabilityManageability |
Decision: Allocate enough IP addresses for the IP pools of infrastructure policies. Rationale: You determine the number of IP addresses reserved for the IP pools according to the requirements and to the high availability topology of the database deployed by using VMware Data Services Manager. For example, a 5node PostgreSQL cluster requires 7 IP addresses - one for each node, one for for kube_VIP, and one for database load balancing). Implication: You must consider planning and subnet sizing. Component: AIR | ||
| AIR-DSM-004 | Define VM classes in VMware Data Services Manager that align to your resource requirements. | Manageability |
Decision: Define VM classes in VMware Data Services Manager that align to your resource requirements. Rationale: Consider the use case, types of workloads using the databases, amount of data, Transactions per Second (TPS), and other factors, such as target infrastructure overcommitment if applicable. See Data Services Manager Documentation and Data Modernization with VMware Data Services Manager. Implication: You must consider VMware Data Services Manager planning and design. Component: AIR | ||
| AIR-DSM-005 | Configure LDAP as Directory Service for VMware Data Services Manager. | Security |
Decision: Configure LDAP as Directory Service for VMware Data Services Manager. Rationale: LDAP (TLS available if needed) can be configured as the identity provider to import users and assign roles on VMware Data Services Manager. Implication: Increased security operation costs. You must allow port access from VMware Data Services Manager to the LDAP identity source: • LDAP - 389 TCP • LDAPS - 636 TCP/UDP Component: AIR | ||
| AIR-DSM-006 | Configure the S3-compatible object store, for example, MinIO, with TLS. | RecoverabilitySecurityManageability |
Decision: Configure the S3-compatible object store, for example, MinIO, with TLS. Rationale: The provider repositories for core VMware Data Services Manager storage, backup, logs and database backups must be enabled with TLS. Implication: • Security and complexity is increased. • You must manage TLS certificates. Component: AIR | ||
| AIR-DSM-007 | Create a VMware Tanzu Network account and use it to configure a refresh token in VMware Data… | Manageability |
Decision: Create a VMware Tanzu Network account and use it to configure a refresh token in VMware Data Services Manager. Rationale: Database templates and software updates are uploaded to VMware Tanzu Network. In a connected environment, you must configure a Tanzu Network Refresh Token as part of the VMware Data Services Manager setup. In a disconnected environment, you must download the airgapped environment repository and uploaded it manually to the Provider Repository. Implication: You must perform this operation manually. Component: AIR | ||
| AIR-DSM-008 | If you plan to run databases managed by VMware Data Services Manager on SAN ESA clusters, create a… | AvailabilitySecurityPerformance |
Decision: If you plan to run databases managed by VMware Data Services Manager on SAN ESA clusters, create a vSphere SPBM policy that is based on erasure coding. Rationale: Provides performance that is equivalent to RAID 1 but with no compromises and with better space efficiency. The available erasure coding, RAID 5 or RAID 6, depends on the size of the all-flash vSAN ESA cluster. Erasure Coding 5 RAID 5 erasure coding requires a minimum of 4 ESXi hosts while RADI 6 erasure coding requires a minimum of 6 ESXi hosts. Implication: • Design complexity, cost, and management overhead of the solution are increased. • You must perform this operation manually or by using PowerCLI. Component: AIR | ||
| AIR-DSM-009 | Use RAID 5 or RAID 6 erasure coding as the default vSAN storage policy for databases. | AvailabilitySecurityPerformance |
Decision: Use RAID 5 or RAID 6 erasure coding as the default vSAN storage policy for databases. Rationale: Eliminates the trade-off of performance and deterministic space efficiency. Set FTT=1 for RAID 5 and FTT=2 for RAID 6 according to the number of hosts in the vSAN ESA cluster and your data availability requirements. Implication: Design complexity is increased. Component: AIR | ||
Prerequisites
- VCF version in Support Matrix (5.1.0-5.2.1)
- DRI prerequisites (vSphere with Tanzu activated)
- SR-IOV enabled globally at BIOS/UEFI on all hosts with GPUs
- GPU-enabled ESXi hosts (vLCM-managed) with compatible NVIDIA GPU (A100, L40S, H100)
- Supported GPU sharing mode: Time slicing or MIG
- NSX Edge cluster large-sized
- AD/DNS/NTP/CA external services (as per IAM + DRI)
- Workspace ONE Access optional for Service Broker catalog
- VMware Data Services Manager license (part of VMware Private AI Foundation with NVIDIA)
- NVIDIA AI Enterprise license (NVAIE) — per-GPU
- Tanzu Network account for DSM refresh token
- VMware Aria Automation 8.18.0, deployed manually or with PowerShell (see the Private Cloud Automation for VMware Cloud Foundation validated solution) (VVS 1.0 p.172)
Implementation Procedure
Implementation stages (VVS 1.0 p.173, Table 134):
- Plan and prepare the Private AI Ready VMware Cloud Foundation environment: work with the technology team on the physical servers, network and storage; record the details in the VMware Cloud Foundation Planning and Preparation Workbook.
- Deploy and configure vSphere with Tanzu on VMware Cloud Foundation: configure vSphere for AI GPU-enabled workloads; deploy a Tanzu Kubernetes Grid cluster on the Supervisor for AI-ready workloads; deploy and configure the NVIDIA Kubernetes Operators.
- Add VMware Private AI Foundation with NVIDIA: create self-service catalog items for Service Broker in VMware Aria Automation; deploy and configure VMware Data Services Manager; configure a content library for deep learning VM images; in environments without Internet access, make the deep learning VM images, TKr images and NGC container images available locally; deploy AI workloads and vector databases for RAG workloads from the Automation Service Broker catalog items.
Personas
| Persona | Responsibility | Component role or group |
|---|---|---|
| Cloud Administrator | Full administrative access to the vSphere with Tanzu infrastructure: configures and activates Supervisors and namespaces | vCenter Server and SDDC Manager: Administrator |
| DevOps Engineer | Deploys vSphere Pods, VMs and Tanzu Kubernetes Grid clusters on namespaces within a Supervisor | vSphere namespace: Can edit |
| Data Scientist | Deploys VMs in vSphere with Tanzu through the VM Service and Kubernetes API | Aria Automation Service Broker, Tanzu Kubernetes Cluster, vSphere Namespace: Can edit |
| Auditor | Read-only access for security and compliance review | vSphere namespace: Can view |
Day-2 Operations Tasks
Operational verification — Supervisor and GPU
After installation or upgradeSupervisor Running with the Large Edge cluster healthy (AIR-TZU-CFG-001/003); hosts show the NVIDIA vGPU/MIG profiles enumerated (BIOS Global SR-IOV on); a test deep-learning VM from the VM Service class with a GPU attaches successfully.
Operational verification — Data Services Manager
After installation or upgradeProvision a test PostgreSQL database (HA 3 or 5 nodes for production, AIR-DSM-002) from the infrastructure policy; confirm IP allocation from the IP pools (AIR-DSM-003); delete it.
NVIDIA licensing
Per licence renewalDLS appliance in the management domain is life-cycled by the operator (AIR-NVD-LIC-001); CLS requires internet egress with firewall and monitoring controls (AIR-NVD-LIC-002); check licence lease status on the GPU VMs.
Certificate management
Before expiryReplace the Supervisor management-interface certificate with a PEM-encoded CA-signed certificate, SHA-2 or higher (AIR-TZU-SEC-003/004) through the vSphere Client CSR workflow; outside SDDC Manager automation.
Life cycle of the GPU workload domain
Per releaseMaintain the vSphere Lifecycle Manager image with the GPU driver and vendor components (AIR-TZU-LCM-001); update the image before host upgrades; Supervisor and TKG clusters update through vSphere with Tanzu and kubectl.
Backup
ScheduledVelero with the vSphere plug-in for Kubernetes objects; DSM database backups to the configured backup location; Harbor registry images (AIR-HRB-CFG-*).
Capacity
QuarterlyTrack GPU utilisation and vSAN ESA capacity (AIR-STORAGE-001); plan IP pools (AIR-DSM-003) and Supervisor control-plane size.
Monitoring Points
- Monitor the parameters of new or reconfigured components in the VMware Cloud Foundation system; for validated monitoring solutions see the VMware Cloud Foundation Validated Solutions (VVS 1.0 p.224)
- VMware Cloud Foundation Operations 8.16.1 or later monitors Supervisor clusters, Tanzu Kubernetes clusters and deep learning VMs through its native VMware Cloud Foundation integration (VVS 1.0 p.224)
- vSphere Client: monitor GPU metrics at cluster level in custom charts (VVS 1.0 p.224)
Troubleshooting
Likely Panelist Questions
Q: Why vSAN ESA with 100 Gbps and RDMA?
AIR-STORAGE-001: AI training and inference I/O profiles need NVMe-class storage throughput; the cost and complexity increase and ReadyNode choice narrows.
Q: DLS or CLS for NVIDIA licensing?
AIR-NVD-LIC-001/002: DLS keeps licensing on-premises at the cost of management-domain resources and life cycle; CLS removes that but needs internet egress with security controls.
Q: Where does Data Services Manager live and how is it sized?
AIR-DSM-001..003: DSM appliance in the management domain, one per vCenter; production PostgreSQL in 3- or 5-node HA; IP pools sized for the HA topology.
Q: Why single-zone Supervisor on NSX?
AIR-TZU-CFG-003/005: NSX is the VCF SDN; a three-zone Supervisor would need three clusters; single-zone keeps the design unchanged.
Failure Scenarios
Trade-off Analysis
DSM in the management domain vs a workload domain
Chosen: Management domain (AIR-DSM-001)
Justification: Consistent with the VVS management-plane pattern; one appliance per vCenter adds management-domain resources.
DLS vs CLS licensing
Chosen: DLS (on-premises) where internet egress is restricted; CLS otherwise
Justification: AIR-NVD-LIC-001/002 trade management overhead against egress security risk.
vSAN ESA + RDMA vs standard vSAN
Chosen: ESA with 100 Gbps and RDMA where possible (AIR-STORAGE-001)
Justification: Performance for AI workloads versus cost, complexity and ReadyNode constraints.
Quiz — Private AI Ready Infrastructure
- Fractional vMotion
- Time slicing and Multi-Instance GPU (MIG)
- MIG only
- Time slicing only
- Secure Boot
- Hyper-Threading
- Global SR-IOV
- TPM 2.0
- /22
- /16
- /24
- /20
- DLS (Delegated License Service)
- NVL (NVIDIA Virtual License)
- NGC (NVIDIA GPU Cloud)
- CLS (Cloud License Service)
- Single node
- Unlimited
- 2-node primary/standby
- 3 or 5 nodes
- 10
- 7
- 6
- 5
- NVIDIA V100 (Volta)
- NVIDIA RTX 4090
- NVIDIA A100 (Ampere)
- NVIDIA H100 (Hopper)
- AIR-DSM-003 (IP pool)
- AIR-DSM-001 (deploy in mgmt)
- AIR-DSM-006 (S3 with TLS)
- AIR-DSM-007 (Tanzu Network refresh token)
- PCIe only
- GPUDirect RDMA + NVLink + NVSwitch
- VMXNET3
- SR-IOV + vMotion
- Device groups (vSphere 8)
- DRS VM groups
- Resource pools
- VM classes
- NFS v3
- iSCSI
- vSAN File Service (via CNS file volumes)
- HDFS
- 50
- 10
- 1000
- 100
- Deduplication
- Erasure coding
- Replication
- Advanced compression at top of stack (enabled by default), data transmitted compressed across network
- FP16 / BF16 / Int8
- FP4 only
- FP32 only
- FP64
- Cloud Administrator
- Auditor
- DevOps Engineer
- Data Scientist
Flashcards — Private AI Ready Infrastructure
Labs
Lab 1: Enable GPU Passthrough and Deploy a Deep Learning VM
Configure Global SR-IOV on ESXi host, install NVIDIA vGPU VIB, create vGPU-enabled VM class, and deploy a deep learning VM with PyTorch.
Starting State: VCF 5.2 with VI workload domain; host has NVIDIA A100/L40S/H100 installed; NVAIE license available; content library for deep learning VM images configured; AI Workstation catalog item available in Automation Service Broker; Supervisor activated.
- Reboot ESXi host into BIOS/UEFI; enable Global SR-IOV; save and boot
- Confirm Global SR-IOV is enabled in the server BIOS/UEFI (vendor tool, for example Dell iDRAC); SR-IOV is required for both MIG and time-slicing vGPU modes (VVS 1.0 p.179)
- Install NVIDIA vGPU VIB via vLCM: Import vendor add-on; add to cluster image; remediate host
- Verify driver: nvidia-smi in ESXi shell shows GPU
- Create PCI passthrough or vGPU profile in vCenter
- Create VM class with vGPU device (e.g., A100-40C for 40 GB profile) and ACS Relax advanced param
- Deploy the deep learning VM from the AI Workstation catalog item in Automation Service Broker (Consume > Catalog > AI Workstation > Request: GPU-enabled VM class, software bundle, NGC API key) (VVS 1.0 p.209)
- Verify the vGPU guest driver installation on the deep learning VM over SSH: tail -f /var/log/nvidia-installer.log (VVS 1.0 p.210)
- SSH to VM: nvidia-smi — verify GPU visible
- Run PyTorch sample: python -c 'import torch; print(torch.cuda.is_available())' — expect True
Lab 2: Deploy PostgreSQL with pgvector via Data Services Manager for RAG
Configure DSM infrastructure policy, deploy 3-node HA PostgreSQL with pgvector extension, verify connectivity.
Starting State: DSM appliance deployed in management domain; Tanzu Network refresh token configured; S3-compatible bucket with TLS (MinIO) provisioned; LDAP integration complete; Storage Policy and VM Classes defined.
- Log in to DSM https://<dsm_fqdn>/login as DSM Admin (LDAP user)
- Navigate to Infrastructure Policies > Create; define compute (VM Class), storage (vSAN SPBM), network (allocate IP pool with 7+ IPs per AIR-DSM-003), and VM resources
- Databases > Create Database
- Engine: Postgres; Version: 15.5+vmware.v2.0.0; Name: paiftest; Replica Mode: Single vSphere Cluster; Topology: 3 (1 Primary, 1 Replica, 1 Monitor)
- In the Advanced Settings pane, do not enable the Database Options (pgvector is activated after creation with CREATE EXTENSION vector;) (VVS 1.0 p.216)
- Storage Policy = vsan-default; Disk Size = 60 GB
- Enable Backups; Backup Location = MinIO S3; Retention = 30 days
- Create; monitor until Ready status
- Click DB instance; Copy Connection String
- Test: psql -h <host> -p 5432 -d paiftest -U pgadmin -W → enter password
- In psql: CREATE EXTENSION IF NOT EXISTS vector; verify \dx shows 'vector'
Lab 3: Deploy a RAG Workload in Jupyter Notebook Using pgvector
Deploy a Jupyter Notebook that ingests text documents into a pgvector-backed knowledge base (NASA history books) and queries an LLM via RAG.
Starting State: Deep learning VM running a PyTorch container (Lab 1); optionally the DSM pgvector database from Lab 2; Docker Hub and NVIDIA NGC registry access; 16 vCPU, 128 GB RAM, a GPU with 40 GB+ memory and 100 GB disk available (VVS 1.0 p.216).
- If using the DSM database from Lab 2, note its connection string (postgres://pgadmin:<pw>@<host>:5432/paiftest) and split it into DB_USER, DB_PASSWD, DB_NAME and DB_HOST (VVS 1.0 p.217)
- Only if not using DSM: in the deep learning VM create a pgvector directory and a docker-compose.yaml for image pgvector/pgvector:pg12, then run sudo docker compose up -d (VVS 1.0 p.217-218)
- Only if not using DSM: create a user-defined Docker network (sudo docker network create my-network) and connect the PostgreSQL and PyTorch containers to it (VVS 1.0 p.218)
- On your workstation, open an SSH tunnel to the PyTorch deep learning VM: ssh -L 8888:localhost:8888 vmware@<dl_vm_ip>, then browse to http://localhost:8888 for JupyterLab (VVS 1.0 p.219)
- In a JupyterLab terminal, clone the reference repository: git clone https://github.com/vmware-private-ai/VMware-generative-ai-reference-architecture.git (VVS 1.0 p.219)
- cd to Starter-Packs/Improved_RAG/02-KB-Documents/ and run ./get_NASA_books.sh to download the NASA history books (VVS 1.0 p.220)
- Create requirements.txt with the pinned package list from the source (vllm, transformers, llama-index packages, psycopg2-binary) and run pip install -r requirements.txt (VVS 1.0 p.220-221)
- In a separate terminal tab, serve the LLM: python -m vllm.entrypoints.openai.api_server --model TheBloke/zephyr-7B-alpha-AWQ --port 8010 --enforce-eager (keep the tab open) (VVS 1.0 p.221)
- Open Starter-Packs/Improved_RAG/03-Document_ingestion/Document_ingestion_pipeline.ipynb; under Global Config Setup reduce the VRAM footprint (LLM_MODEL TheBloke/zephyr-7B-alpha-AWQ, EMB_MODEL BAAI/bge-small-en-v1.5, DEVICE cpu, NUM_WORKERS 2) and set DB_HOST and the database values for your pgvector instance (VVS 1.0 p.221-222)
- Run all cells; at the final cell verify the LLM answers 'What are the main Hubble telescope discoveries about exoplanets?' using context retrieved from the pgvector store (VVS 1.0 p.222)