Academy/VCP-VCF 9.0 Administrator (2V0-17.25)/vLCM Cluster Image Build, Import & Staged Upgrade (VCF 9.0)
This lab targets VCF 9.0

vLCM Cluster Image Build, Import & Staged Upgrade (VCF 9.0)

VCF 9.0Advancedadminarchitectvcdx⏱ 180 min

Images are created in the vSphere Client and imported in VCF Operations (Fleet Management > Lifecycle > VCF Instances > <instance> > Image Management > Import Image). The image must use the ESX version in the VCF BOM. ESX component data reaches the vSphere Client image library through the online depot or an offline depot fed by the VCF Download Tool; SDDC Manager 9.0 acts as the UMDS download source. VCF 5.2.x used the same create-in-vSphere-then-import model from SDDC Manager (Lifecycle Management > Image Management) and still supported baseline clusters; see Task 5. Sources: VCF 9.0.1 docs pp.224, 2829-2830, 1451-1452; VCF 5.2 docs pp.447-448.

Objectives

  • Design and build a vSphere Lifecycle Manager image (ESX 9.0.x base image, vendor add-on, firmware and drivers add-on, components) in the vSphere Client and import it into the VCF Instance from VCF Operations
  • Execute a staged, rolling host remediation workflow that respects DRS constraints and business availability windows
  • Detect and remediate image drift (non-compliant hosts) with the vLCM compliance check and remediation
  • Compare the VCF 9.0 image lifecycle (vSphere Client + VCF Operations Fleet Management) with the VCF 5.2.x SDDC Manager image lifecycle
  • Articulate the architectural implications of lifecycle management as a fleet-level platform service in a multi-domain VCF stack

Prerequisites

VCF 9.0 management and workload domains fully deployed and healthy. One workload domain cluster (4-6 ESX hosts running the ESX 9.0.0.0 GA build) accessible via VCF Operations, SDDC Manager and vCenter. Hosts are compliant with the image applied at deployment. Online depot connectivity or an offline depot populated with the VCF Download Tool so ESX 9.0.x component data is available in the vSphere Client image library. ESX hosts on supported boot devices (32 GB system disk minimum).

Prior labs: holodeck-02

Required skills:

  • vCenter and SDDC Manager UI navigation
  • ESXi host lifecycle operations (power, remediation, snapshot)
  • Understanding of vSphere DRS, vMotion, and resource reservations
  • Basic API navigation (vLCM Cluster Image Spec)
  • Knowledge of vendor-specific VIBs (NIC drivers, HBA firmware, GPU firmware)

Lab Environment

VCF 9.0 management domain (SDDC Manager, vCenter, VCF Operations with fleet management) + 1 workload domain with 4-6 ESX hosts (ESX 9.0.0.0) in a single cluster. All hosts are identical hardware (same NICs, HBAs, firmware revisions). Workload cluster includes: DRS enabled and fully automated (evacuates guest workloads by vMotion), HA enabled (admission control configured), vSAN disabled (to simplify drift scenarios). At least 3-4 test VMs per host to simulate production workload distribution.

Mermaid source
graph TB
  VC["vCenter 10.0.0.6<br/>vLCM image library"] --> EsxiCluster["Workload Cluster<br/>4-6 ESX 9.0.0.0 hosts"]
  Depot["Online / offline depot<br/>(VCF Download Tool)"] --> VC
  VC -->|export / Import from a vCenter| Ops["VCF Operations<br/>Fleet Management > Lifecycle"]
  Ops -->|Image Management| SDDC["SDDC Manager 10.0.0.4"]
  Ops -->|Configure Update for ESX| EsxiCluster
  VC --> VM1["VM workload-01"]
  VC --> VM2["VM workload-02"]
  VC --> VM3["VM workload-03"]
  EsxiCluster --> Host1["ESX-1"]
  EsxiCluster --> Host2["ESX-2"]
  EsxiCluster --> Host3["ESX-3"]
  EsxiCluster --> Host4["ESX-4"]

Credentials

SystemUsernamePassword
VCF OperationsadminFrom VCF deployment
SDDC Manageradministrator@vsphere.localFrom VCF deployment
vCenter Serveradministrator@vsphere.localSame as SDDC Manager
ESXi HostsrootSet during host commissioning

Tasks

Task 1 Assess baseline cluster image compliance and gather upgrade requirements

Manageability

In production, you never design an upgrade in isolation. You must inventory the current state, validate hardware heterogeneity, identify vendor-specific addons, and establish a compatibility matrix. This mirrors a VCDX design consultation where you'd spend 30-40% of your time on discovery before recommending an upgrade path.

Step 1

Log in to VCF Operations (https://<vcf_operations_fqdn>). Go to Fleet Management > Lifecycle, expand VCF Instances and browse to your workload domain. Click Updates and note the current ESX version and any available ESX update for the domain.

Domain Updates view showing the current ESX version (9.0.0.0 GA build) and the available ESX 9.0.x update
Step 2

Open the image the cluster runs today: in the vSphere Client select the workload cluster > Updates > Hosts > Image. Record the base image (ESX 9.0.0.0), vendor add-on, firmware and drivers add-on (requires the vendor hardware support manager) and additional components. Also open VCF Operations > Fleet Management > Lifecycle > VCF Instances > <instance> > Image Management to see which imported images the instance already holds.

Image composition recorded (ESX 9.0.0.0 base image + vendor add-on + components, with versions for later comparison); list of imported images in Image Management
Screenshot this page — it is your baseline. In production you would attach it to the change record.
Step 3

For each ESXi host in the cluster, verify hardware uniformity. In vCenter, go to Hosts and Clusters > [Cluster] > Hosts. For each host, click its name and navigate to Hardware. Verify CPU model, RAM quantity, NIC types (name), and HBA types match across all hosts.

All hosts show identical hardware specifications (e.g., 2x Intel Xeon, 384 GB RAM, 2x 10 Gbe NICs, 1x FC HBA)
Step 4

SSH to one ESX host and gather the component inventory. Run: esxcli software vib list | head -40 and esxcli software component list. Note the NIC and HBA driver versions and the vendor add-on components.

List of components including NIC drivers (e.g., bnx2x, i40en), HBA drivers (e.g., lpfc) and the vendor add-on
Step 5

Identify the target. In the vSphere Client image library (cluster > Updates > Hosts > Image > Edit) list the ESX 9.0.x base image versions available from the depot; if the target 9.0.x version is missing, synchronise ESX component data (online depot) or download it to the offline depot with the VCF Download Tool (VCF 9.0.1 docs p.2830). Compare target to current state: list 3 changes (e.g., ESX 9.0.0.0 → 9.0.x build, NIC driver version bump, BIOS firmware update in the vendor add-on).

Documented list of 3+ changes between the current and target image (example: ESX 9.0.0.0 build 24755229 → 9.0.x, bnx2x driver 8.45.13 → 8.46.21, BIOS 12.5 → 12.7)
Step 6

Document cluster upgrade constraints. In vCenter, check: (a) DRS enabled and fully automated? (b) HA enabled + admission control policy (Cluster Resource Percentage, Slot Policy (Powered-on VMs), Dedicated Failover Hosts, or Disabled) and host failures to tolerate? (c) Any powered-on VMs with memory reservations > 10 GB? (d) vMotion network latency (ping between hosts < 5 ms expected).

Constraint document listing DRS automation level, HA admission control policy and failures to tolerate, VM reservation totals, and network readiness

Validation Gate

Check: Document captured: (1) baseline image composition and the images imported in VCF Operations, (2) hardware inventory for all hosts, (3) 3+ specific changes in the target image, (4) cluster constraints (DRS automation level, HA admission control policy, VM reservations, network RTT)

Expected: Pre-upgrade assessment complete. You have a clear change control document ready for a production change advisory board.

Common Errors

Image Management in VCF Operations shows no imported image, or the vSphere Client image library has no ESX 9.0.x base image
Cause: ESX component data has not been synchronised from the online depot, or the offline depot has not been populated with the VCF Download Tool
Fix: If connected to the online depot, wait for the depot sync or trigger it from the Lifecycle view. For an offline depot, run the VCF Download Tool UMDS commands on the depot host (umds install, umds run -S --patch-store, --add-entitlement-token, --enable-host) and then synchronise ESX component data for the VCF Instance (VCF 9.0.1 docs pp.2830-2831). Then refresh the vSphere Client image library.
📋 KB: VCF 9.0.1 docs, Managing vSphere Lifecycle Manager Images for VMware Cloud Foundation (pp.2829-2831)
Hardware heterogeneity detected: two hosts have different NIC models (bnx2x vs ixgbe)
Cause: Hosts were commissioned from different hardware generations — common in phased expansions
Fix: Use a composite vLCM image: keep one default image and add an alternative image for the outlier hardware class (same base image and solutions, different vendor or firmware/driver add-on) with a rule that selects those hosts by vendor, family or model. Import the composite image into VCF Operations (VCF 9.0.1 docs p.2829).
📋 KB: VCF 9.0.1 docs p.2829 (composite images)
HA admission control is disabled; cluster has memory reservations totaling 500 GB on 4 hosts (512 GB per host)
Cause: Over-provisioning + no HA safety margin
Fix: Before upgrade, migrate 2-3 VMs to reduce per-host reservation. Then enable HA admission control (Cluster Resource Percentage, host failures to tolerate = 1). Remediation still proceeds one host at a time; the point is that the remaining hosts must hold the evacuated VMs.

Task 2 Build the target image in the vSphere Client and import it into the VCF Instance

Manageability

In VCF 9.0 the image is authored where vLCM lives — the vSphere Client — and then imported into the VCF Instance from VCF Operations so that domain upgrades, new workload domains and new clusters can use it. The image is a dependency graph: base image, vendor add-on, firmware and drivers add-on (through the vendor's hardware support manager) and components. Panelists will ask: 'What happens if the vendor add-on for this hardware generation is not available for the target ESX build?'

Step 1

In the vSphere Client select the workload cluster > Updates > Hosts > Image and click Edit (or set up the image on a staging cluster if you do not want to touch the production cluster's desired state yet). Under ESX Version select the target ESX 9.0.x base image from the depot. Use the ESX version named in the VCF 9.0.x BOM (VCF 9.0.1 docs p.2830: 'Use the ESX version specified in the VMware Cloud Foundation BOM').

Image editor shows the target ESX 9.0.x base image
Step 2

Select the Vendor Addon for your server vendor and model generation. Compare the add-on's driver versions with your baseline inventory from Task 1 (for example bnx2x 8.45.13 → 8.46.21). For a major version bump, read the vendor release notes and check the Broadcom Compatibility Guide (https://compatibilityguide.broadcom.com/). Document any breaking changes.

Vendor add-on selected; driver deltas documented
In production you would validate the new image on 2-3 hosts in an isolated cluster first. For this lab, you trust the image, but the concept is critical.
Step 3

If a hardware support manager (Dell OMIVV, HPE iLO Amplifier/OneView, Lenovo XClarity Integrator) is registered with vCenter, select the Firmware and Drivers Addon it exposes so BIOS/BMC/NIC firmware is part of the image (VCF 9.0.1 docs p.1811). Without an HSM, firmware stays out of the image and is handled out-of-band.

Firmware and drivers add-on selected, or the firmware gap documented
Step 4

Under Components, add any additional components your design needs (for example an NVIDIA vGPU host driver or a vendor management agent). Check the composition and click Validate; vLCM resolves component dependencies and flags conflicts. Save the image. Do not remediate yet.

Image validated and saved; Image pane shows the new desired image and marks the hosts non-compliant until remediation
Step 5

Import the image into the VCF Instance: in VCF Operations go to Fleet Management > Lifecycle, click VCF Instances and your instance name, click Image Management > Import Image and choose Import from a vCenter (select the vCenter and the image). The alternative is Import from a file (JSON, ZIP and optional ISO exported from the vSphere Client).

Image Management lists the imported image for the instance
VCF 9.0.1 docs pp.2829-2830. The imported image is reusable for upgrading domains, creating a workload domain and adding clusters.
Step 6

Stage the image to the hosts ahead of the maintenance window: in the vSphere Client, cluster > Updates > Hosts > Image, click Stage All. vLCM downloads the image payload to all hosts in parallel without maintenance mode; hosts stay online and VMs keep running. Monitor the Stage tasks in Recent Tasks.

Stage tasks complete for every host; the Image pane shows the staged status

Validation Gate

Check: The target image is saved on the cluster in the vSphere Client (ESX 9.0.x base image + vendor add-on + components), imported into the VCF Instance (Image Management) and staged to all hosts.

Expected: Target image built, imported and staged. Cluster is ready for remediation.

Common Errors

The vendor add-on or a driver version you need does not appear in the image editor
Cause: ESX component data for that add-on has not reached the vSphere Client image library: the online depot has not synchronised, or the offline depot does not contain it
Fix: Synchronise ESX component data for the VCF Instance, or download it to the offline depot with the VCF Download Tool (VCF 9.0.1 docs pp.2830-2831), then refresh the image editor. If the vendor has not released the add-on for the target ESX build, keep the previous add-on only if the vendor supports it on that build, or wait.
Image validation fails with a component dependency or conflict error
Cause: Two components depend on incompatible versions, or a component is not supported on the selected base image
Fix: Remove the conflicting component or pick the version the base image supports; re-validate. Record the decision in the change record.
Import from a vCenter does not list your image
Cause: The image was edited but not saved on the cluster, or VCF Operations is pointed at the wrong vCenter
Fix: Save the image on the cluster in the vSphere Client, then retry Import Image and select the correct vCenter in the drop-down.

Task 3 Execute a rolling host remediation with DRS-aware sequencing

Availability

This is where theory meets practice. You orchestrate the remediation of 4-6 hosts one at a time: DRS evacuates each host by vMotion as it enters maintenance mode, vLCM applies the image and the host reboots (or Quick Boots). In VCDX you will be asked: 'If you have a 6-host cluster with 10 VMs per host and HA admission control reserving one host, how long does the upgrade take and what blocks it?'

Step 1

In vCenter, navigate to your workload cluster. Right-click the cluster and select 'Expand' or click each host individually. For each host, note the current VM count. Create a remediation order starting with the host that has the fewest powered-on VMs (or manually vMotioned VMs to other hosts first). Target order: Host 1 (least VMs) -> Host 2 -> Host 3 -> Host 4.

Remediation sequencing plan documented (e.g., 'Host-1: 2 VMs, Host-2: 3 VMs, Host-3: 4 VMs, Host-4: 3 VMs')
Step 2

Start the update from VCF Operations: Fleet Management > Lifecycle > VCF Instances > your workload domain > Updates. Click Precheck and resolve findings. In Available Updates click Configure Update for ESX, select the cluster (Custom selection), click Assign Image and pick the imported image.

ESX update wizard shows the selected cluster and the assigned image
Step 3

Select the upgrade options: Enable Quick Boot (skips the physical reboot where supported), Migrate Powered Off and Suspended VMs, and leave Enforce Live Patch unselected. Review and click Run Precheck (VCF runs a cluster image hardware compatibility and compliance precheck). When it passes, click Schedule Update and run now. vLCM then takes the hosts one at a time: maintenance mode (DRS evacuates VMs), apply image, reboot, exit maintenance mode.

Update in progress in the Updates view and Tasks; Host 1 transitions to maintenance mode
VCF 9.0.1 docs pp.1452-1453. Remediation that needs maintenance mode fails if DRS is not fully automated (p.7068).
Step 4

While Host 1 is remediating, monitor in parallel: (a) In vCenter, watch vMotion activity as DRS evacuates the host. (b) Check the cluster's DRS recommendations. (c) In VCF Operations Tasks (or SDDC Manager Tasks) follow the per-step progress of the ESX update.

vMotion activity observed (VMs migrating off Host 1), DRS recommendations visible, update task showing per-step progress
Step 5

Host 1 remediation should complete in 15-25 minutes (image apply + reboot or Quick Boot). vLCM moves to the next host automatically once Host 1 is compliant and out of maintenance mode. Confirm in the vSphere Client (cluster > Updates > Hosts > Image) that Host 1 shows compliant and that the update continues to Host 2.

Host 1 compliant and back in the cluster; Host 2 entering maintenance mode
Hosts within a cluster are remediated sequentially; the parallel option in the wizard is at cluster level (several clusters at once, 'Enable sequential cluster upgrade' to serialise them).
Step 6

After all hosts are remediated, verify cluster-wide compliance: vSphere Client > cluster > Updates > Hosts > Image shows every host compliant with the image; VCF Operations > Lifecycle > domain > Updates shows the ESX update complete. In vCenter, verify all hosts are connected, all VMs are running, and DRS has rebalanced VM distribution.

All hosts compliant, all VMs running, update task complete

Validation Gate

Check: (1) vSphere Client Image pane shows all hosts compliant with the ESX 9.0.x image, (2) VCF Operations Updates view shows the ESX update complete, (3) vCenter cluster shows all hosts connected with no alarms, (4) all powered-on VMs are running, (5) documented remediation duration per host (target: 15-25 min/host)

Expected: Cluster upgraded from ESX 9.0.0.0 to ESX 9.0.x with zero guest downtime (DRS evacuation by vMotion during maintenance mode).

Common Errors

Host enters maintenance mode but vMotion stalls (VMs do not migrate off). Remediation times out after 30 minutes.
Cause: VM has a memory reservation >= available capacity on remaining hosts, or vMotion network is congested. DRS cannot successfully evacuate the host.
Fix: In vCenter, reduce the VM memory reservation or migrate the blocking VM manually (cold migrate if needed), then retry the update task from VCF Operations. If DRS is not fully automated, set it to fully automated first (p.7068).
📋 KB: VCF 9.0.1 docs p.7068 (DRS automation level and remediation)
After remediation the host shows compliant in the vSphere Client but 'vmware -vl' still prints the old build
Cause: The host booted the previous boot-bank (for example a failed or skipped reboot)
Fix: Check the host's boot sequence and /var/log/lifecycle.log; re-run the update for that cluster from VCF Operations so vLCM remediates the host again.
Second or third host remediation fails with a depot or payload download error
Cause: The host could not reach the depot for the image payload (staging was skipped or the depot became unreachable)
Fix: Verify depot connectivity from vCenter and the host, run Stage All again from the vSphere Client Image pane, and retry the update.

Task 4 Detect and remediate infrastructure drift with vLCM compliance baselines

Security

Post-upgrade your cluster is compliant. In production, hosts drift: an administrator installs a driver out-of-band, firmware is patched outside the image, or a tool drops a component. The vLCM compliance check against the cluster image is your enforcement point: it reports every host whose software state differs from the desired image, and remediation brings the host back.

Step 1

Simulate drift on Host 2: SSH to that host. Check the current NIC driver component: esxcli software component list | grep -i <driver> (or esxcli software vib list | grep bnx2x for your NIC). Note the version.

Current NIC driver version shown (baseline after remediation)
Step 2

Introduce drift out-of-band. Put Host 2 in maintenance mode from vCenter (DRS evacuates it), then on the host run esxcli software vib remove -n <component-vib-name> for an optional component that the image includes (for example a vendor management agent), or install an older version of a driver from a pre-staged offline bundle: esxcli software component apply -d /vmfs/volumes/<datastore>/<bundle>.zip. Reboot if the component requires it. Do not remove the NIC driver carrying the management network.

Host 2 software state now differs from the cluster image (component missing or older)
Lab only: this is exactly the behaviour vLCM exists to prevent. In production, esxcli software changes on image-managed hosts should be blocked by process.
Step 3

In the vSphere Client select the cluster > Updates > Hosts > Image and, in the Image Compliance card, click Check Compliance (vSphere 9.0 docs: Check the Compliance of a Cluster or Standalone Host Against a vSphere Lifecycle Manager Image). The compliance check compares every host against the desired image.

Compliance check runs. Host 2 shows non-compliant with the drifted component listed (expected version versus installed/missing)
Step 4

Review the compliance details for Host 2: the desired image specification, the host's actual state, and the component differences. Document the mismatches.

Compliance report showing the component delta on Host 2
Step 5

Remediate the drift: in the vSphere Client Image pane, select Host 2 in the Image Compliance card and click Remediate (or Remediate All to let vLCM handle every non-compliant host; vSphere 9.0 docs: Remediate a Cluster or Selected Hosts in a Cluster Against an Image). vLCM re-applies the image to the host: maintenance mode (DRS evacuates VMs), apply image, reboot if required, exit maintenance mode.

Host 2 enters maintenance mode, remediates (5-15 minutes), exits maintenance mode and shows compliant
Step 6

After remediation, verify Host 2 is back to the desired state. SSH to the host and re-run esxcli software component list (or esxcli software vib list | grep <driver>). Confirm the component version matches the image.

Component version is back to the image baseline

Validation Gate

Check: (1) Drift detected on Host 2 by the vLCM compliance check, (2) compliance details identified the component delta, (3) Host 2 remediated to compliant, (4) SSH verification confirms the correct version

Expected: Drift detection and remediation workflow demonstrated against the cluster image.

Common Errors

Check Compliance reports all hosts compliant although you changed Host 2
Cause: The component you changed is not part of the desired image (vLCM only tracks image components), or the change was reverted by a reboot to the other boot-bank
Fix: Confirm the component is in the image (cluster > Updates > Hosts > Image). Change a component that is in the image and re-run Check Compliance.
Remediation of Host 2 fails because the host cannot enter maintenance mode
Cause: DRS cannot evacuate a VM (reservation, affinity rule) or DRS is not fully automated
Fix: Resolve the blocking VM in vCenter, confirm DRS is fully automated, then retry Remediate.

Task 5 Compare VCF 9.0 vLCM workflow with VCF 5.2.x and document operational complexity reduction

Manageability

This is the VCDX perspective: you are not just executing steps, you are comparing two design paradigms. VCF 5.2.x managed lifecycle from SDDC Manager with vLCM images created in vSphere and imported (and still supported baseline-managed clusters). VCF 9.0 moves fleet lifecycle into VCF Operations (Fleet Management > Lifecycle) with images only, imported per VCF Instance, and NSX VIBs delivered as part of the ESX image. Document the operational differences and what they mean for governance at scale.

Step 1

Document the VCF 9.0 workflow you just executed: (1) Task 1: cluster assessment (manual, ~30 min), (2) Task 2: image build in the vSphere Client + import in VCF Operations + Stage All (~20 min), (3) Task 3: rolling remediation (VCF Operations update task, vLCM orchestrated, ~90 min for 4 hosts), (4) Task 4: compliance check and remediation (~15 min). Total operator time: ~2 hours. Total wall-clock time: ~2.5 hours.

Timeline document for VCF 9.0 workflow
Step 2

Now review the VCF 5.2.x approach (VCF 5.2 docs pp.447-448, 'Creating a vSphere Lifecycle Manager Image in VMware Cloud Foundation'). In 5.2: (a) images were also created in vSphere and then extracted from a cluster or imported in SDDC Manager > Lifecycle Management > Image Management (JSON, ZIP, optional ISO); (b) baseline-managed and image-managed clusters could coexist in a workload domain and had to be transitioned before 9.0; (c) NSX kernel modules were installed by NSX, not as part of the ESX image; (d) lifecycle was per SDDC Manager instance, not fleet-wide. Document 3 differences that 9.0 changes.

Comparison document: 3-5 VCF 5.2 vs 9.0 operational differences (images only, fleet-level Lifecycle in VCF Operations, NSX VIBs in the ESX image, Import from a vCenter, patch planner)
Step 3

Quantify operator effort: VCF 9.0 required ~2 hours of hands-on time (largely monitoring). Estimate the 5.2 effort for the same cluster including the baseline-to-image transition where needed and the separate NSX host upgrade step. Document the hours saved per cluster upgrade and extrapolate to a multi-cluster environment (e.g., 10 workload clusters).

Operator time savings calculation
Step 4

Identify where complexity shifted. In 9.0, complexity shifted LEFT (earlier): image design and composition (vendor add-on, firmware through the HSM, component validation) happens up-front in the vSphere Client and the precheck covers hardware compatibility and compliance. In 5.2 more issues surfaced during remediation. Document this as a design statement with your own measured data rather than a generic percentage.

Complexity shift analysis backed by the timings you measured
Step 5

For a VCDX panelist, articulate: 'What are the architectural trade-offs?' Prompt: If you had 50 workload domains with mixed hardware, would you prefer per-domain images (flexibility) or a small set of fleet-level images with composite alternatives (consistency)? Document your answer and reasoning (2-3 paragraphs).

Architectural trade-off analysis (consistency vs flexibility in multi-domain lifecycle management)
Step 6

Finally, measure upgrade time empirically. Document: (1) Time from 'Start Remediation Host 1' to 'Host 1 Back Online', (2) Average time per host, (3) Total cluster remediation time. In a real production run, this data informs your change advisory board's risk assessment ('If we have a 4-hour maintenance window and 6 hosts, can we complete the upgrade? What's our rollback decision point?').

Measured upgrade timeline: Host remediation duration, total cluster time, rollback threshold (e.g., 'If not 3 of 4 hosts done by 2-hour mark, we abort and revert').

Validation Gate

Check: (1) VCF 9.0 workflow timeline documented (all 4 tasks + wall-clock time), (2) VCF 5.2 workflow comparison with 3+ specific differences, (3) Operator time delta calculated for 1 cluster + extrapolated to 10 clusters, (4) Complexity shift analysis (left-shifting image validation), (5) VCDX-level trade-off analysis (consistency vs flexibility), (6) Empirically measured host remediation times

Expected: You can now articulate to a VCDX panelist: 'VCF 9.0's fleet-level lifecycle in VCF Operations, with images authored in vSphere and NSX VIBs delivered in the ESX image, reduced lifecycle operator overhead versus 5.2 by the amount I measured in this lab, but it requires stronger upfront image design governance. Here are the measured timelines and the architectural trade-offs.'

Common Errors

VCF 5.2 documentation is hard to find
Cause: Broadcom techdocs lists the 5.2 documentation under the earlier VCF versions; search results favour 9.x
Fix: Use the local VCF 5.2 documentation set (Image Management pp.447-448) and the Broadcom techdocs 5.2 landing page; the Academy's VCF Evolution section summarises the 5.2 → 9.0 lifecycle changes.

Final Validation

You have designed and built a vSphere Lifecycle Manager image in the vSphere Client, imported it into the VCF Instance from VCF Operations, staged it, and executed a rolling ESX cluster upgrade from the VCF Operations Lifecycle view. You detected and remediated image drift with the vLCM compliance check, and you articulated the architectural implications versus VCF 5.2. All hosts are compliant, the cluster is healthy, and you have measured data to inform production change control.

✓ Baseline cluster image assessment: documented current state, target state, and change list → 3+ specific VIB/firmware changes identified

✓ Target image built in the vSphere Client (ESX 9.0.x + vendor add-on + components), imported into the VCF Instance and staged to all hosts → Image Management lists the image; vSphere Client Image pane shows staged status on every host

✓ All 4-6 hosts remediated successfully: zero guest downtime (vMotion during maintenance mode) → All hosts online, compliant, running their original VM workload (or redistributed by DRS)

✓ Drift detection: drift introduced on 1 host, detected by the vLCM compliance check, remediated back to compliant → Host showed non-compliant, then compliant post-remediation

✓ Workflow comparison: VCF 9.0 vs 5.2 analyzed; operator time savings quantified; architectural trade-offs articulated → Documented timeline comparison + 2-3 paragraph VCDX-level analysis

Cleanup / Restore

Snapshot: workload-domain-post-vlcm-upgrade

• Take a snapshot of the workload domain cluster post-upgrade (all hosts compliant, all VMs running): snapshot all cluster VMs as 'workload-domain-post-vlcm-upgrade'

• Document the final image (name, ESX build, add-ons, components) as shown in VCF Operations Image Management for reference in subsequent labs

• If you need to revert for future labs, restore to the 'workload-domain-baseline' snapshot (pre-upgrade state)

Design Reflection (VCDX)

A VCDX panelist examining a large-scale VCF lifecycle management design would probe: How do you handle image composition governance across 50+ workload domains with mixed hardware? What's your rollback strategy if drift remediation fails on 20% of hosts? How does vLCM integrate into your change management process, and who approves image changes? How would you design automated compliance checks that alert on drift within 24 hours? The vLCM cluster image is not just a technical feature—it's a policy enforcement layer. Be ready to argue whether centralized (one image per VCF instance) or distributed (one image per workload domain) image governance scales better, and defend the trade-offs with operational data from this lab.

Requirements

  • Cluster image must incorporate ESXi base OS, vendor-specific drivers (NIC, HBA), firmware (BIOS, BMC), and optional VIBs (GPU, management agents)
  • Remediation must be zero-downtime (vMotion-based) and respect DRS constraints; no guest VM downtime allowed
  • Drift detection must identify non-compliant hosts within 1 hour of deviation and alert operators
  • Lifecycle operations must be auditable: log every image change, remediation, and drift detection event to a centralized audit trail

Constraints

  • vLCM images are cluster-scoped but reusable: VCF Operations imports an image once per VCF Instance and reuses it for upgrades, new domains and new clusters. Mixed hardware in one cluster is handled with a composite image (one default image plus alternative images with host-selection rules), not by host retirement (VCF 9.0.1 docs p.2829).
  • Remediation bandwidth is limited by vMotion network capacity and the per-host apply-and-reboot time; hosts in a cluster remediate sequentially, typically 1 host per 15-30 minutes. Quick Boot shortens the reboot where the hardware supports it.
  • Vendor add-on and component availability depends on ESX component data reaching the vSphere Client image library through the online depot or an offline depot populated with the VCF Download Tool; a stale depot delays the image build.
  • DRS constraints (VM memory reservations, resource affinity rules) can block vMotion, causing remediation stalls. Operator must pre-validate cluster has enough free capacity.

Assumptions

  • Hosts in the cluster are identical hardware, or a composite image with alternative images and host rules covers the hardware classes present
  • vMotion network is available and has > 1 Gbps capacity for concurrent VM migrations
  • Each host has a supported boot device (ESX 9.0: 32 GB system disk minimum, 1 GB or 4 GB boot-banks); staging does not consume datastore space
  • No VMs have 'must stay on this host' affinity rules that would block remediation
  • The online depot (or the offline depot via the VCF Download Tool) is synchronised so the target ESX build and vendor add-ons are available before the maintenance window

Risks

  • Image composition error (conflicting VIB versions): if two VIBs depend on incompatible kernel ABIs, remediation fails mid-flight and host rolls back, causing remediation stall. MITIGATION: test image on 1-2 hosts before full-scale rollout; maintain rollback image.
  • vMotion stall during maintenance mode: if remaining cluster capacity cannot absorb all VMs from remediating host, vMotion never completes and remediation times out. IMPACT: host stuck in maintenance mode, unavailable to guest workload. MITIGATION: pre-validate DRS has enough free capacity; migrate non-critical VMs off before starting.
  • Depot unavailable during the maintenance window: hosts that were not staged cannot download the payload and the update task fails. IMPACT: window lost. MITIGATION: Stage All before the window; for air-gapped sites keep the offline depot current with the VCF Download Tool.
  • Drift remediation fails on a host (component install error) and the host stays non-compliant. IMPACT: compliance gap on that host. MITIGATION: the compliance check reports per host and a failed remediation on one host does not change the image on others; investigate lifecycle.log and re-run Remediate.
  • Operator error: manually removing a critical VIB (as in Task 4 drift simulation) is easy and undetectable until drift scan runs hours later. In production, this could cause silent compliance gaps on 10% of hosts. MITIGATION: enforce vib removal/install via vLCM only, disable direct esxcli vib commands on production hosts.

Self-Assessment Discussion Prompts

  1. If you have 50 workload domains, each with slightly different hardware (vendor A's NICs in 20 domains, vendor B's in 30), how would you design your vLCM image strategy? One image per vendor? One per domain? What are the governance implications?
  2. Your change control process requires 2-week approval before any ESXi upgrade. But a critical NIC driver CVE is patched, and you need to update the cluster image within 48 hours. How do you fast-track the image change while maintaining governance?
  3. A host fails remediation mid-upgrade (VIB installation error, host panics during reboot). What's your rollback decision logic? Do you revert the host from snapshot, or do you re-remediate with a different image? What data do you need to decide?
  4. The depot is your source of ESX component data. If the online depot (or your offline depot) becomes unavailable, can you still remediate hosts? What does Stage All buy you, and what is your air-gapped contingency with the VCF Download Tool?
  5. After upgrading 4 hosts successfully, you discover a driver bug in the new vendor add-on (bnx2x 8.46.21) under load. What is your rollback strategy? Can you roll back 4 hosts to the previous image while the 2 remaining hosts are still on the old build, and what does the cluster image model force you to do?
  6. You have 1000 hosts across 20 clusters and want to move to the new ESX 9.0.x build. Each cluster takes 8 hours to remediate. You have a 4-day window. How many clusters can you upgrade in parallel from VCF Operations (clusters run in parallel by default, up to ten at a time), and what are the bottlenecks (vMotion bandwidth, depot throughput, change-control approvals)?

Extensions

Build and test a heterogeneous cluster image strategy for mixed hardware

Repeat Tasks 2-3 with a composite image: one default image for hosts with Intel NICs (i40en vendor add-on) and an alternative image for Broadcom (bnx2x) hosts, with host-selection rules by vendor/model. Deploy the lab cluster with mixed hardware. Demonstrate that vLCM applies the right alternative image to each host and that VCF Operations imports the composite image once (VCF 9.0.1 docs p.2829). Document the operational complexity of managing composite images in production (versioning, testing, approval workflows).

harder

Simulate a vLCM remediation failure and execute a rollback recovery

Deliberately break the depot path for one host (for example block the host's access to the depot after staging only three hosts) to cause an update failure on Host 3. Document the failure mode, the error in lifecycle.log and the update task, and the recovery procedure (re-stage, retry). How long does recovery take? What does this teach you about staging before the window and about rollback decision thresholds?

harder

Design a multi-cluster upgrade orchestration workflow

You have 5 workload clusters (each with 6 hosts). Design a vLCM upgrade strategy that maximizes parallelization while respecting business constraints: (a) max 1 cluster can be in remediation at a time (to avoid vMotion contention), (b) each cluster must complete within 12 hours, (c) you want minimum guest downtime. Build a timeline and sequencing plan. Validate it against vMotion bandwidth, datastore I/O, and SDDC Manager API concurrency limits. This is a production-scale orchestration problem.

harder

Compare online-depot image builds with an offline depot (VCF Download Tool)

In Task 2 you used the online depot. Simulate an air-gapped environment: set up an offline depot web server, populate it with the VCF Download Tool UMDS commands (VCF 9.0.1 docs pp.2830-2831) and connect the VCF Instance to it. Build the image from the offline component data. Compare: (a) time to build the image (online vs offline), (b) how you learn that the offline depot is stale, (c) the operational process to refresh it. This is relevant for financial/government VCF deployments where Broadcom connectivity is restricted.

same

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.