Academy/VCF 9.0 Support (2V0-15.25)/Upgrade Failure & Remediation
This lab targets VCF 9.0

Upgrade Failure & Remediation

VCF 9.0Advancedvcp-foundationvcp-support⏱ 150 min

VCF lifecycle upgrade failure scenarios with rollback, log analysis, and SDDC Manager workflow recovery.

Objectives

  • Understand the VCF 9.0 update sequence and component dependencies (KB 390634: management planes first, then SDDC Manager → NSX → vCenter → ESX)
  • Diagnose upgrade failures using SDDC Manager logs, vCenter upgrade logs, and NSX upgrade status
  • Execute vCenter upgrade recovery for a reduced-downtime upgrade (retry, Cancel, automatic rollback, file-based restore)
  • Recover failed SDDC Manager upgrade workflows using API retry and manual intervention
  • Design pre-upgrade validation checklists to prevent common upgrade failures
  • Document upgrade failure recovery using RCAR methodology

Prerequisites

VCF lab with management domain running pre-upgrade version

Prior labs: vcp-support-01

Required skills:

  • SDDC Manager UI and API navigation
  • vCenter snapshot management
  • SSH access to VCF components

Lab Environment

VCF management domain with SDDC Manager, vCenter, NSX Manager cluster, and 4 ESXi hosts

  • Hosts needed: 4
  • VMs needed: 3
  • Storage: vSAN ESA datastore

Tasks

Task 1 VCF Upgrade Architecture & Sequence Planning

Foundation for upgrade troubleshooting. Panelists probe: Why this specific order? What happens if you skip a step? Which components can be rolled back independently?

Master the 19-step VCF upgrade sequence, understand component interdependencies, and identify which steps are most likely to fail and why.

Step 1

Document the VCF 9.0 update sequence per Broadcom KB 390634 (Update Sequence for VCF 9.0 and Compatible VMware Products). Fleet management plane first: (0) VCFDT, (1) VMware Aria Suite Lifecycle 8.18 patch (patch only; no upgrade to 9.0 — the KB names Patch 2, the VCF 9.0 upgrade documentation requires 8.18 Patch 6 or later, so apply Patch 6 or later), (2) VCF Operations + VCF Operations fleet management appliance, (3) VCF Automation and VCF Operations for networks (parallel), (4) VCF Operations for logs (new installation). Then (5) VADP-based backup, (6) vSphere Replication, (7) SRM, (8) Avi. Core SDDC: (9) SDDC Manager, (10) HCX, (11) NSX, (12) vCenter, (13) VCF Identity Broker (new installation) and Supervisor, (14) vSAN witness host, (15) ESX, then NSX finalize, (16) VMware Tools, (17) virtual hardware, (18) vSAN on-disk format / VMFS, (19) vSAN File Service.

Sequence documented from KB 390634. Key dependencies: the management planes (VCF Operations and fleet management) go first; the core order is SDDC Manager → NSX → vCenter → ESX (also in the VCF 9.0 docs, 'Upgrade the Management Domain Core Components'); VCF Identity Broker can only be installed after VCF Operations, SDDC Manager and the management vCenter are on 9.0; the vSAN on-disk format comes after ESX because the new format needs the new ESX version. KB note: patches can be applied in any order; the order matters for major and minor releases.
The upgrade sequence is the most frequently tested topic in VCF support exams. Memorize the order and understand WHY each step must precede the next.
Step 2

Identify the most failure-prone upgrade steps and common causes. Document: (a) SDDC Manager upgrade — database migration issues, insufficient disk space; (b) vCenter upgrade — VCSA migration assistant failures, customization conflicts; (c) NSX Manager cluster upgrade — cluster stability during rolling upgrade, backup controller issues; (d) ESXi host upgrade — vSAN maintenance mode failures, incompatible hardware/drivers.

Failure matrix: Step | Common Failure | Root Cause | Prevention. SDDC Manager: DB migration = clean up stale data pre-upgrade. vCenter: migration assistant = verify DNS and NTP pre-upgrade. NSX: cluster instability = ensure 3-node cluster healthy pre-upgrade. ESXi: maintenance mode = verify vSAN capacity allows data evacuation.
Many upgrade failures trace back to conditions the precheck would have caught; the rest need a Broadcom support case. No public statistic splits the two, so do not quote percentages to a panel.
Step 3

Map the recovery options for each component. vCenter: VCF 9.0 upgrades vCenter with a reduced-downtime upgrade (RDU, a new appliance on a temporary IP then switchover) — retry the failed task, Cancel to revert to the source vCenter while the RDU runs, automatic rollback on failure, or restore from the file-based backup (VCF 9.0.1 docs p.1417). SDDC Manager: restore from backup or revert the cold snapshot taken before the update (p.1366). NSX Manager: Management Plane rollback from the local /config_bak backup on all nodes (p.1401), or restore from backup. ESX: revert to the alternate boot bank (Shift+R at boot, KB 316592) or rebuild from the cluster image. Document the recovery time estimate for each.

Recovery matrix: vCenter RDU retry/Cancel/auto-rollback (minutes) or file-based restore (30-60 min), SDDC Manager cold-snapshot revert or backup restore (30-60 min), NSX Manager Management Plane rollback from local backup (60-120 min, before any edge/host upgrade) or backup restore, ESX revert via Shift+R to the alternate boot bank (KB 316592) or rebuild. Key insight: the documented prerequisites are file-based (or image-based) backups of vCenter and NSX plus a cold snapshot of SDDC Manager — not snapshots of every appliance.
VCDX defense: the rollback strategy must be planned BEFORE the upgrade starts. Panelists ask: What is your rollback plan if step 11 (NSX) fails? Answer with specific RTO and procedure.
Step 4

Review the pre-upgrade validation checklist. Items: (a) verify bundle compatibility matrix in SDDC Manager, (b) check all VCF component health (SDDC Manager dashboard), (c) verify vSAN health and capacity (must be below 80% for host upgrades), (d) confirm DNS forward/reverse resolution for all components, (e) verify NTP synchronization across all components (<5 second drift), (f) take VM snapshots of vCenter, SDDC Manager, NSX Managers, (g) export NSX Manager backup, (h) document current versions of all components.

Pre-upgrade checklist with 15+ items, each mapped to the upgrade failure it prevents. DNS and NTP faults are a common, preventable cause of upgrade failures. Snapshot/backup omissions make rollback impossible.
Create a pre-upgrade checklist template. Run it every time — even for minor patches. The checklist catches issues that are obvious in hindsight but easy to miss under upgrade pressure.

Validation Gate

Check: Complete VCF upgrade sequence and pre-upgrade checklist documentation

Expected: 19-step sequence documented with dependency justification. Failure matrix and rollback matrix created. Pre-upgrade checklist with 15+ items.

Common Errors

Attempting to upgrade vCenter before SDDC Manager — SDDC Manager must be upgraded first to orchestrate subsequent steps
Skipping VM snapshots before component upgrades — makes rollback impossible if upgrade fails
Not verifying vSAN capacity before host upgrades — maintenance mode fails if capacity is insufficient for data evacuation
Ignoring DNS/NTP pre-checks — a common, preventable cause of upgrade failures

Task 2 Upgrade Failure Diagnosis & Log Analysis

Core troubleshooting skill. Panelists ask: Where do you look first when an upgrade fails? How do you distinguish a transient error from a blocking issue? When do you escalate to VMware support?

Develop systematic log analysis skills for diagnosing VCF upgrade failures — from SDDC Manager workflow failures to vCenter migration issues to ESXi remediation errors.

Step 1

Map the critical log locations for VCF upgrade troubleshooting. SDDC Manager upgrade logs: /var/log/vmware/vcf/lcm/ — lcm-debug.log is the log for failed upgrades of all components; from SDDC Manager 9.0 all LCM logging goes to lcm-debug.log (VCF 9.0.1 docs p.1456). NSX Manager: /var/log/proton/nsxapi.log. vCenter: the upgrade logs under /var/log/vmware/upgrade/ on the appliance (check the file names in the vCenter upgrade documentation).

Each component has specific log files for upgrade operations. SDDC Manager lcm-debug.log shows the upgrade workflow and the step that failed; follow it into the component log.
Start with SDDC Manager lcm-debug.log for any upgrade driven from SDDC Manager — even if the failure appears to be in vCenter or NSX, SDDC Manager orchestrates the workflow and captures the triggering error.
Step 2

Analyze a simulated vCenter upgrade failure. Common scenario: VCSA upgrade fails at database migration step. Log analysis: search the vCenter upgrade logs under /var/log/vmware/upgrade/ for 'ERROR' and 'FAILED'. The error typically shows: database schema migration failure, disk space exhaustion, or service dependency conflict. Document the error pattern and correlate with pre-upgrade checklist items.

Error patterns: 'Database migration failed' = insufficient /storage/db space or corrupted VCDB entries. 'Service start failed' = dependency service (SSO/STS) not healthy. 'Migration assistant failed' = DNS resolution failure between source and target VCSA.
For vCenter upgrade failures, start in the vCenter upgrade logs under /var/log/vmware/upgrade/; when the upgrade was driven from SDDC Manager, correlate with lcm-debug.log.
Step 3

Analyze SDDC Manager workflow failure. Navigate to SDDC Manager UI > Lifecycle Management > view failed workflow. Document: workflow ID, failed step, error message, and retry eligibility. Via API: GET /v1/tasks/{taskId} to retrieve detailed error information. Check if the workflow can be retried (some failures are retryable, others require manual intervention).

SDDC Manager workflows have states: IN_PROGRESS, SUCCESSFUL, FAILED, CANCELLED. Failed workflows show the specific sub-task that failed. Retryable failures: transient network issues, temporary service unavailability. Non-retryable: version incompatibility, missing prerequisites.
SDDC Manager API is your best friend for upgrade troubleshooting. The UI shows summary information; the API provides detailed error context including stack traces.
Step 4

Practice ESXi host upgrade failure diagnosis. Common scenario: host enters maintenance mode but vSAN data evacuation stalls. Diagnose: (a) check vSAN resync status (objects still syncing from previous operation), (b) verify cluster capacity allows data migration, (c) check for VM anti-affinity rules preventing evacuation, (d) verify no VMs are pinned to the host. Document remediation for each cause.

ESXi upgrade failure during maintenance mode entry: vSAN evacuation stall is the most common cause. Remediation: wait for pending resyncs to complete, verify capacity, check DRS constraints. If a specific VM cannot migrate, check affinity rules and resource reservations.
Never force maintenance mode on a vSAN host — this bypasses data evacuation and risks data loss. Always resolve the evacuation blocker before proceeding.
Step 5

Build an upgrade failure escalation decision tree. Level 1: retry the failed workflow (if retryable). Level 2: analyze logs, fix the identified issue, retry. Level 3: rollback the failed component (snapshot/backup restore), fix root cause, re-attempt. Level 4: escalate to VMware support with: support bundle, upgrade logs, and detailed failure description.

Escalation decision tree with 4 levels, each with specific criteria for escalation to the next level. Include time-boxing: Level 1 (5 min), Level 2 (30 min), Level 3 (60 min), Level 4 (open SR). Collect support bundle at Level 2 for VMware SR preparation.
Time-boxing escalation levels prevents excessive troubleshooting during maintenance windows. If Level 2 analysis doesn't resolve in 30 minutes, proceed to rollback rather than extending the outage.

Validation Gate

Check: Complete upgrade failure diagnosis with log analysis and escalation procedures

Expected: Log locations mapped for all VCF components. Failure patterns documented. Escalation decision tree created with time-boxing.

Common Errors

Not starting with SDDC Manager lcm-debug.log — it is the documented log for failed upgrades of all components (p.1456)
Retrying failed workflows without fixing the root cause — same failure recurs
Forcing ESXi maintenance mode when vSAN evacuation stalls — risks data loss
Not collecting support bundle before attempting fixes — evidence lost for VMware SR

Task 3 Rollback Procedures & Recovery Operations

Critical operational skill. Panelists ask: What is your rollback procedure for each component? How do you ensure consistency after partial upgrade? What is the maximum rollback window?

Execute component-level recovery procedures for failed VCF upgrades — vCenter RDU retry/Cancel/auto-rollback and file-based restore, SDDC Manager cold-snapshot revert or backup restore, NSX Manager Management Plane rollback.

Step 1

Practice vCenter upgrade recovery. VCF 9.0 upgrades vCenter with a reduced-downtime upgrade: a new appliance is deployed on a temporary IP, data is migrated, then switchover. Steps: (1) If the task fails, read the error in the Updates view, resolve it and retry the failed task. (2) While the RDU is running, Cancel reverts the configuration to the source vCenter. (3) RDU upgrades roll back automatically on failure. (4) If the issue cannot be resolved, restore vCenter from its file-based backup (Restoring a vCenter Instance). (5) Only after a successful upgrade remove the old appliance; allow ~15 minutes for post-upgrade drift reconciliation before the next upgrade (VCF 9.0.1 docs p.1417).

A cancelled or auto-rolled-back RDU leaves the source vCenter running at the pre-upgrade version with no restore needed; a file-based restore takes 30-60 minutes and brings vCenter back to the backup point. SDDC Manager shows the pre-upgrade version again; ESX hosts remain connected to the source appliance throughout.
Snapshot revert of the upgraded appliance is not the documented path for an RDU — the old appliance is still there until you remove it. Take the file-based backup before the upgrade (and lower DRS automation per the procedure) so the restore path exists if retry and Cancel both fail.
Step 2

Practice SDDC Manager rollback from backup. Steps: (1) Power off and rename the failed SDDC Manager VM, (2) download and decrypt the file-based backup (openssl) and read metadata.json for the OVA version, hostname, vsan_datastore and port_group values, (3) deploy that same-version SDDC Manager OVA with the matching hostname, datastore and port group, snapshot it and power it on (VCF 9.0.1 docs p.3011), (4) copy the backup to /tmp, obtain an API token and start the restore with POST /v1/restores/tasks (there is no restore wizard; p.364), then verify the SDDC Manager services are up, (5) Verify all domains and hosts appear in inventory, (6) Reconcile any state differences.

SDDC Manager restore takes 30-60 minutes. Backup includes: database (PostgreSQL), configuration files, and certificate state. After restore, SDDC Manager must re-establish communication with all managed components (vCenter, NSX, ESXi hosts).
SDDC Manager rollback is more complex than vCenter because it is the orchestration layer. After restore, verify all managed component versions match SDDC Manager's expected state.
Step 3

Practice NSX Manager recovery after a failed Management Plane upgrade. Steps: (1) Check cluster status (get cluster status) and the upgrade coordinator error. (2) Do not add or delete Manager nodes during an upgrade (VCF 9.0.1 docs p.1400). (3) If NSX prompts a rollback and edge/host upgrades have not started, run on all nodes as root '/etc/init.d/corfu-server stop', then as admin 'node-rollback run-step step1_start_rollback' on all nodes, 'node-rollback run-step step2_restore_data' on one node, and 'node-rollback run-step step3_exit_rollback' on all nodes — the local backup in /config_bak is used and the nodes return to the pre-upgrade version (p.1401). (4) If edge or host upgrades have already started, open a Broadcom support request instead. (5) Verify all three nodes STABLE and retry the upgrade. Document the timeline.

NSX Manager Management Plane rollback: 60-120 minutes across the three nodes. DFW rules continue enforcing on the ESX hosts regardless of NSX Manager state. After rollback the cluster reports STABLE at the pre-upgrade version and the upgrade can be retried.
NSX Manager cluster is designed for rolling upgrade — only one node upgrades at a time. If a node fails, the remaining 2 nodes maintain management plane availability. Never attempt to upgrade a second node while the first is in failed state.
Step 4

Handle the partial upgrade scenario: NSX upgraded (management plane and edges) but the vCenter upgrade failed — in the documented order NSX precedes vCenter (SDDC Manager → NSX → vCenter → ESX, KB 390634), so the realistic mixed state is NSX 9.0 with vCenter 8.0 U3. Document: (a) is this a supported mixed-version state? Check VMware Interoperability Matrix. (b) Can you proceed with NSX upgrade retry? (c) Should you rollback vCenter to match NSX? Analyze the decision tree.

Partial upgrade state assessment: VMware Interoperability Matrix defines which version combinations are supported. Check the Broadcom Product Interoperability Matrix for the exact NSX 9.0 / vCenter 8.0 U3 pairing rather than assuming a generic 'N+1 with N' allowance. Decision: retry the vCenter upgrade if the failure was transient; roll NSX back from its pre-upgrade backup only if the matrix shows the pairing is fundamentally incompatible.
VCDX design point: upgrade sequence design must account for partial failure states. Document the supported mixed-version windows and the decision criteria for retry vs. rollback.
Step 5

Design post-rollback validation procedure. After any rollback: (1) Verify all VCF component versions match expected state, (2) Run SDDC Manager health check, (3) Verify vSAN cluster health, (4) Verify NSX controller cluster stability, (5) Run DFW Traceflow tests for critical paths, (6) Verify all VMs are running and accessible, (7) Test VCF operations (VM provisioning, host add). Document as a runbook.

Post-rollback validation checklist with 7+ items ensuring environment integrity. Each item has specific commands/tests and expected results. All checks must pass before declaring rollback successful.
Post-rollback validation is as important as the rollback itself. A 'successful' rollback that leaves orphaned objects or broken state is worse than the original failure.

Validation Gate

Check: Execute rollback procedures and validate environment recovery

Expected: vCenter snapshot revert practiced, SDDC Manager restore documented, NSX cluster recovery understood, post-rollback validation checklist created.

Common Errors

Reverting vCenter snapshot while VMs are still referencing new-version features — can cause VM configuration issues
Not reconciling SDDC Manager state after vCenter rollback — SDDC Manager shows version mismatch
Attempting to upgrade second NSX Manager node when first is in failed state — risks losing cluster quorum
Skipping post-rollback validation — orphaned objects and broken state go undetected

Task 4 Upgrade Prevention Strategy & VCDX Defense

Synthesis task connecting upgrade operations to architectural design. Panelists expect: upgrade strategy document, maintenance window planning, and risk mitigation.

Design a comprehensive upgrade strategy that minimizes failure risk and prepare to defend VCF lifecycle management design decisions in VCDX context.

Step 1

Design a VCF upgrade strategy document. Include: (a) upgrade policy (frequency: quarterly patches, semi-annual minor, annual major), (b) maintenance window schedule (management domain: weekend 1, workload domain 1: weekend 2, workload domain 2: weekend 3), (c) pre-upgrade checklist reference, (d) rollback decision criteria (time-boxed: if not resolved in 2 hours, rollback), (e) communication plan (stakeholders notified 2 weeks before, day-of status updates).

Upgrade strategy document with 5 sections covering policy, scheduling, preparation, execution, and communication. Each section has specific templates and decision criteria.
A well-documented upgrade strategy demonstrates operational maturity. VCDX panelists specifically ask about lifecycle management — show that your design includes day-2 operations, not just day-0 deployment.
Step 2

Calculate maintenance window requirements. For each upgrade step, estimate: preparation time, execution time, verification time, and rollback buffer. Total management domain upgrade: pre-checks (2h) + SDDC Manager (1h) + vCenter (2h) + NSX (3h) + ESXi hosts (4h for rolling upgrade of 4 hosts) + validation (1h) = ~13 hours + 2h rollback buffer = 15-hour maintenance window.

Maintenance window calculation showing each step with time estimate. Total management domain upgrade: 13-15 hours. Each workload domain: 6-8 hours (NSX, then vCenter, then ESX — the same component order as the management domain). Stagger across weekends to limit blast radius.
Maintenance window planning is a VCDX design deliverable. Show the math: hours per component, rollback buffer, verification time. Under-estimating maintenance windows causes rushed upgrades and increased failure risk.
Step 3

Design upgrade testing strategy using Holodeck. Steps: (1) Deploy Holodeck VCF environment matching production version, (2) Run the complete upgrade in Holodeck first, (3) Document any issues encountered and resolution, (4) Validate upgrade success in Holodeck, (5) Use Holodeck results to refine production upgrade plan. Document the value of pre-production upgrade testing.

Holodeck upgrade testing catches product bugs, validates procedures, and trains operations team before production upgrade. Investment: 4-8 hours of testing saves potential multi-day outage from untested upgrade.
Holodeck pre-production testing is your best upgrade risk mitigation tool. VCDX panelists ask: How do you validate your upgrade procedure? 'We test in Holodeck first' is a strong answer.
Step 4

Prepare four VCDX defense responses: (1) 'In what order do you upgrade the domains?' — SDDC Manager first, because it orchestrates every other upgrade and must be at the target version; after that, VCF 9.0 allows workload domains to be upgraded before or after the management domain as long as every component in the domain is compatible (p.1366) — the hard rule is only that the management vCenter precedes workload vCenters (p.1416). SDDC Manager must be at the latest version to manage the upgrade workflow for workload domains. vCenter and NSX in management domain are upgraded next because workload domain upgrades depend on management domain health. (2) 'What is your rollback strategy for a failed ESX host upgrade?' — reboot the host and press Shift+R when the hypervisor progress bar starts loading to boot the previous image from the alternate boot bank (Broadcom KB 316592; reverting between 9.0, 8.0 and 7.0 is possible); if the alternate bank is unusable, rebuild the host from the cluster image and rejoin. vSAN resync handles data redistribution. (3) 'How do you handle a upgrade that spans multiple maintenance windows?' — Document the partial upgrade state, verify component interoperability using VMware matrix, schedule remaining steps in next window. Never leave an environment in untested partial upgrade state indefinitely. (4) 'What is the most critical pre-upgrade step?' — the documented backups: file-based (or image-based) backups of vCenter and NSX Manager and a cold snapshot of SDDC Manager, verified before the window (VCF 9.0.1 docs p.1366). Without them, recovery from a failed step means a rebuild; with them, each component has a documented restore path.

Four defense responses demonstrating lifecycle management expertise and operational maturity.
VCDX defense for upgrades: demonstrate that you plan for failure, not just success. The rollback strategy is as important as the upgrade strategy.

Validation Gate

Check: Complete upgrade strategy document and VCDX defense preparation

Expected: Upgrade strategy with policy, scheduling, and communication plan. Maintenance window calculations documented. Holodeck testing strategy defined. Four VCDX defense responses prepared.

Common Errors

Underestimating maintenance window duration — rushed upgrades increase failure risk
Not testing upgrades in Holodeck first — production becomes the test environment
Leaving environments in partial upgrade state across multiple weeks — untested mixed-version states accumulate risk
Not including rollback buffer in maintenance window — no time for recovery if upgrade fails

Final Validation

Complete VCF upgrade failure remediation with rollback procedures, prevention strategy, and VCDX defense

✓ Upgrade sequence mastered → KB 390634 update sequence documented (management planes, then SDDC Manager → NSX → vCenter → ESX) with dependency justification

✓ Failure diagnosis capability → Log analysis for SDDC Manager, vCenter, NSX, and ESXi upgrade failures

✓ Rollback procedures tested → vCenter snapshot revert, SDDC Manager restore, NSX cluster recovery documented

✓ Prevention strategy designed → Pre-upgrade checklist, Holodeck testing, maintenance window planning

✓ VCDX defense prepared → Four defense responses with lifecycle management expertise

Cleanup / Restore

• Verify all VCF components are running at consistent versions

• Remove any temporary snapshots taken during practice

• Confirm SDDC Manager health dashboard is green

• Revert to snapshot if needed

Design Reflection (VCDX)

VCF upgrade lifecycle management demonstrates operational maturity and risk management. The design connects upgrade strategy to business continuity requirements with structured maintenance windows, Holodeck pre-validation, and graduated rollback procedures. The KB 390634 update sequence (management planes first, then SDDC Manager → NSX → vCenter → ESX) is a core VCF operational competency.

Requirements

  • R-001: VCF upgrades must complete within defined maintenance windows
  • R-002: Upgrade rollback must be possible within 2 hours of failure detection
  • R-003: Zero data loss during upgrade operations

Constraints

  • Upgrade sequence must follow KB 390634 order
  • Maintenance windows limited to weekends (business constraint)
  • Holodeck test environment must match production version baseline

Assumptions

  • VM snapshots taken before every component upgrade
  • SDDC Manager and NSX Manager backups available and tested
  • Operations team trained on rollback procedures

Risks

  • Product bug in upgrade path causes non-retryable failure — mitigate with Holodeck pre-testing and VMware SR readiness
  • Maintenance window exceeded due to unexpected failure — time-boxed rollback decision prevents extended outage
  • Partial upgrade state left across multiple windows — verify interoperability matrix and schedule remaining steps promptly

Self-Assessment Discussion Prompts

  1. How would you handle a VCF upgrade in a stretched cluster environment where both sites must be upgraded?
  2. What is your strategy for upgrading 10+ workload domains efficiently?
  3. How do you decide between patch-level updates and full version upgrades?
  4. What metrics would you track to measure upgrade program maturity?

Extensions

Automated Pre-Upgrade Validation

Build a PowerCLI or Python script that runs the complete pre-upgrade checklist automatically: DNS/NTP validation, component health, vSAN capacity, snapshot verification, backup verification. Output a go/no-go report.

VCF Upgrade with Async Patch Tool

Practice using the VCF Async Patch Tool for applying emergency patches outside the normal upgrade cycle. Document: when to use Async Patch vs. full upgrade, how to validate async patch compatibility, and rollback procedures.

Multi-Domain Upgrade Orchestration

Design an upgrade plan for a VCF environment with 1 management domain and 5 workload domains. Calculate total maintenance window requirements, stagger domain upgrades, and design the verification matrix for cross-domain compatibility.

⚠ Known Pitfalls

Upgrading vCenter before SDDC Manager — SDDC Manager must upgrade first to orchestrate subsequent components; out-of-order upgrade breaks the workflow engine
Skipping the documented pre-upgrade backups (file-based or image-based backups of vCenter and NSX, cold snapshot of SDDC Manager) — without them a failed step means a rebuild; vCenter RDU has its own Cancel and automatic rollback, so appliance snapshots are not the mechanism
Forcing ESXi host into maintenance mode when vSAN evacuation stalls — risks data loss; diagnose and resolve the evacuation blocker first
Not time-boxing upgrade troubleshooting — spending 4+ hours diagnosing a failure during a maintenance window when rollback takes 30 minutes; define rollback decision criteria upfront

References

  • Broadcom KB 390634 — VCF Upgrade Sequence and Component Order
  • VMware VCF 9.0 Lifecycle Management Guide — Upgrade Planning and Execution
  • vCenter Server 9.0 Upgrade Guide — VCSA Migration and Rollback
  • NSX 9.0 Upgrade Guide — Manager Cluster Upgrade Procedures
  • VMware VCF Async Patch Tool Documentation
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.