Upgrade Failure & Remediation
Objectives
- Understand the VCF 9.0 update sequence and component dependencies (KB 390634: management planes first, then SDDC Manager → NSX → vCenter → ESX)
- Diagnose upgrade failures using SDDC Manager logs, vCenter upgrade logs, and NSX upgrade status
- Execute vCenter upgrade recovery for a reduced-downtime upgrade (retry, Cancel, automatic rollback, file-based restore)
- Recover failed SDDC Manager upgrade workflows using API retry and manual intervention
- Design pre-upgrade validation checklists to prevent common upgrade failures
- Document upgrade failure recovery using RCAR methodology
Prerequisites
VCF lab with management domain running pre-upgrade version
Prior labs: vcp-support-01
Required skills:
- SDDC Manager UI and API navigation
- vCenter snapshot management
- SSH access to VCF components
Lab Environment
VCF management domain with SDDC Manager, vCenter, NSX Manager cluster, and 4 ESXi hosts
- Hosts needed: 4
- VMs needed: 3
- Storage: vSAN ESA datastore
Tasks
Task 1 VCF Upgrade Architecture & Sequence Planning
Master the 19-step VCF upgrade sequence, understand component interdependencies, and identify which steps are most likely to fail and why.
Document the VCF 9.0 update sequence per Broadcom KB 390634 (Update Sequence for VCF 9.0 and Compatible VMware Products). Fleet management plane first: (0) VCFDT, (1) VMware Aria Suite Lifecycle 8.18 patch (patch only; no upgrade to 9.0 — the KB names Patch 2, the VCF 9.0 upgrade documentation requires 8.18 Patch 6 or later, so apply Patch 6 or later), (2) VCF Operations + VCF Operations fleet management appliance, (3) VCF Automation and VCF Operations for networks (parallel), (4) VCF Operations for logs (new installation). Then (5) VADP-based backup, (6) vSphere Replication, (7) SRM, (8) Avi. Core SDDC: (9) SDDC Manager, (10) HCX, (11) NSX, (12) vCenter, (13) VCF Identity Broker (new installation) and Supervisor, (14) vSAN witness host, (15) ESX, then NSX finalize, (16) VMware Tools, (17) virtual hardware, (18) vSAN on-disk format / VMFS, (19) vSAN File Service.
Identify the most failure-prone upgrade steps and common causes. Document: (a) SDDC Manager upgrade — database migration issues, insufficient disk space; (b) vCenter upgrade — VCSA migration assistant failures, customization conflicts; (c) NSX Manager cluster upgrade — cluster stability during rolling upgrade, backup controller issues; (d) ESXi host upgrade — vSAN maintenance mode failures, incompatible hardware/drivers.
Map the recovery options for each component. vCenter: VCF 9.0 upgrades vCenter with a reduced-downtime upgrade (RDU, a new appliance on a temporary IP then switchover) — retry the failed task, Cancel to revert to the source vCenter while the RDU runs, automatic rollback on failure, or restore from the file-based backup (VCF 9.0.1 docs p.1417). SDDC Manager: restore from backup or revert the cold snapshot taken before the update (p.1366). NSX Manager: Management Plane rollback from the local /config_bak backup on all nodes (p.1401), or restore from backup. ESX: revert to the alternate boot bank (Shift+R at boot, KB 316592) or rebuild from the cluster image. Document the recovery time estimate for each.
Review the pre-upgrade validation checklist. Items: (a) verify bundle compatibility matrix in SDDC Manager, (b) check all VCF component health (SDDC Manager dashboard), (c) verify vSAN health and capacity (must be below 80% for host upgrades), (d) confirm DNS forward/reverse resolution for all components, (e) verify NTP synchronization across all components (<5 second drift), (f) take VM snapshots of vCenter, SDDC Manager, NSX Managers, (g) export NSX Manager backup, (h) document current versions of all components.
Validation Gate
Check: Complete VCF upgrade sequence and pre-upgrade checklist documentation
Expected: 19-step sequence documented with dependency justification. Failure matrix and rollback matrix created. Pre-upgrade checklist with 15+ items.
Common Errors
Task 2 Upgrade Failure Diagnosis & Log Analysis
Develop systematic log analysis skills for diagnosing VCF upgrade failures — from SDDC Manager workflow failures to vCenter migration issues to ESXi remediation errors.
Map the critical log locations for VCF upgrade troubleshooting. SDDC Manager upgrade logs: /var/log/vmware/vcf/lcm/ — lcm-debug.log is the log for failed upgrades of all components; from SDDC Manager 9.0 all LCM logging goes to lcm-debug.log (VCF 9.0.1 docs p.1456). NSX Manager: /var/log/proton/nsxapi.log. vCenter: the upgrade logs under /var/log/vmware/upgrade/ on the appliance (check the file names in the vCenter upgrade documentation).
Analyze a simulated vCenter upgrade failure. Common scenario: VCSA upgrade fails at database migration step. Log analysis: search the vCenter upgrade logs under /var/log/vmware/upgrade/ for 'ERROR' and 'FAILED'. The error typically shows: database schema migration failure, disk space exhaustion, or service dependency conflict. Document the error pattern and correlate with pre-upgrade checklist items.
Analyze SDDC Manager workflow failure. Navigate to SDDC Manager UI > Lifecycle Management > view failed workflow. Document: workflow ID, failed step, error message, and retry eligibility. Via API: GET /v1/tasks/{taskId} to retrieve detailed error information. Check if the workflow can be retried (some failures are retryable, others require manual intervention).
Practice ESXi host upgrade failure diagnosis. Common scenario: host enters maintenance mode but vSAN data evacuation stalls. Diagnose: (a) check vSAN resync status (objects still syncing from previous operation), (b) verify cluster capacity allows data migration, (c) check for VM anti-affinity rules preventing evacuation, (d) verify no VMs are pinned to the host. Document remediation for each cause.
Build an upgrade failure escalation decision tree. Level 1: retry the failed workflow (if retryable). Level 2: analyze logs, fix the identified issue, retry. Level 3: rollback the failed component (snapshot/backup restore), fix root cause, re-attempt. Level 4: escalate to VMware support with: support bundle, upgrade logs, and detailed failure description.
Validation Gate
Check: Complete upgrade failure diagnosis with log analysis and escalation procedures
Expected: Log locations mapped for all VCF components. Failure patterns documented. Escalation decision tree created with time-boxing.
Common Errors
Task 3 Rollback Procedures & Recovery Operations
Execute component-level recovery procedures for failed VCF upgrades — vCenter RDU retry/Cancel/auto-rollback and file-based restore, SDDC Manager cold-snapshot revert or backup restore, NSX Manager Management Plane rollback.
Practice vCenter upgrade recovery. VCF 9.0 upgrades vCenter with a reduced-downtime upgrade: a new appliance is deployed on a temporary IP, data is migrated, then switchover. Steps: (1) If the task fails, read the error in the Updates view, resolve it and retry the failed task. (2) While the RDU is running, Cancel reverts the configuration to the source vCenter. (3) RDU upgrades roll back automatically on failure. (4) If the issue cannot be resolved, restore vCenter from its file-based backup (Restoring a vCenter Instance). (5) Only after a successful upgrade remove the old appliance; allow ~15 minutes for post-upgrade drift reconciliation before the next upgrade (VCF 9.0.1 docs p.1417).
Practice SDDC Manager rollback from backup. Steps: (1) Power off and rename the failed SDDC Manager VM, (2) download and decrypt the file-based backup (openssl) and read metadata.json for the OVA version, hostname, vsan_datastore and port_group values, (3) deploy that same-version SDDC Manager OVA with the matching hostname, datastore and port group, snapshot it and power it on (VCF 9.0.1 docs p.3011), (4) copy the backup to /tmp, obtain an API token and start the restore with POST /v1/restores/tasks (there is no restore wizard; p.364), then verify the SDDC Manager services are up, (5) Verify all domains and hosts appear in inventory, (6) Reconcile any state differences.
Practice NSX Manager recovery after a failed Management Plane upgrade. Steps: (1) Check cluster status (get cluster status) and the upgrade coordinator error. (2) Do not add or delete Manager nodes during an upgrade (VCF 9.0.1 docs p.1400). (3) If NSX prompts a rollback and edge/host upgrades have not started, run on all nodes as root '/etc/init.d/corfu-server stop', then as admin 'node-rollback run-step step1_start_rollback' on all nodes, 'node-rollback run-step step2_restore_data' on one node, and 'node-rollback run-step step3_exit_rollback' on all nodes — the local backup in /config_bak is used and the nodes return to the pre-upgrade version (p.1401). (4) If edge or host upgrades have already started, open a Broadcom support request instead. (5) Verify all three nodes STABLE and retry the upgrade. Document the timeline.
Handle the partial upgrade scenario: NSX upgraded (management plane and edges) but the vCenter upgrade failed — in the documented order NSX precedes vCenter (SDDC Manager → NSX → vCenter → ESX, KB 390634), so the realistic mixed state is NSX 9.0 with vCenter 8.0 U3. Document: (a) is this a supported mixed-version state? Check VMware Interoperability Matrix. (b) Can you proceed with NSX upgrade retry? (c) Should you rollback vCenter to match NSX? Analyze the decision tree.
Design post-rollback validation procedure. After any rollback: (1) Verify all VCF component versions match expected state, (2) Run SDDC Manager health check, (3) Verify vSAN cluster health, (4) Verify NSX controller cluster stability, (5) Run DFW Traceflow tests for critical paths, (6) Verify all VMs are running and accessible, (7) Test VCF operations (VM provisioning, host add). Document as a runbook.
Validation Gate
Check: Execute rollback procedures and validate environment recovery
Expected: vCenter snapshot revert practiced, SDDC Manager restore documented, NSX cluster recovery understood, post-rollback validation checklist created.
Common Errors
Task 4 Upgrade Prevention Strategy & VCDX Defense
Design a comprehensive upgrade strategy that minimizes failure risk and prepare to defend VCF lifecycle management design decisions in VCDX context.
Design a VCF upgrade strategy document. Include: (a) upgrade policy (frequency: quarterly patches, semi-annual minor, annual major), (b) maintenance window schedule (management domain: weekend 1, workload domain 1: weekend 2, workload domain 2: weekend 3), (c) pre-upgrade checklist reference, (d) rollback decision criteria (time-boxed: if not resolved in 2 hours, rollback), (e) communication plan (stakeholders notified 2 weeks before, day-of status updates).
Calculate maintenance window requirements. For each upgrade step, estimate: preparation time, execution time, verification time, and rollback buffer. Total management domain upgrade: pre-checks (2h) + SDDC Manager (1h) + vCenter (2h) + NSX (3h) + ESXi hosts (4h for rolling upgrade of 4 hosts) + validation (1h) = ~13 hours + 2h rollback buffer = 15-hour maintenance window.
Design upgrade testing strategy using Holodeck. Steps: (1) Deploy Holodeck VCF environment matching production version, (2) Run the complete upgrade in Holodeck first, (3) Document any issues encountered and resolution, (4) Validate upgrade success in Holodeck, (5) Use Holodeck results to refine production upgrade plan. Document the value of pre-production upgrade testing.
Prepare four VCDX defense responses: (1) 'In what order do you upgrade the domains?' — SDDC Manager first, because it orchestrates every other upgrade and must be at the target version; after that, VCF 9.0 allows workload domains to be upgraded before or after the management domain as long as every component in the domain is compatible (p.1366) — the hard rule is only that the management vCenter precedes workload vCenters (p.1416). SDDC Manager must be at the latest version to manage the upgrade workflow for workload domains. vCenter and NSX in management domain are upgraded next because workload domain upgrades depend on management domain health. (2) 'What is your rollback strategy for a failed ESX host upgrade?' — reboot the host and press Shift+R when the hypervisor progress bar starts loading to boot the previous image from the alternate boot bank (Broadcom KB 316592; reverting between 9.0, 8.0 and 7.0 is possible); if the alternate bank is unusable, rebuild the host from the cluster image and rejoin. vSAN resync handles data redistribution. (3) 'How do you handle a upgrade that spans multiple maintenance windows?' — Document the partial upgrade state, verify component interoperability using VMware matrix, schedule remaining steps in next window. Never leave an environment in untested partial upgrade state indefinitely. (4) 'What is the most critical pre-upgrade step?' — the documented backups: file-based (or image-based) backups of vCenter and NSX Manager and a cold snapshot of SDDC Manager, verified before the window (VCF 9.0.1 docs p.1366). Without them, recovery from a failed step means a rebuild; with them, each component has a documented restore path.
Validation Gate
Check: Complete upgrade strategy document and VCDX defense preparation
Expected: Upgrade strategy with policy, scheduling, and communication plan. Maintenance window calculations documented. Holodeck testing strategy defined. Four VCDX defense responses prepared.
Common Errors
Final Validation
Complete VCF upgrade failure remediation with rollback procedures, prevention strategy, and VCDX defense
✓ Upgrade sequence mastered → KB 390634 update sequence documented (management planes, then SDDC Manager → NSX → vCenter → ESX) with dependency justification
✓ Failure diagnosis capability → Log analysis for SDDC Manager, vCenter, NSX, and ESXi upgrade failures
✓ Rollback procedures tested → vCenter snapshot revert, SDDC Manager restore, NSX cluster recovery documented
✓ Prevention strategy designed → Pre-upgrade checklist, Holodeck testing, maintenance window planning
✓ VCDX defense prepared → Four defense responses with lifecycle management expertise
Cleanup / Restore
• Verify all VCF components are running at consistent versions
• Remove any temporary snapshots taken during practice
• Confirm SDDC Manager health dashboard is green
• Revert to snapshot if needed
Design Reflection (VCDX)
VCF upgrade lifecycle management demonstrates operational maturity and risk management. The design connects upgrade strategy to business continuity requirements with structured maintenance windows, Holodeck pre-validation, and graduated rollback procedures. The KB 390634 update sequence (management planes first, then SDDC Manager → NSX → vCenter → ESX) is a core VCF operational competency.
Requirements
- R-001: VCF upgrades must complete within defined maintenance windows
- R-002: Upgrade rollback must be possible within 2 hours of failure detection
- R-003: Zero data loss during upgrade operations
Constraints
- Upgrade sequence must follow KB 390634 order
- Maintenance windows limited to weekends (business constraint)
- Holodeck test environment must match production version baseline
Assumptions
- VM snapshots taken before every component upgrade
- SDDC Manager and NSX Manager backups available and tested
- Operations team trained on rollback procedures
Risks
- Product bug in upgrade path causes non-retryable failure — mitigate with Holodeck pre-testing and VMware SR readiness
- Maintenance window exceeded due to unexpected failure — time-boxed rollback decision prevents extended outage
- Partial upgrade state left across multiple windows — verify interoperability matrix and schedule remaining steps promptly
Self-Assessment Discussion Prompts
- How would you handle a VCF upgrade in a stretched cluster environment where both sites must be upgraded?
- What is your strategy for upgrading 10+ workload domains efficiently?
- How do you decide between patch-level updates and full version upgrades?
- What metrics would you track to measure upgrade program maturity?
Extensions
Automated Pre-Upgrade Validation
Build a PowerCLI or Python script that runs the complete pre-upgrade checklist automatically: DNS/NTP validation, component health, vSAN capacity, snapshot verification, backup verification. Output a go/no-go report.
VCF Upgrade with Async Patch Tool
Practice using the VCF Async Patch Tool for applying emergency patches outside the normal upgrade cycle. Document: when to use Async Patch vs. full upgrade, how to validate async patch compatibility, and rollback procedures.
Multi-Domain Upgrade Orchestration
Design an upgrade plan for a VCF environment with 1 management domain and 5 workload domains. Calculate total maintenance window requirements, stagger domain upgrades, and design the verification matrix for cross-domain compatibility.
⚠ Known Pitfalls
References
- Broadcom KB 390634 — VCF Upgrade Sequence and Component Order
- VMware VCF 9.0 Lifecycle Management Guide — Upgrade Planning and Execution
- vCenter Server 9.0 Upgrade Guide — VCSA Migration and Rollback
- NSX 9.0 Upgrade Guide — Manager Cluster Upgrade Procedures
- VMware VCF Async Patch Tool Documentation