Lab: Plan & Execute VCF Operations Cluster Upgrade
Objectives
- Plan a VCF Operations cluster upgrade with pre-flight validation
- Execute a whole-cluster software update covering the primary, replica and data nodes (and the cloud proxies)
- Verify upgrade completion and data integrity post-upgrade
- Understand PAK file management and offline upgrade procedures
- Design a rollback plan for failed upgrade scenarios
- Monitor upgrade progress and handle common failure modes
Prerequisites
VCF Operations cluster (multi-node preferred: primary node + 2 data nodes minimum) running a version that can be upgraded, upgrade PAK file downloaded, snapshot/backup of VCF Operations nodes taken
Required skills:
- VCF Operations administration
- vSphere snapshot management
- Change management procedures
Lab Environment
VCF Operations cluster with minimum 3 nodes (primary node, data node 1, data node 2) or single-node for smaller lab. PAK file for the target version available. vCenter backup taken. SSH access to VCF Operations nodes for troubleshooting.
Tasks
Task 1 VCF Operations Cluster Upgrade with Rollback Planning
Execute a controlled upgrade of a VCF Operations cluster following production best practices: pre-flight validation, whole-cluster software update (all nodes, cluster offline for the update window), post-upgrade verification, and rollback planning. Understanding the upgrade lifecycle is critical for VCAP-level operations and ensures zero data loss during version transitions.
Pre-flight: Check current version and compatibility. VCF Operations → Administration → Cluster Status. Note: current version (e.g., 9.0.0.x), node count, node health. (For an Aria Operations 8.x source, the supported path is 8.14 or later → VCF Operations 9.0 through Aria Suite Lifecycle 8.18 Patch 6 or later, VCF 9.0.1 docs p.1350.) Verify the target version is compatible with your current version (check the upgrade path matrix on techdocs.broadcom.com — some versions require intermediate upgrades). Verify: vCenter version compatibility, NSX version compatibility, browser compatibility.
Pre-flight: Verify cluster health. Administration → Cluster Status → all nodes should show 'Online' with green health. Check: (a) the primary node is available, (b) All adapters collecting data (no 'Down' adapters), (c) No active critical alerts on the VCF Operations cluster itself, (d) no node shows an error or degraded state in the admin UI cluster status. Do not check the internal database directly from the shell: its engine and tools are not documented product interfaces.
Pre-flight: Take snapshots the documented way. Log in to the admin UI https://<primary-node>/admin and click Take Offline under the cluster status. When all nodes are offline, in the vSphere Client right-click each VCF Operations VM → Snapshot → Take Snapshot: name 'Pre-Update-<version>-<date>', deselect 'Snapshot the Virtual Machine Memory' and deselect 'Ensure Quiesce Guest File System'. Repeat for every node. Consistency comes from the cluster being offline, not from a timing window; memory snapshots of the running analytics cluster are what the procedure avoids. These snapshots are the rollback safety net — if the update fails, revert all nodes to them. Also verify: vCenter backup is current (VCF Operations upgrade can affect vCenter integration).
Pre-flight: Check disk space. SSH to the primary node: df -h /storage. VCF Operations upgrade requires at least 5 GB free space for PAK extraction and upgrade staging. If space is low: (a) reduce metric retention temporarily, (b) clean log files: $VCOPS_BASE/user/log/*.gz, (c) extend the data disk in vSphere if needed (VCF Operations supports online disk expansion).
Upload PAK file. Administration → Software Update → Upload. Select the PAK file (e.g., Operations-Upgrade-9.0.0.X.XXXXXXXX.pak; VCF 9.0.1 docs p.1353). The upload process validates the file integrity (checksum) and compatibility with the current version. If validation fails: re-download the PAK file (corrupt download), or check the upgrade path matrix. For air-gapped environments, transfer the PAK via SCP to /tmp on the primary node.
Execute the update. Admin UI → Software Update → Install a Software Update → upload the PAK file (VCF Operations and cloud proxy together) → accept the EULA → Install. The installer stages the PAK to every node, applies it and restarts all appliances; all nodes are upgraded by default. During the update the whole cluster is offline: no metrics are collected until the status returns to Online (log back in to the primary node admin UI; the cluster comes online automatically — do not click Bring Online; clear the browser cache and watch Going Online → Online). Cloud proxies start their upgrade after the cluster upgrade completes. If the cluster drops to offline mid-install, take it offline manually in the admin UI and click Finish Installation.
Monitor upgrade progress. After you click Install, the installer restarts the admin interface and logs you out; log back in to the primary node admin UI when it is ready and follow the update status on the Software Update page. Optional CLI view (not part of the documented procedure): SSH to the primary node → tail -f $VCOPS_BASE/user/log/upgradeStatus.log. Key log entries to watch: 'Upgrade started', 'Database migration in progress' (longest phase), 'Services restarting', 'Upgrade completed'. If a node is stuck > 60 minutes: check upgradeStatus.log for errors, do NOT reboot (may corrupt the database migration).
Post-upgrade verification. After the Software Update page reports that the update completed successfully and the cluster status is Online: (a) Administration → Cluster Status — all nodes Online with new version number; (b) Check adapter status — all adapters should resume collection within 15 minutes; (c) Verify dashboards display data (some may need refresh); (d) Check super metrics — custom formulas should survive the upgrade, but verify values are computing; (e) Re-check the admin UI cluster status — no node in an error or degraded state (the internal database is not checked directly; see step 2); (f) Verify API access: curl -s -k https://<vcf-ops>/suite-api/api/versions — should return the new version.
Post-upgrade: Validate data continuity. Compare key metrics before and after upgrade: (a) Check a VM's CPU usage chart — historical data is continuous up to Take Offline and resumes when the cluster is Online; the only gap is the documented offline window; (b) Verify super metric calculations match pre-upgrade values; (c) Check custom group membership — dynamic groups should have the same member count; (d) Verify scheduled reports are still configured and scheduled. Any discrepancy indicates a data migration issue — escalate to VMware support before deleting pre-upgrade snapshots.
Cleanup and documentation. As soon as the update is verified (cluster Online, adapters collecting, dashboards and super metrics working): (a) delete the pre-update snapshots — the procedure requires it to avoid performance degradation, and a later revert would discard every metric collected since the update; (b) Update CMDB with new VCF Operations version; (c) Document the upgrade in the change management system: start time, duration of the offline window, any issues encountered, rollback snapshots deleted; (d) If the upgrade ran through Aria Suite Lifecycle or VCF Operations fleet management, verify its inventory shows the new version. Rollback option (only while the snapshots exist): power off all nodes, revert all to the pre-update snapshots, power on the primary node first, then the others, and bring the cluster online from the admin UI. This restores the pre-update state and loses the data collected after the snapshot — which is why the rollback window is kept short.
Validation Gate
Check: VCF Operations cluster upgraded with data integrity verified
Expected: All nodes upgraded to target version, cluster health green, all adapters collecting, metrics continuous except for the documented offline window, super metrics and dashboards functional, scheduled reports intact
Common Errors
Final Validation
VCF Operations cluster successfully upgraded with verified data integrity and operational continuity
✓ Cluster version → All nodes show target version in Administration → Cluster Status
✓ Node health → All nodes Online with green health status
✓ Adapter collection → All adapters showing 'Collecting' status with current timestamps
✓ Data continuity → The only gap in historical charts is the documented offline window (Take Offline through Online); no gaps before or after it
✓ Super metrics → All custom super metrics computing values correctly
✓ Dashboards and reports → All dashboards display data, scheduled reports intact
Cleanup / Restore
• Delete the pre-update snapshots once the update is verified (cluster Online, collection resumed)
• Update change management records with upgrade details
• Archive upgrade PAK file for rollback reference
Design Reflection (VCDX)
Upgrade planning demonstrates operational maturity — a VCDX differentiator. Key points: (1) pre-flight validation prevents failed upgrades (disk space, cluster health, snapshot safety net); (2) the update takes the whole cluster offline, so the monitoring gap is a scheduled, accepted risk — state the window and what is not monitored during it; (3) data continuity verification confirms collection resumed cleanly after Online; (4) snapshots are deleted as soon as the update is verified, so the rollback window is deliberately short — a longer bake period would trade metric history for rollback. In VCF 9.0, discuss how VCF Operations fleet management handles lifecycle of the management components, and how the 8.x → 9.0 move through Aria Suite Lifecycle 8.18 Patch 6+ differs from a direct PAK update.
Requirements
- Zero data loss during upgrade
- Monitoring gap limited to the scheduled update window (cluster offline through Online)
- Rollback capability until post-update verification completes (snapshots then deleted)
Constraints
- Upgrade is non-reversible once all nodes complete — rollback requires snapshot revert
- Some adapters may require re-registration post-upgrade
- Air-gapped environments require manual PAK file transfer
Assumptions
- Pre-update snapshots are taken on every node while the cluster is offline
- Sufficient disk space (5+ GB) for PAK extraction and migration
- vCenter and NSX versions are compatible with target VCF Operations version
Risks
- Database migration failure during upgrade — long outage if rollback is needed
- Snapshot revert after partial upgrade leaves cluster in inconsistent state — must revert ALL nodes simultaneously
- Third-party management pack incompatibility with new version — breaks monitoring for non-VMware systems
Self-Assessment Discussion Prompts
- How do you plan VCF Operations upgrades when the cluster monitors the infrastructure being upgraded?
- What is the difference between upgrading through VCF Operations fleet management (or Aria Suite Lifecycle for the 8.x → 9.0 move) and a direct PAK update from the admin UI?
- How do you handle upgrade testing in environments without a non-production VCF Operations instance?
- What monitoring gaps are acceptable during upgrade, and how do you mitigate them?
Extensions
Practice the 8.x → 9.0 move with VMware Aria Suite Lifecycle 8.18 Patch 6 or later (which deploys the VCF Operations fleet management appliance) instead of a direct PAK update
Design an upgrade runbook with pre/post checks as a reusable template
Test rollback by intentionally reverting snapshots after a successful upgrade to validate the procedure
Create an automated pre-flight check script using the VCF Operations API
⚠ Known Pitfalls
References
- VCF Operations Upgrade Guide: techdocs.broadcom.com
- VCF 9.0 documentation — Upgrade VMware Aria Operations to VCF Operations 9.0 / Install a Software Update: techdocs.broadcom.com