Academy/VCF 9.0 Support (2V0-15.25)
VCF

VCF 9.0 Support (2V0-15.25)

VCF 9.0supportvcp-foundationadminvcdx

Comprehensive study guide for the VMware Cloud Foundation 9.0 Support exam (2V0-15.25). Covers VCF and vSphere Foundation architecture, installation, operations, upgrades, licensing, workload domains, VCF Automation, Broadcom support processes, and deep troubleshooting scenarios. Enriched with content from the VCFFTS9 lecture manual as supplementary material; the exam guide's recommended course is VMware Cloud Foundation: Troubleshooting.

2V0-15.25
VCP
60
Questions
135m
Duration
300 (scaled)
Pass Score
59
Objectives

Exam Blueprint Weights

Section titles, groupings and weights below are VCDX Academy study groupings, NOT the official Broadcom blueprint structure. Broadcom publishes no section weights. Always cross-check the official exam guide. Official exam guide ↗
Section 1 — Troubleshooting Methodology and Tools
~10%
Section 2 — vSphere and Compute Troubleshooting
~22%
Section 3 — vSAN Storage Troubleshooting
~22%
Section 4 — NSX Networking Troubleshooting
~22%
Section 5 — VCF Operations and Platform Troubleshooting
~24%

Version Evolution

VCFFTS9 is VCF 9.0 focused. Covers both VCF Private Cloud and vSphere Foundation. Key VCF 9.0 changes: VCF Installer replaces Cloud Builder, core-based licensing replaces socket-based, VCF Operations console is integrated, ESXi host prep via VCF Installer wizard. Course includes upgrade paths from VCF 5.x and vSphere-to-VCF conversion/import.

Learning Outcomes

  • Articulate VCF and vSphere Foundation use cases, differentiating between them
  • Deploy a VCF fleet using both the Installer wizard and JSON deployment specification
  • Navigate VCF Operations console to monitor storage, compute, and security health
  • Use Log Assist for log analysis and troubleshooting
  • Prepare and configure ESXi hosts for VCF deployment
  • Explain vSAN architecture and its role in VCF storage
  • Describe NSX core components and their role in VCF networking
  • Plan VCF 5.x to 9.0 upgrades and vSphere-to-VCF conversions
  • Understand the VCF 9.0 licensing model (core-based subscription)
  • Create and manage workload domains and network pools
  • Describe VCF Automation capabilities
  • Navigate Broadcom Wolken case management for support cases
  • Prepare a support case: exact error text, SoS bundles from every affected host, vCenter logs for the last 24 hours, the list of recent changes (dates, names, approvals) and the business impact statement.
  • Map a reported environment (component versions, build numbers) to the applicable KB articles and fixed-in versions before proposing a change; version-specific fixes and the current BOM decide the remediation.
  • Obj 5.2: Troubleshoot a VCF 5.x to 9.0 upgrade and a vSphere to VCF 9.0 conversion — prechecks, binaries, upgrade sequence
  • Obj 5.3: Troubleshoot creating and scaling workload domains (adding/removing clusters and ESX hosts) and importing a vSphere environment as an isolated workload domain
  • Obj 5.11: Troubleshoot HCX configuration and HCX workload migrations — RAV, Bulk, cold migration, network extension

VCFFTS9 Exam Strategy & Lecture Manual Overview#

The VMware Cloud Foundation 9.0 Support exam (2V0-15.25) is a 60-item, 135-minute proctored exam with a passing score of 300 (scaled); it leads to VCP-VCF Support (exam guide last updated 15 Aug 2025). It tests only Section 5, Troubleshoot and Optimize: Sections 1 to 4 carry no testable objectives.

Objectives 5.1 to 5.11: 5.1 VCF fleet deployment and adding or removing VCF Instances; 5.2 VCF 5.x to 9.0 upgrade and vSphere to VCF 9.0 conversion; 5.3 workload domains (create, scale clusters and hosts, import a vSphere environment as an isolated workload domain); 5.4 VCF Operations fleet management (certificate CA, CSR, import, replacement and auto-renewal; password management; Identity Broker IAM); 5.5 license assignment, entitlements and expiry; 5.6 compute (ESX, vCenter, VMs, cluster configuration); 5.7 storage (vSAN, stretched clusters as VCF availability zones, supplemental iSCSI/NFS/FC); 5.8 networking (VDS, NSX configuration, routing, gateways, services, NSX VPC); 5.9 VCF Operations (configuration incl.
Operations for Logs and for Networks, Observability Workbench — the product docs call it the Troubleshooting Workbench, p.3912 — log bundle via Log Assist, dashboards, reports and alerts); 5.10 Identity Broker, VCF Automation configuration, Supervisor, provider management (VPCs, regions, content libraries, regional networks, storage and VM classes) and organizations (VM-App and All-App); 5.11 HCX configuration and workload migrations.

The minimally qualified candidate has about a year of supporting and troubleshooting VCF components and two years of IT experience, and may research advanced topics or involve Broadcom support. Recommended course: VMware Cloud Foundation: Troubleshooting. Study strategy: weight your time by the objective list above (compute, storage, networking and fleet management carry the most sub-objectives), and practise the VCF 9.0 tool surface: SoS on SDDC Manager, VCF Operations (fleet management, logs, networks, Observability Workbench, Log Assist), NSX Manager and ESX CLI.

Key Takeaways

  • The exam tests only Section 5 (objectives 5.1–5.11, all troubleshooting tasks); Broadcom publishes no weighting — prioritise by the objective list, not by lecture-manual page count
  • Recommended course: VMware Cloud Foundation: Troubleshooting; the VCFFTS9 lecture manual used in this section is supplementary material
  • Log Assist is a blueprint item (objective 5.9: generate a log bundle and upload it via Log Assist); case-management tooling and KB research are support practice, not blueprint objectives
  • Licensing is objective 5.5 (assignment, entitlements, expiry): understand the move from license keys to a license file in VCF 9.0
  • Cross-reference with existing Academy sections for deeper coverage

VCF Troubleshooting Methodology#

Structured Troubleshooting Approach

Effective troubleshooting follows a systematic process: identify the problem precisely, isolate the failing component, collect evidence, form hypotheses, test, and document the solution for future reference.

Troubleshooting Framework (The Four Ps):

Problem Statement:

"VMs cannot be deployed on cluster-1" is clear. "The system is slow" is vague. Quantify: "Deploy times increased from 2min to 10min starting Tuesday 3pm".

Planes Affected:

Which plane is failing? Management (vCenter down → cannot manage), Control (NSX controller cluster degraded → networking flakes), Data (ESXi host PSOD → VMs crash). Identify plane first to narrow scope.

Peripheral Devices:

Does the issue trace to storage (vSAN), network (NSX, physical NIC), compute (CPU/memory/hypervisor), or external dependencies (NTP, DNS)?

Past Changes:

What changed recently? Patch Tuesday, host reboot, network config update, SRM test failover? Track all changes in a log.

Quick Triage Questions:

Is management plane up? (Can you log into vCenter?)

Is control plane healthy? (NSX Manager cluster responding?)

Which hosts are affected? (All in cluster? One rack? One host?)

When did it start? (Correlate with change logs, maintenance windows, upgrades)

Is it intermittent or consistent? (Intermittent = harder to diagnose; may be time/load-dependent)

Log File Locations & Log Collection

VCF logs are distributed across the appliances. Know where each component writes before an incident.

SDDC Manager appliance (its own VM; SSH as vcf, then su or sudo):

# Lifecycle (upgrades, bundles) — VCF 9.0.1 docs, Accessing the VCF Upgrade Log Files
/var/log/vmware/vcf/lcm/lcm-debug.log     (debug level; once SDDC Manager is on 9.0 all LCM logging goes here)
/var/log/vmware/vcf/lcm/lcm.log           (info level; no longer updated after the 9.0 upgrade)
# Domain and cluster workflows (create domain, add cluster, commission hosts)
/var/log/vmware/vcf/domainmanager/domainmanager.log
# SoS output
/var/log/vmware/vcf/sddc-support/sos-<timestamp>/
# Bundle store (depot downloads)
/nfs/vmware/vcf/nfs-mount/bundle/

VCF Installer appliance (9.0 deployment and convergence; SSH as admin, then su):

/var/log/vmware/vcf/domainmanager/domainmanager.log   (validation and bring-up failures)

VCF Operations fleet management appliance:

/var/log/vrlcm/vmware_vrlcm.log   (VCF Operations and VCF Automation deployment and lifecycle issues)

vCenter appliance (SSH as root, shell):

/var/log/vmware/vpxd/vpxd.log                 (vCenter daemon: tasks, inventory, HA/DRS decisions)
/var/log/vmware/sso/                          (STS and identity services)
/var/log/vmware/vsphere-ui/logs/              (vSphere Client)
/var/log/vmware/vsan-health/                  (vSAN health service)
# Support bundle: vSphere Client > Menu > Administration > ... or https://<vcenter>:443/appliance/support-bundle; CLI: vc-support

ESX host (SSH or DCUI shell; /var/log is a symlink into /scratch/log):

/var/log/vmkernel.log      (VMkernel: storage, networking, memory; the first place for PSOD and APD/PDL analysis)
/var/log/vmkwarning.log    (VMkernel warnings only)
/var/log/vobd.log          (VMkernel Observations, VOBs: the event feed behind vCenter alarms)
/var/log/hostd.log         (host agent: VM operations, API calls)
/var/log/vpxa.log          (vCenter agent on the host)
/var/log/fdm.log           (vSphere HA agent)
/var/log/clomd.log         (vSAN CLOM: object placement and rebuild decisions)
/var/log/vsanmgmt.log      (vSAN management daemon, vsand)
# Bundle: vm-support (writes to /var/tmp by default) or the vSphere Client export

NSX Manager node (SSH as admin: NSX CLI; or root shell):

/var/log/syslog                   (system log; NSX CLI: get log-file syslog)
/var/log/proton/nsxapi.log        (management plane API)
/var/log/cloudnet/nsx-ccp.log     (central control plane, runs inside each Manager node)
# Bundle: System > Support Bundle in the NSX UI, or NSX CLI get support-bundle

NSX Edge node: /var/log/syslog and the NSX CLI (get logical-routers, get bgp neighbor summary inside a vrf).
ESX NSX data plane: VMkernel log entries from the nsx modules; nsxcli -c get controllers for CCP connectivity.

Supportability and Serviceability (SoS) Utility

SoS runs on the SDDC Manager appliance, not on ESX hosts. SSH to SDDC Manager as vcf, then:

sudo /opt/vmware/sddc-support/sos --help
sudo /opt/vmware/sddc-support/sos --health-check
sudo /opt/vmware/sddc-support/sos --get-vcf-summary

During a VCF 9.0 deployment SoS also exists on the VCF Installer appliance (/opt/vmware/sddc-support/sos as root) for health checks of a failing bring-up; after deployment run it only on SDDC Manager. SoS gathers ESX vm-support bundles, vCenter support bundles, NSX Manager and Edge tech-support bundles and SDDC Manager logs in one run, scoped with --domain-name <name>|ALL (run sos --help on the appliance for further scope options and the output location). Detail in Log Collection Deep Dive below.

Log Bundles from vCenter and ESX without SoS:

# vCenter: vSphere Client > Menu > Administration > Support > Export System Logs (hosts + vCenter), or on the appliance shell:
vc-support
# ESX host:
vm-support
# Both write .tgz bundles that Log Assist in VCF Operations can upload to a Broadcom case (objective 5.9).

VMware Support Engagement Process

When to escalate to VMware and how to prepare for support call.

Critical issues:

Production outage, data corruption, security breach → open a Severity 1 case immediately on the Broadcom Support portal (support.broadcom.com — the portal the VCF 9.0 docs point to, p.1267) and phone it in; Broadcom defines Severity 1–4 (KB 140745), which the case portal labels Critical - P1, High - P2, Medium - P3 and Low - P4.

High severity:

Feature not working, performance degraded, recurring errors → open a Severity 2 or 3 case, provide initial diagnostics.

Information needed for support case:

Exact error messages and timestamps (converted to UTC, with the time zone of the operator), the SoS bundle or the component bundles from the affected systems, the list of recent changes with dates and approvals, the business impact and the number of affected VMs or users, and the steps already tried. Severity is set by impact: a production outage or data loss is Severity 1, a degraded production service Severity 2, non-production or a question Severity 3 or 4. Cases are opened on the Broadcom Support Portal (support.broadcom.com); Log Assist in VCF Operations can attach bundles to the case directly.

Key Takeaways

  • Support Readiness: Before calling support, have ready: (1) exact error, (2) SoS bundles from all affected hosts, (3) vCenter logs (last 24 hours), (4) list of recent changes (dates, names, approvals), (5) network diagram showing which subnets are affected.

vSphere Troubleshooting#

ESXi Host Issues

Purple Screen of Death (PSOD):

ESXi kernel panic. Host stops executing VMs and displays purple error code.

Common causes:

Hardware failure (CPU, memory), firmware bug (BIOS update available), driver incompatibility (NIC, HBA), VM code triggering hypervisor bug, memory corruption.

Diagnosis:

Photograph the purple screen and note the VMkernel build, the panic string (for example '#PF Exception 14 in world ...', '#GP Exception 13', 'LINT1/NMI (motherboard nonmaskable interrupt)', 'PCPU n didn't have a heartbeat') and the first module frames of the backtrace; these, not a numeric code table, identify the fault. After reboot confirm the dump was written (esxcli system coredump file list; esxcli system coredump partition get) and collect vm-support for the case.

Recovery:

Reboot host (VMware HA restarts VMs on other hosts). Isolate host from cluster, run diagnostics (hardware test, firmware update). If PSODs recur, RMA hardware.

Prevention:

Keep ESX, firmware and drivers at the levels in the VCF BOM and the Broadcom Compatibility Guide (vSphere Lifecycle Manager images carry the vendor add-on and firmware); search the panic string and top frames in the Broadcom knowledge base before opening a case.

Network Connectivity Loss:

Host cannot reach vCenter, storage, or other hosts.

# Diagnose network on ESXi:
esxcli network ip interface list
# Shows vMotion, management, vSAN port groups and IPs

esxcli network ip route ipv4 list
# Verify routes (should have default gateway)

vmkping -I vmk0 <gateway-ip>        # management; -I vmk1 for vMotion, -I vmk2 for vSAN
vmkping -I vmk2 -d -s 8972 <peer-vsan-ip>   # MTU 9000 path check (no fragmentation)
# Test connectivity per VMkernel interface

esxcli network nic list
# List physical NICs; link status, speed, duplex, driver

esxcli network nic get -n vmnic0
# Driver and firmware versions, auto-negotiation, link detected

esxcli network nic stats get -n vmnic0
# Receive/transmit errors and drops on the uplink
esxcli network vm port list -w <world-id>
# Ports of a VM (world id from esxcli network vm list): team uplink, port id for pktcap-uw

# Check vSwitch/DVS configuration:
esxcli network vswitch standard list
esxcli network vswitch dvs vmware list
# Distributed switches as the host sees them: uplinks, port ids, MTU

Storage Disconnects:

LUNs disappear or go offline intermittently.

Symptoms:

Datastore goes "Inaccessible" or "Unmounted" in vCenter. VMs freeze. Error: "No accessible replicas".

Causes:

SAN fabric loop, switch port flapping, HBA firmware mismatch, SCSI reservation conflict, storage array failover.

Diagnosis:

Check the VMkernel log for SCSI, NMP, APD and PDL events:

grep -i -E 'scsi|nmp|apd|pdl' /var/log/vmkernel.log

. Check SAN switch logs for port resets. Verify HBA firmware is up-to-date and matches SAN vendor compatibility list.

Recovery:

Rescan storage:

esxcli storage core adapter rescan --adapter=vmhba0

(or all adapters: esxcli storage core adapter rescan --all). Check paths and the multipathing policy: esxcli storage nmp device list; esxcli storage core path list. If the LUN is still missing, verify zoning and masking on the fabric and array; an APD that times out (default 140 s) needs the paths restored before the datastore returns.

vCenter Server Issues

vCenter Services Not Starting:
Web UI unreachable, cannot connect to vCenter.

# Check service status on the vCenter appliance (SSH as root, then 'shell'):
service-control --status --all
service-control --status vmware-vpxd vmware-stsd vmware-vpostgres

# View recent errors:
tail -n 200 /var/log/vmware/vpxd/vpxd.log
tail -n 100 /var/log/vmware/vmon/vmon.log      # the service manager that starts the others

# Database: the appliance uses only its embedded PostgreSQL (VCDB); no external DB since vCenter 7
service-control --status vmware-vpostgres

# Restart services (vmon resolves the dependency order):
service-control --stop --all
service-control --start --all
# or a single service: service-control --restart vmware-vpxd

# If still failing, check the appliance partitions:
df -h /storage/db /storage/seat /storage/log /storage/core
# A full /storage/seat or /storage/db stops vpxd; grow the disk from the VAMI (https://<vcenter>:5480) or the autogrow script

Database Issues:

vCenter cannot connect to its database, or database is slow/corrupt.

Embedded PostgreSQL (the only option):

vCenter 8.x and 9.0 run the VCDB on the appliance. Core inventory lives in /storage/db, Stats, Events, Alarms and Tasks (SEAT) in /storage/seat; watch both with df -h and the VAMI Monitor > Disks view.

Retention, not vacuuming:

Growth of SEAT is controlled by the task and event retention settings: vSphere Client > vCenter > Configure > Settings > General > Database (statistics levels, task and event retention days). Never edit VCDB rows by hand.

Full partition:

Grow /storage/seat or /storage/db from the VAMI (or enlarge the matching VMDK and run the appliance autogrow script), then start the services.

Backup regularly:

File-based backup from the VAMI (Backup > Configure, FTPS/SFTP/NFS/SMB target); the backup contains the database, so no separate DB dump is needed. VCF schedules it from VCF Operations.

SSO (Single Sign-On) Problems:

Cannot log into vCenter; "Authentication Failed".

Causes:

SSO service down, local user database corrupted, LDAP/AD connection broken, certificate expired.

Check SSO:

service-control --status vmware-stsd vmware-vmdird vmware-vmafdd

. Restart with service-control --restart vmware-stsd (allow a few minutes).

Certificate check:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A2 Validity

(also --store TRUSTED_ROOTS; and the STS signing certificate via checksts.py from the Broadcom KB). In VCF, certificates are renewed from VCF Operations > Fleet Management > Certificates, not by hand on the appliance.

LDAP test:

If using AD, test connectivity to DC:

ldapsearch -H ldap://ad-server:389 -D 'cn=user,dc=corp,dc=local' -w password

. If fails, verify AD reachability and credentials.

Local admin fallback:

If the identity source is down you can still log in with administrator@vsphere.local; reset its password from the appliance shell with /usr/lib/vmware-vmdir/bin/vdcadmintool (option 3) if it is lost. In VCF 9.0 with VCF SSO, the Identity Broker is the identity source: check it in VCF Operations > Fleet Management > Identity & Access.

VM Issues

VM Cannot Power On:

VM stays in "Powering On" or fails with error.

# Check vCenter task history:

Recent Tasks panel → look for VM power-on task with error message

# Common errors and causes:

"No compatible hosts found" → Admission control failed (HA insufficient capacity)

  Fix: Check HA admission control settings; may need to power off non-critical VM

"Cannot open disk" → VMDK is locked or storage is inaccessible

  Fix: Check storage connectivity; if lock persists, restart hostd:
    /etc/init.d/hostd restart

"Insufficient memory" → Host RAM full

  Fix: Power off non-critical VMs; check VM ballooning in esxtop

"VM config file missing" → VM registered but datastore moved/renamed

  Fix: Re-register VM from datastore (Datacenter → Storage → browse datastore → .vmx file)

Snapshot Consolidation Failure:

VM has delta disk snapshots that won't consolidate/delete.

Causes:

A file lock held by a backup proxy (hot-add disk still attached), a full datastore (consolidation needs space for the merge), very high write rate on the VM, or an orphaned delta chain after a failed backup.

Diagnosis:

vSphere Client > VM > Snapshots shows 'Virtual machine disks consolidation is needed'; vmware.log in the VM folder names the locked file; vmkfstools -D <disk-flat.vmdk> on the host shows the owning host MAC of the lock.

Fix:

Release the lock (detach the disk from the backup proxy, restart hostd on the owning host if the lock is stale), free datastore space, then Snapshots > Consolidate; if the chain is damaged, clone the VM to a new disk chain rather than editing descriptor files by hand.

Key Takeaways

  • Purple Screen of Death (PSOD) reading: the panic string, VMkernel build and top backtrace frames identify the fault (there is no fixed BugCheck code table); confirm the core dump with esxcli system coredump file list, collect vm-support and search the panic string in the Broadcom KB before escalating.
  • Storage path failover: vSAN resyncs automatically; monitor resync rate and adjust throttle if impacting production I/O.
  • VM power-on failures: admission control, memory balloon, and snapshot consolidation are top 3 causes; triage in order.

vSAN Troubleshooting#

vSAN Health Check & Diagnostics

vSAN health check performs deep diagnostics on cluster, storage, network, and disk health. Interpret failures and remediate systematically.

Running vSAN Health Check:

# vCenter → Cluster → Monitor → vSAN → Health
# Click "Run Health Check Now"

# CLI method (on any ESXi host in vSAN cluster):
esxcli vsan health cluster list
# Shows: Storage health, network health, limit health, host health

esxcli vsan debug object health summary get
# Object health counts; details: esxcli vsan debug object list -u <object-uuid>

esxcli vsan cluster get
# Membership as this host sees it: local node state and role, sub-cluster member count (a count lower than the cluster size = partition)

Health Check Categories & Failures:

Storage health:

Issues with vSAN devices or disk groups (failed or unhealthy disks, controller problems; vSAN protects data through storage policies, not controller RAID sets). Fix: Replace failed disk, rebuild vSAN data.

Network health:

Latency between hosts >5ms (stretched cluster), lost packets (MTU mismatch). Fix: Adjust network config, enable MTU 9000 on switches.

Limit health:

Cluster resource limits hit (max VMs, max objects, dedup memory). Fix: Reduce object count, upgrade hardware, enable compression.

Host health:

ESXi host unable to contribute storage (disk errors, not claimed). Fix: Re-claim disks, or replace host.

Disk Group & Capacity Issues

Disk Group Failures:

One or more disks in a disk group are failed or degraded.

# List disk groups on host:
esxcli vsan storage list
# Shows each claimed device, its VSAN Disk Group UUID (OSA) and Is Capacity Tier (false = cache device)

# If a capacity device has failed (OSA):
esxcli vsan storage remove -d naa.id
# Removes the device from its disk group (data on it is rebuilt elsewhere); -s <ssd> removes a whole disk group

# Replace the physical device, then add it back to the disk group:
esxcli vsan storage add -d naa.id -s <cache-device-naa>
# ESA has no disk groups: esxcli vsan storagepool list / add / remove

# Monitor rebuild progress:
esxcli vsan debug resync summary get
# Bytes left to resync; in the vSphere Client: Monitor > vSAN > Resyncing Objects (ETA)

Capacity Warnings:

vSAN cluster is 80%+ full.

Symptom:

Health check warns "Capacity limit reached", new objects cannot be placed.

Root cause:

Insufficient capacity (add hosts/disks), dedup/compression not enabled (data is not compressing as expected), orphaned objects (snapshot leftovers, failed VMs).

Remediation:

(1) Identify large VMs/snapshots; delete unnecessary snapshots, (2) Add new disk group to host or new host to cluster, (3) On OSA, enable compression or deduplication and compression on the cluster (on ESA, compression is on by default), (4) Analyze dedup ratio to confirm effectiveness.

vSAN Resync & Object Health

Resync Operations:

After disk/host failure, vSAN automatically resyncs data to maintain FTT.

# Monitor active resyncs:
esxcli vsan debug resync summary get
esxcli vsan debug resync list
# Bytes and objects left; the vSphere Client (Monitor > vSAN > Resyncing Objects) adds the ETA and the reason (repair, policy change, evacuation, rebalance)

# During a large resync vSAN adaptive resync throttles rebuild traffic when VM I/O is contended;
# the resync throttle is no longer in the vSphere Client (KB 326830); for resync latency that stalls a VM the docs point to that KB

esxcli vsan cluster get
# Membership (master, backup, agents), sub-cluster UUID, health state; the number of partitions is visible here

Object Health States:

vSAN reports objects in various states (healthy, degraded, inaccessible).

Component states (VCF 9.0 documentation, Component State):

Absent:

Temporary failure: a host rebooting or down, a lost vSAN network, an unplugged device. The component and its data may return, so vSAN waits for the object repair timer (default 60 minutes) before rebuilding it elsewhere.

Degraded:

Permanent failure: a failed cache or capacity device or controller. vSAN starts rebuilding the degraded components immediately if resources allow.

Out of date:

The assigned storage policy is outdated; Reapply VM Storage Policy brings the objects back to compliant.

Object states:

Healthy while at least one full RAID-1 mirror (or the minimum number of RAID-5/6 data segments) is available; Unhealthy (inaccessible) when no full mirror is available or 50 percent or fewer of the object's votes are reachable. An inaccessible object stops the VM's I/O: bring the hosts or devices back, or restore from backup.

Network Partition Handling

Network Partition (Split-Brain):

Cluster splits into non-communicating partitions, each with its own master. An object stays accessible only in the partition holding more than 50 percent of its votes; in the other partitions it is inaccessible and its VMs stall (vSphere HA may restart them where the object is accessible).

Symptom:

VMs on minority partition freeze or are stopped. Health check shows high latency, packet loss to other hosts.

Diagnosis:

Check network connectivity between hosts:

ping each-host

. Check membership per host with esxcli vsan cluster get (hosts in different sub-clusters are partitioned) and the vSAN network with esxcli vsan network list and vmkping -I <vsan-vmk> -d -s 8972 <peer>.

Recovery:

Fix network issue (reconnect switch, repair cable). vSAN automatically remerges partitions (quorum retaken by full cluster). Monitor resync during merge.

Data consistency:

vSAN resynchronises the stale components from the partition that kept the objects accessible; after a long partition verify the object health in Monitor > vSAN > Virtual Objects and run Skyline Health before trusting the cluster for further changes.

Witness Node in Stretched Cluster

Witness Replacement:

Witness node is down or replaced; cluster loses tiebreaker.

Symptom:

Health check fails; stretched cluster cannot fail over sites.

Fix witness placement:

Witness must be at third location (not co-located with either primary site). Re-configure the witness in the vSphere Client: Cluster > Configure > vSAN > Fault Domains > Change.

Replace witness host:

Deploy a new vSAN witness appliance (or host) at the third site, add it to vCenter outside the stretched cluster, then Cluster > Configure > vSAN > Fault Domains > Change witness host; vSAN recreates the witness components. Objects stay accessible meanwhile (noncompliant only).

vSAN Performance Troubleshooting

HCIBench (Hyper-Converged Infrastructure Benchmark):

Performance testing tool for vSAN.

# Download HCIBench from VMware site
# Deploy on vSAN cluster to measure:
# - Sequential read/write IOPS
# - Random read/write IOPS
# - Latency (avg, p50, p99, p100)

# Baseline: measure with HCIBench on your own hardware; there is no universal IOPS figure.
# vSAN ESA is single-tier (NVMe storage pool, every device contributes capacity, no cache/capacity split);
# vSAN OSA has a cache tier and a capacity tier, so only OSA has cache-hit vs capacity-tier latency.

# If performance is below baseline, compare with the vSAN Performance Service (Monitor > vSAN > Performance: cluster, host, disk group and VM latency, IOPS, congestion, outstanding I/O), check Skyline Health for network and device findings, confirm the vSAN network (MTU, uplink speed, drops) and look for resync or rebalance traffic competing with the workload before changing hardware.

Key Takeaways

  • vSAN health check must pass all categories (cluster, storage, network, limits) before production deployment changes.
  • Component states: Absent (temporary, repair timer 60 minutes before rebuild), Degraded (permanent, rebuild immediately), Out of date (reapply policy); an object is unhealthy when no full mirror remains or 50 percent or fewer of its votes are reachable. Resync queue and adaptive resync determine recovery time.
  • Dedup/compression performance: monitor compression ratio; ratios <1.5:1 indicate dedup not working as expected, verify configuration.

NSX Troubleshooting#

Control Plane vs Data Plane Failures

Management and control plane (the NSX Manager cluster):

NSX Manager nodes host the management plane (API, UI, policy) and the central control plane (CCP, which computes and pushes forwarding state to the hosts and edges). Since NSX-T 2.4 there are no separate controller appliances. If the cluster is down you cannot change configuration and new VMs may not receive forwarding state, but existing traffic keeps flowing on the data plane.

Failure symptom:

Cannot log into NSX Manager, cannot create segments or rules; hosts show "Controller connectivity: down" in System > Fabric > Hosts.

Check NSX Manager:

NSX CLI on any Manager (SSH as admin): get cluster status (every service group STABLE, all three nodes UP); get service http (reverse proxy; restart service http if down); UI: System > Appliances. CCP log: /var/log/cloudnet/nsx-ccp.log. A three-node cluster keeps quorum with two nodes; with one node left it goes read-only.

Fix:

Restore or redeploy the failed node from System > Appliances (one node at a time; never reboot all three together). Hosts re-sync with the CCP automatically once it is back; check nsxcli -c get controllers on a host (controller connected, session up).

Data plane (NSX modules on ESX, Edge nodes):

Forwards packets. If a host's NSX modules or an Edge fail, traffic on that host or through that gateway is dropped.

Failure symptom:

VMs on one host cannot reach VMs elsewhere (web cannot reach app), NSX Manager is healthy.

Check data plane:

On the host: esxcli software vib list | grep -i nsx (modules installed), nsxcli -c get controllers, nsxcli -c get logical-switches (segments realised on this host). In the UI: System > Fabric > Hosts shows the transport node state (Success/Failed) and the NSX version; a failed host preparation is resolved with Resolve on that host.

Fix:

Resolve host preparation; if the host is partially prepared, put it in maintenance mode and re-run; a host reboot clears a wedged data plane (HA or DRS moves the VMs). Edge: System > Fabric > Nodes > Edge Transport Nodes shows the node and tunnel status; an Edge VM is redeployed from the same screen.

Tunnel Endpoint (TEP) Connectivity

TEPs are the host and Edge VMkernel/uplink addresses that carry Geneve overlay traffic (UDP 6081). TEP failure means overlay traffic between those endpoints drops; the physical network must carry the TEP VLAN end to end with MTU 1600 or more (9000 recommended).

# Host TEP interfaces (vmk10/vmk11 by default) and their IP pool or DHCP addresses:
esxcli network ip interface ipv4 get
esxcli network ip interface list | grep -A5 vmk10

# Overlay path check from host to host at Geneve MTU (1572 payload for a 1600 MTU; use -s 8972 for 9000):
vmkping ++netstack=vxlan -I vmk10 -d -s 1572 <remote-tep-ip>

# Tunnel state as NSX sees it:
nsxcli -c get vteps            # this host's TEPs
nsxdp-cli bfd sessions list    # BFD to peer TEPs on ESX (state up); on an Edge: get bfd-sessions
# UI: System > Fabric > Hosts > <host> > Tunnels, or Plan & Troubleshoot > Traffic Analysis > Traceflow

# Rebuilding TEPs: System > Fabric > Hosts > Resolve (re-applies the transport node profile and the uplink profile with its TEP VLAN and IP pool)

DFW Troubleshooting

Traceflow:

Plan & Troubleshoot > Traffic Analysis > Traceflow (or PUT /policy/api/v1/infra/traceflows/<traceflow-id>). Inject a synthetic packet from a VM port and read the observations: Forwarded, Dropped (with the DFW rule id that dropped it), Received, Delivered.

DFW rule debugging:

Rule not matching: rules evaluate top to bottom within the policy categories (Ethernet, Emergency, Infrastructure, Environment, Application); first match wins; check Applied To, the group membership (Inventory > Groups > View Members) and the service definition.

Traffic still passing when the rule says Drop: a higher rule allows it, or the default rule is still Allow (it is Allow after host preparation; set it to Drop once the allow rules are verified).

Per-VM view: nsxcli -c get firewall <vif-id> ruleset rules on the host shows the rules actually realised on that port; summarize-dvfilter | grep <vm-name> gives the filter name.

Packet capture:

# On the ESX host (not esxcli): list the VM's switch port, then capture at the port with pktcap-uw
net-stats -l                                        # port ids and VM names
pktcap-uw --switchport <port-id> --dir 2 -o /tmp/vm.pcap
pktcap-uw --uplink vmnic0 --capture UplinkRcvKernel --ng -o /tmp/uplink.pcap   # uplink receive (VCF 9.0.1 docs)
# Edge: NSX CLI 'start capture interface <fp-ethX> file edge.pcap'; copy out with scp. Open in Wireshark.

Edge Node Issues

Edge node down:

Tier-0 or Tier-1 service router on that Edge is unavailable; the standby Edge of the cluster takes over (active-standby Tier-0/Tier-1), or the remaining active Edge carries the ECMP share (active-active Tier-0).

Check Edge status:

System > Fabric > Nodes > Edge Transport Nodes (configuration state, node status, tunnels); NSX CLI on the Edge: get logical-routers; get high-availability status; get tunnel-ports.

Recover:

Power the Edge VM back on from vCenter or redeploy it from System > Fabric > Nodes (Edge VM redeploy keeps the configuration). Failover is automatic; after recovery check that the preemptive setting matches the design.

BGP & Routing Failures

BGP neighbour down:

The Tier-0 cannot establish BGP with the top-of-rack routers.

# On the Edge node that hosts the Tier-0 SR (NSX CLI, admin):
get logical-routers                       # find the SERVICE_ROUTER_TIER0 and its VRF number
vrf <n>
get bgp neighbor summary                  # state: Established is good; Idle/Active/Connect means no session
get bgp neighbor <ip> advertised-routes
get route                                 # learned routes in this VRF
get interfaces                            # uplink addresses and VLAN

# If the session is not Established, check in order:
  1. Uplink reachability: ping <neighbor-ip> source <uplink-ip> from the VRF (uplink VLAN, MTU, interface address present; VCF 9.0.1 release notes: missing interface addresses leave BGP Idle, rescan the interfaces)
  2. BGP parameters: local and remote AS, neighbour address, password, hold/keepalive timers, BFD settings on both sides (Networking > Tier-0 > BGP)
  3. Filters: route maps and prefix lists on the neighbour; redistribution (Tier-0 route redistribution must include the connected and Tier-1 prefixes you expect to advertise)
# Restarting the session: clear bgp neighbor <ip> on the Edge CLI, or toggle the neighbour in the Tier-0 BGP page

NSX Manager Cluster Health

Three-node cluster:

Required for production; VCF deploys three Manager nodes with a VIP (or an external load balancer). The cluster needs a majority: with two nodes it keeps working, with one it is read-only (no configuration changes, data plane unaffected).

Check cluster status:

get cluster status on any node (groups: DATASTORE, CLUSTER_BOOT_MANAGER, CONTROLLER, MANAGER, POLICY, HTTPS, MONITORING, IDPS_REPORTING, CORFU_NONCONFIG, SITE_MANAGER; all STABLE); get cluster config (VIP, nodes); get services. In VCF: VCF Operations > Fleet Management > Lifecycle and Inventory show the NSX cluster health as SDDC Manager reports it.

Node down:

Detach or redeploy the failed node from System > Appliances; never remove two nodes at once. Backups: NSX configuration backup to SFTP (System > Backup & Restore) is the recovery path if the cluster is lost, restored onto a new Manager of the same version with the same FQDN or VIP.

Key Takeaways

  • Control plane vs data plane: the Manager cluster (management plane plus CCP, no separate controllers) down means no configuration changes and no new forwarding state; a host or Edge data-plane fault drops traffic. Diagnose with get cluster status and nsxcli -c get controllers separately.
  • TEP connectivity: TEP VLAN end to end, MTU 1600 or more (9000 recommended); vmkping ++netstack=vxlan -I vmk10 -d -s 1572 validates the Geneve path; nsxdp-cli bfd sessions list on the host (get bfd-sessions on an Edge) shows the tunnel state.
  • DFW: Traceflow names the rule that dropped a packet; rules evaluate top to bottom through the categories, first match wins, and the default rule is Allow after host preparation until you change it; nsxcli get firewall <vif> ruleset rules shows what the host enforces.

VCF Lifecycle Troubleshooting#

SDDC Manager Troubleshoot — Lifecycle Failures & Recovery (VCF 9.0)

Lifecycle in VCF 9.0 is driven from VCF Operations (Fleet Management > Lifecycle) with SDDC Manager executing the VCF Instance workflows; the SDDC Manager UI is deprecated but its Updates tab still shows the detailed task view. Troubleshoot in this order: precheck results, the task's error and remediation text, then the LCM log.

VCF Upgrade Failure:

A component upgrade halts with status Failed.

Check status:

VCF Operations > Fleet Management > Lifecycle > VCF Instances > <instance> > Available Updates / In-Progress Updates > View Status (per-task green ticks; a failed task shows the reason and remediation steps), or SDDC Manager > Inventory > Workload Domains > <domain> > Updates and Update History (View Past Attempts).

Logs:

/var/log/vmware/vcf/lcm/lcm-debug.log on SDDC Manager (all LCM logging once SDDC Manager is on 9.0); component-side logs for the failing step (vCenter: /var/log/vmware/applmgmt/ and the RDU logs; NSX: upgrade coordinator in the NSX UI; ESX: vSphere Lifecycle Manager remediation task in vCenter).

Common failures:

Precheck warnings ignored (a failed precheck does not block the start but makes the update fail), depot or binary download incomplete, vCenter or NSX backup prerequisites missing, hosts that cannot enter maintenance mode (admission control, pinned VMs, vSAN data evacuation), expired certificates or passwords, time skew.

Recovery, not rollback:

There is no automatic rollback of the VCF upgrade as a whole. Fix the cause and Retry the failed component; the sequence resumes. The exception is vCenter: a reduced-downtime upgrade can be cancelled while it runs (the configuration reverts to the source vCenter) and rolls back automatically if it fails (VCF 9.0 docs, Upgrade to vCenter 9.0 for VMware Cloud Foundation).
Protection comes from the documented prerequisites taken before the upgrade: an SFTP backup of SDDC Manager, a snapshot of the SDDC Manager appliance, and recent backups of vCenter and NSX (VCF 9.0.1 docs, Upgrade SDDC Manager to 9.0). Reverting SDDC Manager to its snapshot or restoring NSX/vCenter from backup is done with Broadcom support, because the components must stay version-consistent (NSX can roll back only before Edge and host upgrades start; vCenter uses reduced downtime upgrade with its own switch-back; ESX and vSAN on-disk format do not roll back).

Binary (Bundle) Download Failures

VCF 9.0 downloads binaries from the Broadcom online depot (dl.broadcom.com) with a download token, or from an offline depot populated by the VCF Download Tool. download.vmware.com no longer exists (VCF 9.0.1 release notes).

# Where binaries land on SDDC Manager:
/nfs/vmware/vcf/nfs-mount/bundle/
df -h /nfs/vmware/vcf/nfs-mount

# If a download is slow or fails:
  1. Depot reachability from SDDC Manager: curl -I https://dl.broadcom.com (through the configured proxy if any); confirm the download token is valid and set in VCF Operations > Fleet Management > Lifecycle > Settings (Depot)
  2. DNS and NTP on SDDC Manager (DNS failure is the most common cause of a depot error)
  3. Free space on the nfs-mount; remove binaries you no longer need from the Lifecycle UI
  4. Dark sites: download with the VCF Download Tool on a connected machine, publish to an offline depot (HTTP/HTTPS) and point SDDC Manager/VCF Operations at it; VCF 5.2 used the Bundle Transfer Utility for the same purpose

Precheck Failures & Remediation

Update precheck (VCF Operations > Lifecycle > Precheck, or SDDC Manager Updates tab) validates the domain before you plan an upgrade. Results are green, warning or failed; resolve them before starting.

Common precheck findings:

Certificates expiring or expired:

Renew from VCF Operations > Fleet Management > Certificates (Renew Certificate per component; Show ESX Hosts to include hosts). Check quickly with sudo /opt/vmware/sddc-support/sos --certificate-health.

Passwords expired or unknown to SDDC Manager:

Remediate from VCF Operations > Fleet Management > Passwords (rotate or remediate); check the password state in VCF Operations (sos --help lists any SoS password check on your release).

DNS/NTP:

Forward and reverse DNS for every component, NTP in sync (sos --ntp-health; check DNS with forward and reverse lookups from the appliance).

Host cannot enter maintenance mode:

Free capacity (add a host, power off non-critical VMs), clear DRS rules and pinned VMs, ensure vSAN can evacuate (Ensure accessibility is the default for upgrades); do not disable HA to force it.

Storage and inventory:

Datastores accessible on every host, no hosts or clusters in error state, no failed tasks pending.

Version and license:

The target release must be a documented upgrade path (SDDC Manager 5.0 and later to 9.0); licensing in 9.0 uses a subscription license file imported into VCF Operations: license keys are no longer used, the license is assigned to the vCenter instance (which licenses its ESX hosts, NSX and the other components), and license usage must be submitted from VCF Operations every 180 days or hosts disconnect from vCenter and new workloads cannot be started (VCF 9.0 release notes, What's New; Broadcom KB 437242).

Certificate Management (Objective 5.4)

VCF 9.0 manages component certificates centrally in VCF Operations > Fleet Management > Certificates:

  1. Configure a certificate authority: VMCA (default), Microsoft CA (service account with Request Certificates and Issue and Manage Certificates on the CA, Read and Enroll on the template) or OpenSSL
  2. Generate CSRs per component, or import externally signed certificates
  3. Install or Renew Certificate per component; root certificates cannot be renewed from here
  4. Activate Auto-renewal per VCF Management or per VCF Instance/domain; automatic renewal runs 60 days before expiry (Set Up Automatic Renewal of Certificates)

Troubleshooting: sos --certificate-health for expiry per component; a renewal failure usually traces to CA connectivity or template permissions (Microsoft CA), a component password unknown to SDDC Manager, or a service left in a prior failed renewal (check the task in VCF Operations and the SDDC Manager operations manager log).

Configuration Drift

Configuration drift in VCF 9.0 means vCenter instances and vSphere clusters deviating from the desired-state configuration templates and configuration profiles managed in VCF Operations.

Detecting drift:

VCF Operations > Fleet Management > Configuration > Configuration Drifts (Configuration Drift dashboard): states Not Drifted, Drift Detected, Drift Check Failed (object unavailable or computation pending), Unavailable; drift detection is skipped when the template is not in the source control repository. Compare the drifted settings against the template.

Remediation:

Re-apply the template or the cluster configuration profile from VCF Operations (vSphere 9.0 vCenter and clusters with Configuration Profiles enabled); where the change was intended, update the template (and its source control branch) so the new value becomes the desired state. Prevention: change control through VCF Operations and the templates, not ad-hoc edits in the vSphere Client or NSX.

Key Takeaways

  • Precheck findings (certificates, passwords, DNS/NTP, hosts that cannot enter maintenance mode, licensing) are fixed before Plan Upgrade; a failed precheck does not block the start, it makes the update fail later.
  • Binary download: verify access to dl.broadcom.com (or the offline depot) and the download token, DNS/NTP on SDDC Manager, and free space on /nfs/vmware/vcf/nfs-mount; download.vmware.com no longer exists.
  • There is no automatic rollback of a VCF upgrade as a whole: retry after fixing the cause (exception: a vCenter reduced-downtime upgrade can be cancelled and rolls back automatically on failure); recovery relies on the pre-upgrade SFTP backup and snapshot of SDDC Manager and backups of vCenter and NSX. Certificates auto-renew 60 days before expiry from VCF Operations > Fleet Management > Certificates; drift is read on the Configuration Drifts dashboard.

Performance Troubleshooting#

esxtop: CPU Metrics Deep Dive

esxtop is the primary performance diagnostic tool on ESX. It samples CPU, memory, network and storage statistics in real time (vSphere docs, Performance Monitoring Utilities: resxtop and esxtop).

Running esxtop:

ssh root@esx-host
esxtop
# Interactive mode: press 'c' for CPU, 'm' for memory, 'n' for network, 'd' for disk
# Press '?' for help

# Batch capture for later analysis or a support case (Broadcom KB 308926):
esxtop -b -d 5 -n 60 > /tmp/esxtop.csv
# -b batch mode, -d delay between samples (s), -n number of samples; open the CSV in Windows perfmon or Excel

CPU Metrics Explained (CPU panel):

%USED  = physical CPU core cycles used by the VM or world: %USED = %RUN + %SYS - %OVRLP
         It does not include ready or co-stop time (100% = %RUN + %RDY + %CSTP + %WAIT)
%RDY   = time the VM was ready to run but had no physical CPU to run on
         Sustained high %RDY per vCPU = CPU contention (too many vCPUs scheduled on too few cores), unless %MLMTD is high: %MLMTD time is counted inside %RDY, so a VM held back by a CPU limit shows high %RDY with no contention
         Action: right-size vCPUs, rebalance with DRS, or lower the vCPU:pCPU overcommit
%CSTP  = time in a ready, co-deschedule state (wide VMs whose vCPUs drift apart)
         High %CSTP usually means the VM has more vCPUs than its workload uses
         Action: reduce the vCPU count (right-size)
%MLMTD = time the VMkernel deliberately did not run the VM because of its CPU limit (this time is included in %RDY)
         High %MLMTD = the configured CPU limit is too low; raise or remove the limit
%SYS   = time in the VMkernel on behalf of the VM (interrupts, I/O, system activity); part of %USED

Example diagnosis:

  VM with 8 vCPUs: %RDY 160 on the VM row (about 20% per vCPU), %CSTP 30, %USED 250
  Reading: the VM waits for cores (contention) and its vCPUs are co-descheduled (oversized)
  Fix: right-size the VM to the vCPUs it actually uses, then rebalance the cluster

Memory Metrics (memory panel):

MCTLSZ  = physical memory reclaimed from the VM by ballooning (balloon size, MB)
MCTLTGT = memory the host attempts to reclaim by ballooning (balloon target, MB)
          MCTLSZ growing toward MCTLTGT = host memory pressure; the balloon driver is reclaiming
SWCUR   = current swap usage of the VM (MB); it can be historical
SWR/s, SWW/s = rate of swap-in from disk / swap-out to disk (MB/s): non-zero = ACTIVE swapping
          Active swapping is a severe performance problem; host swap is far slower than RAM
ZIP/s, UNZIP/s = memory compressed / decompressed per second (MB/s); compression is cheaper than swapping but costs CPU
Action for memory pressure: migrate VMs (DRS), right-size VM memory from active-memory trends, add host memory; use reservations only for workloads that must not balloon or swap

Network & Storage Metrics

Network (press 'n'):

%DRPTX = percentage of transmit packets dropped
%DRPRX = percentage of receive packets dropped
         Check the uplink (TEAM-PNIC column) and the physical NIC: esxcli network nic stats get -n vmnicX for errors and drops, esxcli network nic get -n vmnicX for link speed and driver
         For vSAN and vMotion traffic, check MTU end to end and uplink saturation

Storage (press 'd' for adapters, 'u' for devices, 'v' for VM disks):

DAVG/cmd = average response time per command sent to the device (ms)
           High DAVG: the device, array or fabric is slow (array congestion, failing disk, path problems)
KAVG/cmd = average time a command spends in the VMkernel (ms); should be close to zero
           High KAVG: queueing in the host (queue depth limits, QAVG), check queue depths before the array
GAVG/cmd = DAVG + KAVG (Broadcom KB 344099): the latency the guest perceives
           Sustained values above about 10 ms warrant investigation (KB 344099)
           High GAVG with high DAVG = device or fabric; high GAVG with high KAVG = host queueing
ACTV / QUED = commands currently active / queued in the VMkernel; a persistent QUED shows the queue depth is the limit

Example diagnosis:

  DAVG 50 ms, KAVG 0.1 ms, GAVG 50.1 ms, QUED 0, array showing 95% utilisation
  Reading: latency comes from the device side, not the host queue
  Fix: move the workload to a less-congested datastore or fix the array; then re-measure

VCF Operations Metrics

VCF Operations (formerly VMware Aria Operations) provides centralised monitoring across the VCF fleet and is deployed by VCF Operations fleet management. Metrics cover CPU, memory, network, storage and VM latency. Anomaly detection and dynamic thresholds flag metrics outside their learned range; alerts combine symptoms into definitions with recommendations; the Observability Workbench (objective 5.9) correlates metrics, logs (VCF Operations for logs) and events for one object on one timeline. Use it to confirm an esxtop finding across the cluster and over time (was the latency spike a one-off or a daily pattern) before changing hardware or policy.

Key Takeaways

  • esxtop CPU metrics: %USED = %RUN + %SYS - %OVRLP and excludes ready and co-stop time; sustained high %RDY per vCPU indicates contention only after checking %MLMTD, because CPU-limit time is counted inside %RDY; high %CSTP (co-deschedule) usually means more vCPUs than the workload uses, so right-size. Drill into VM-level CPU contention.
  • Memory: a non-zero SWCUR shows pages were swapped at some point; active swapping is SWR/s and SWW/s now, with MCTLSZ growing toward MCTLTGT. Relieve pressure by migrating or right-sizing VMs rather than powering them off reflexively.
  • Storage latency: GAVG = DAVG + KAVG; sustained GAVG above about 10 ms warrants investigation (KB 344099); split the cause by whether DAVG (device/fabric) or KAVG (host queueing) carries it, and check outstanding I/O (QUED) and queue depth.

Support Engineering Fundamentals#

VMware Support Case Workflow & SLAs

Severity Definitions (Critical to Low):

SeverityDefinitionExampleInitial call service level objective (Broadcom Software and Maintenance Policy Handbook, quoted in Broadcom KB 140745; your contract governs)Workaround Available
Severity 1 (Critical)No workaround immediately available, and a production or mission-critical system is down or has had a substantial loss of service, or a substantial portion of mission-critical data is at significant risk of loss or corruptionEntire vSAN cluster offline; vCenter unreachable; all VMs lost network30 minutes (24x7 continuous support for eligible customers covers Severity 1 only)No (urgent escalation required)
Severity 2 (High)Major functionality severely impaired; operations continue in a restricted fashion, though long-term productivity may be affectedvSAN single host down; vMotion broken; DRS not working2 business hoursPartial (e.g., manual DRS workaround)
Severity 3 (Medium)Limited adverse effect on business operationsNSX segment slow to create; vCenter event log filling (cleanup available)4 business hoursYes (clear workaround)
Severity 4 (Low)Business operations not adversely affected (for example a cosmetic issue or a how-to question)"How do I configure DRS affinity rules?"; "Dashboard page slow to load"8 business hoursN/A (informational)

Support Tiers & Offerings

Broadcom's VMware Support Offerings page points to the Broadcom support policies (the Software and Maintenance Policy Handbook) and lists the former VMware Production Support and Select Support datasheets under Retired Support Offerings; the offering, coverage and response targets that apply are the ones in your contract.

Severity 1 coverage:

24x7 continuous support is available for eligible customers for Severity 1 problems only; eligible customers may request continuous efforts on a Severity 1 case.

Lower severities:

Logged 24x7 but handled within regional business hours, so a response may come on the next regional business day.

Entitlement check:

support.broadcom.com > My Entitlements shows what each site ID is entitled to.

Case Lifecycle & Engagement

Customer submits case → Broadcom Support Portal
↓

Tier 1 (Front-line triage):

  • Collect info (environment details, error logs, reproduce steps)
  • Run diagnostics (KB matching, basic troubleshooting)
  • If simple → resolve; if complex → escalate

↓ (if escalated)

Tier 2 (Senior engineer):

  • Deep troubleshooting (log analysis, performance profiling)
  • May request SoS bundle, GuestInfo dumps
  • Engage with product engineering if needed
  • Typical resolution: 2–5 days for Sev 2

↓ (if still unresolved)

Tier 3 / Engineering escalation:

  • Product engineering team (VMware internal)
  • Potential bug fix, workaround, or design review
  • May require patch release (if critical)
  • Timeline: 1–4 weeks for bug fix

Case resolution:

  • Solution documented, case closed
  • Customer feedback survey, NPS (Net Promoter Score)

Broadcom Support Portal

After the 2023 acquisition, VMware support, downloads and KBs moved to the Broadcom Support portal:

Broadcom portal:

https://support.broadcom.com — cases, entitlements, downloads (the VCF 9.0 docs send you here for the VCF Installer, p.1267); KBs at knowledge.broadcom.com.

Legacy VMware Customer Connect:

no longer the case-entry point; use the Broadcom portal with your Broadcom site ID.

Key points:

One portal for all Broadcom products; single sign-on; KB search at knowledge.broadcom.com; case collaboration and Log Assist uploads from VCF Operations land in the same case.

TSE (Technical Services Engineer) Engagement Pattern

For high-value customers or extended deployments:

Assigned TSE:

Dedicated engineer assigned for the engagement duration.

Proactive monitoring:

TSE monitors environment health, anticipates issues, offers optimization recommendations.

Scheduled check-ins:

Weekly/monthly syncs to review logs, alert trends, capacity planning.

Escalation path:

Direct phone/email access (vs. ticket queue); faster response.

Reading KB Articles for Build/Version Applicability

KB article structure (Broadcom/VMware):

Title:

Concise, includes key terms (e.g., "vCenter password change fails on VCF 9.0").

Applicable Products/Versions:

CRITICAL section at top. Example: "Applies to: vSAN 7.0 U1–9.0.1; does NOT apply to vSAN 6.x or 9.0.2+".

Symptoms:

Observable behavior; reproduction steps.

Root Cause:

Technical explanation (bug, config issue, limitation).

Resolution:

Fix (upgrade/patch), workaround, or configuration change.

Affected Builds:

Specific version ranges (e.g., "vCenter 6.7 GA through 6.7 U3d"; "Fixed in 7.0 U2").

Key Takeaways

  • Support Exam Insight: a question such as 'case SID-12345, customer Y, vSAN 9.0.1 and vCenter 9.0 GA — which KB articles apply?' tests whether you resolve the exact versions first, then cross-reference the KB and the release notes' resolved-issues list for those versions, and check the VCF BOM (KB 314608) before recommending a patch. Know how to find fixed-in versions rather than memorising KB numbers.

Log Collection Deep Dive (SoS Utility)#

SoS Utility: Options & Usage

The SoS (Supportability and Serviceability) utility runs on the SDDC Manager appliance (and, during a VCF 9.0 deployment, on the VCF Installer appliance; on Cloud Builder in 5.2). It does not run on ESX hosts. SSH to SDDC Manager as vcf; privileged (fix-it-up) operations need su and ./sos from /opt/vmware/sddc-support.

Basic syntax:

sudo /opt/vmware/sddc-support/sos --help
sudo /opt/vmware/sddc-support/sos --version

Scope option documented in the 9.0 docs: --domain-name <name> (example in the NSX Manager recovery procedure: sos --domain-name <domain_name> --nsx-logs). Run sos --help on your appliance for its full scope options and defaults.

Log collection options (VCF 9.0 docs, Troubleshooting > SoS Utility Log File Options, written for the VCF Installer appliance; run sos --help on SDDC Manager for its exact list):

  --api-logs             output of APIs
  --vcf-installer-logs   VCF Installer appliance support logs
  --esx-logs             ESX host logs, from each ESX host in the deployment
  --nsx-logs             NSX Manager logs
  --vc-logs              vCenter logs, from each vCenter server in the deployment
  --sddc-manager-logs    SDDC Manager logs
  --automation-logs      VCF Automation support logs
  --operations-logs      VCF Operations support logs
  --operations-fleet-logs  VCF Operations fleet management support logs
  --vm-screenshots       screenshots from all VMs
  --collect-all-logs     logs for all components
  --no-health-check      skip the health check that runs as part of log collection
  --no-clean-old-logs    keep the output of the previous run

Generic options (9.0 docs): --log-dir <dir> and --log-folder <name> set the output location; --setup-json <file> collects for a pre-defined set of components; --zip creates a zipped tar of the output; --short shows only failures and warnings in health results; --history shows the last twenty SoS operations; --force allows SoS while workflows are running (not normally used, but required for health checks on the VCF Installer appliance); --help / -h and --version / -v.
Further log-collection and health-check options that appear in other releases' docs (for example per-component log switches for WCP or HCX, and extra health checks such as storage, DNS, password and version) are not in the 9.0 option tables: confirm them with sos --help on the appliance before relying on them. The default output directory, the default collection set and the collection time-out are not stated in the 9.0 option tables either.

Health check options (9.0 docs, VCF Installer appliance; add --force there):

  --health-check         all available health checks
  --connectivity-health  ping reachability of ESX hosts, vCenter, NSX Manager and SDDC Manager VM
  --compute-health       compute health check
  --services-health      confirms services are running
  --ntp-health           time synchronised with the NTP server
  --general-health       ESX entries, ESX error dumps, NSX Manager and cluster status
  --certificate-health   component certificates valid (within expiry date)
  --get-host-ips         returns server information

Also in the 9.0 docs: --get-vcf-summary (view the details of the VCF system, used to verify SDDC Manager health after a restore) and --skip-cert-check (used with --health-check --domain-name ALL after cluster recovery).

Example commands (all taken from the 9.0 docs):

  sudo /opt/vmware/sddc-support/sos --get-vcf-summary
  sudo /opt/vmware/sddc-support/sos --domain-name <domain_name> --nsx-logs
  ./sos --health-check --force          (on the VCF Installer appliance)

Bundle Size and Upload

A full --collect-all-logs run on a management domain is typically hundreds of MB to a few GB (vm-support per host dominates). Keep bundles small by scoping with --domain-name and by collecting only the component in question. The 9.0 SoS option tables list no redaction option (searched; none surfaced); review before sharing. In VCF 9.0 the supported upload path to a Broadcom case is Log Assist in VCF Operations (objective 5.9): the link to Log Assist is in the Diagnostics Findings tab; it generates the bundle for the selected inventory object and transfers it with the Support case ID (VCF 9.0 docs, Use Log Assist to Accelerate Your Support Cases); alternatively upload the SoS folder through the Broadcom support portal.

Log Rotation & Forwarding

ESX: logs rotate by size (per-logger rotation size and count, esxcli system syslog config logger list); persistent location via Syslog.global.logDir (/scratch/log on a local datastore or a VMFS path); remote hosts with esxcli system syslog config set --loghost='tcp://<collector>:514' then esxcli system syslog reload. vCenter: /var/log/vmware/vpxd/vpxd.log rotates by size with ~10 generations; forward from the VAMI (Syslog). NSX: System > Fabric > Profiles > Node Profiles, or set logging-server on each node. VCF: VCF Operations for Logs is the fleet log platform (deployed by VCF Operations fleet management, or from the .ova with a VVF licence); configure the VCF components to forward syslog to it and use its content packs for vSAN, NSX and vCenter; an external SIEM receives a forward from VCF Operations for Logs or directly by syslog.

Log File Location Reference by Component

ComponentLog FileUse CaseKey Fields to Check
ESX VMkernel/var/log/vmkernel.logPSOD analysis, storage (SCSI, NMP, APD/PDL), memory, driver errorspanic string and backtrace, 'NMP: ...', 'APD', 'PDL', 'MCE'
ESX hostd/var/log/hostd.logVM lifecycle (power on/off, snapshots), API calls from vCenter'Error', 'Failed', 'Task Created'
ESX vpxa/var/log/vpxa.logvCenter agent: host-to-vCenter communication, heartbeats'Lost connection', 'Heartbeat'
ESX fdm/var/log/fdm.logvSphere HA: master election, host isolation, VM restarts'isolation', 'election', 'restart'
ESX vSAN/var/log/clomd.log, /var/log/vsanmgmt.log, /var/log/vobd.logObject placement and rebuild (CLOM), vSAN management daemon, VMkernel observations'Resync', 'component', 'degraded', VOB ids
vCenter vpxd/var/log/vmware/vpxd/vpxd.logInventory operations, tasks, permissions, HA/DRS decisions'Exception', 'Task failed'
NSX Manager/var/log/syslog, /var/log/proton/nsxapi.log, /var/log/cloudnet/nsx-ccp.logSystem log, management plane API, central control plane'ERROR', realization failures, CCP connectivity
SDDC Manager/var/log/vmware/vcf/lcm/lcm-debug.log, /var/log/vmware/vcf/domainmanager/domainmanager.logUpgrades and binaries; domain, cluster and host workflowstask ids, 'FAILED', validation messages
VCF Installer / fleet management/var/log/vmware/vcf/domainmanager/domainmanager.log (Installer), /var/log/vrlcm/vmware_vrlcm.log (fleet management appliance)Deployment and convergence; VCF Operations and Automation lifecyclevalidation failures, request ids

Symptom-to-Log-File Quick Reference

Symptom: "VMs won't restart after a host failure"

  Check: fdm.log on the HA master (admission control, restart attempts), hostd.log on the target host, vpxd.log
  Command: grep -i -E 'restart|admission|isolation' /var/log/fdm.log | tail -40

Symptom: "vSAN object missing component, slow resync"

  Check: clomd.log, vsanmgmt.log, esxcli vsan debug object health summary get, esxcli vsan debug resync summary get
  Command: esxcli vsan debug object list | grep -i -B3 -A3 'absent\|degraded'

Symptom: "Workload domain creation or host commissioning failed"

  Check: domainmanager.log on SDDC Manager (task id from the VCF Operations task), then the vCenter task that failed
  Command: grep -i -E 'error|failed' /var/log/vmware/vcf/domainmanager/domainmanager.log | tail -40

Symptom: "Upgrade stuck or failed"

  Check: lcm-debug.log on SDDC Manager, the component's own upgrade log (vCenter applmgmt, NSX upgrade coordinator, vLCM task)
  Command: grep -i -E 'exception|failed' /var/log/vmware/vcf/lcm/lcm-debug.log | tail -40

Key Takeaways

  • SoS runs on SDDC Manager (vcf user, sudo /opt/vmware/sddc-support/sos): scope with --domain-name; collect per component (--esx-logs, --vc-logs, --nsx-logs, --sddc-manager-logs, --api-logs) or --collect-all-logs; health with --health-check and the specific checks (certificate, connectivity, compute, general, ntp, services; run sos --help for further checks on your release).
  • Log file locations: /var/log/vmkernel.log, hostd.log, vpxa.log, fdm.log, clomd.log on ESX; /var/log/vmware/vpxd/vpxd.log on vCenter; /var/log/syslog and /var/log/proton/nsxapi.log on NSX Manager; /var/log/vmware/vcf/lcm/lcm-debug.log and domainmanager/domainmanager.log on SDDC Manager; grep strategically.
  • Bundle size: a full collection runs to GBs because of per-host vm-support; scope by domain and cluster, collect only the component in question, and upload through Log Assist in VCF Operations; SoS has no redaction option, so review before sharing.

ESXi & Hardware Troubleshooting#

PSOD (Purple Screen of Death) Reading

A PSOD is a VMkernel panic. The screen shows the ESX build, a panic string, the PCPU and world that faulted and a backtrace; there is no fixed table of numeric BugCheck codes to memorise. Read it in this order:

  1. VMkernel build (first lines) — confirm it is the BOM build; a mismatch is itself a finding
  2. Panic string — the fault class. Typical strings:
   #PF Exception 14 in world <id>:<name> IP 0x... addr 0x...   page fault: driver or kernel bug, memory corruption
   #GP Exception 13 ...                                           general protection fault: similar causes
   LINT1/NMI (motherboard nonmaskable interrupt), undiagnosed    hardware or firmware NMI (BMC watchdog, ECC, PCIe)
   PCPU n didn't have a heartbeat for s seconds; *may* be locked up   a CPU stuck in a spin or a hung driver
   Spin count exceeded / Lost Heartbeat                            lock or scheduling problem
   MCE (machine check exception) with bank/status values           hardware: CPU, memory, PCIe
  1. First module frames of the backtrace — which module was executing (a driver such as a NIC/HBA driver, vmklinux, a vSAN or NSX module, the scheduler)
  2. Whether the core dump was written ("Coredump to disk. Slot ..." or a netdump message)

Example (abridged):

  VMware ESX 9.0.0 [Releasebuild-24755229 x86_64]
  #PF Exception 14 in world 2099180:vmm0:web-01 IP 0x42000f1a2b3c addr 0x0
  PCPU 12: ...
  Backtrace: ... [0x4200...]nvmxnet3_rx_... ... [0x4200...]Net_Receive ...
  Coredump to disk. Slot 1 of 1 on device naa.6000...:7.

Interpretation: a page fault inside a NIC driver receive path on PCPU 12 while running web-01's monitor; the dump was written. Search the panic string and the top frame (driver name) in the Broadcom knowledge base and compare the driver/firmware pair with the Broadcom Compatibility Guide.

Action after reboot:

  1. esxcli system coredump file list (or esxcli system coredump partition get) — confirm the dump exists
  2. vm-support — collect the bundle (includes the dump when configured); photograph the screen for the case
  3. grep the VMkernel log before the panic for storage or network events: grep -i -E 'nmp|apd|pdl|link down|mce' /var/log/vmkernel.log
  4. Check hardware logs (iDRAC/iLO/BMC SEL) for ECC, PCIe and watchdog events around the timestamp
  5. If the frame is a driver, compare its version with the BOM image and the BCG; remediate through the vSphere Lifecycle Manager image, not by hand

Hardware Compatibility (Broadcom Compatibility Guide)

Every host must be on the Broadcom Compatibility Guide (compatibilityguide.broadcom.com) for the ESX release in the VCF BOM, including the I/O devices (driver and firmware pairs) and, for vSAN, the vSAN ReadyNode or the certified cache/capacity devices and controller (ESA: certified NVMe).

Validation on the host:

  esxcli hardware platform get                      → vendor, model, serial
  esxcli hardware pci list                          → PCI devices with vendor/device ids and the bound module
  esxcli network nic get -n vmnic0                  → driver name/version and firmware version of a NIC
  esxcli storage core adapter list                  → HBAs and their drivers
  esxcli software vib list | grep -i -E 'nic|hba|driver'   → installed driver VIB versions
  vSAN: Skyline Health > Hardware compatibility (controller, driver, firmware against the vSAN HCL); an SoS hardware compatibility check exists only where your release offers one (see sos --help on SDDC Manager)

Firmware & Driver Stack

BIOS/UEFI, BMC (iDRAC/iLO/XCC), NIC, HBA/RAID and NVMe firmware must match the driver versions certified for the ESX build. In VCF the vendor add-on and firmware (via the hardware support manager) are part of the vSphere Lifecycle Manager image; drift from the image is reported in the cluster's Updates > Image compliance view.

  BIOS version: esxcli hardware platform get (BIOS fields) or the BMC
  NIC firmware: esxcli network nic get -n vmnicX
  Storage controller: vendor tool (storcli/perccli) through the BMC or the vendor's hardware support manager

Risk pattern: a new ESX build with an old inbox-replaced driver, or new firmware with an old driver, shows up as PSODs in that driver, link flaps, or vSAN device errors.

Quick Boot & Coredump Configuration

Quick Boot restarts the hypervisor without the hardware POST during vSphere Lifecycle Manager remediation, cutting minutes from each host upgrade. It is enabled in the cluster's remediation settings (Updates > Hosts > Remediation Settings > Quick Boot) and applies only to hosts and drivers that vLCM reports as Quick Boot compatible; it is skipped automatically otherwise. It is not a general boot setting toggled with esxcli.

Coredump:

  esxcli system coredump file list            → VMFS core dump files configured on the host
  esxcli system coredump partition get        → active dump partition (the ESX installer creates one on the boot device; VCF 9.0.1 docs, Flash Boot Devices)
  esxcli system coredump file add / set -e true -p <file>   → create and activate a dump file on a VMFS datastore if none is active
  esxcli system coredump network set -v vmk0 -i <vcenter-ip> -o 6500; esxcli system coredump network set -e true   → netdump to vCenter's ESXi Dump Collector (data in /var/core/netdumps on vCenter)

Recommendation: an active local dump file or partition on every host, plus netdump as the second copy for hosts booting from small devices.

ESXCLI Deep Troubleshooting Commands

Maintenance mode (enter/exit):

Enter maintenance mode (from the host: VMs are NOT evacuated; use the vSphere Client or vCenter API so DRS migrates them first):

  esxcli system maintenanceMode set --enable true

Exit maintenance mode:

  esxcli system maintenanceMode set --enable false

Check status:

  esxcli system maintenanceMode get
  Output: Enabled/Disabled

Storage path analysis (SAN connectivity):

List devices with their multipathing policy:

  esxcli storage nmp device list
  → PSP (Round Robin, MRU, Fixed) and working paths per device; SATP rules: esxcli storage nmp satp rule list

Check path state:

  esxcli storage core path list
  → Each path with state active/standby/dead and the adapter/target

Rescan storage:

  esxcli storage core adapter rescan --adapter vmhba2
  → Re-discovers LUNs attached to RAID adapter

Monitor path failures:

  grep -i "path lost\|device offline" /var/log/vmkernel.log

Network checks (host side):

  esxcli network nic list                              → link state, speed, duplex per uplink
  esxcli network nic get -n vmnic0                     → driver, firmware, auto-negotiation
  esxcli network nic stats get -n vmnic0               → errors and drops
  esxcli network vswitch dvs vmware list               → distributed switch, uplinks and MTU on this host
  esxcli network ip interface list / ipv4 get          → VMkernel interfaces, their netstack and MTU
  vmkping -I vmk1 -d -s 8972 <peer-vmotion-ip>         → MTU 9000 path test for vMotion (vmk2 for vSAN; ++netstack=vxlan -I vmk10 for TEPs)
  esxcli network ip neighbor list                      → ARP table (duplicate IP symptoms)

Key Takeaways

  • Broadcom Compatibility Guide: server, controller, NIC and (for vSAN) device certification against the ESX build in the VCF BOM, with the driver/firmware pairs carried in the vSphere Lifecycle Manager image.
  • Coredump: keep an active dump file or partition on local storage (esxcli system coredump file list / partition get) and netdump to vCenter's ESXi Dump Collector as a second copy; without a dump a PSOD case rests on the photograph alone.
  • Firmware and drivers move together through the vLCM image (vendor add-on plus hardware support manager); Quick Boot is a vLCM remediation setting for compatible hosts, not a BIOS or esxcli toggle.

vCenter / VCSA Troubleshooting#

vCenter Appliance Filesystem Layout

The vCenter appliance is a Photon OS VM with one VMDK per data partition so each can be grown independently:

  /storage/core       core dumps of vCenter services
  /storage/log        service logs (/var/log/vmware)
  /storage/db         VCDB: core inventory, PostgreSQL data
  /storage/dblog      PostgreSQL write-ahead log
  /storage/seat       Stats, Events, Alarms and Tasks (SEAT) tables of the VCDB (VCF 9.0.1 docs call this SEAT data)
  /storage/netdump    ESXi Dump Collector output
  /storage/autodeploy, /storage/imagebuilder, /storage/updatemgr, /storage/lifecycle, /storage/archive

Monitoring disk:

  df -h on the appliance shell, or VAMI (https://<vcenter>:5480) > Monitor > Disks; vCenter raises the "disk exhaustion" alarms at 75 percent (yellow warning) and 85 percent (red critical) (Broadcom KB 313077).
  /storage/seat growing = statistics level or task/event retention too high; /storage/log = a looping service or a debug level left on.

Expand a partition (no downtime for the service):

  1. Increase the matching VMDK of the appliance VM in the vSphere Client (identify it from the VAMI Disks view or the appliance's lsblk)
  2. On the appliance shell run /usr/lib/applmgmt/support/scripts/autogrow.sh (or VAMI > Monitor > Disks > Resize)
  3. Verify with df -h

Never resize with resize2fs by hand and never edit VCDB tables to free space.

vCenter Service Architecture

Services run under the vMon service manager; use service-control, not systemctl, for the vCenter services:

Service (service-control name)FunctionLog
vpxd (vmware-vpxd)core vCenter server: inventory, tasks, events, HA/DRS/var/log/vmware/vpxd/vpxd.log
vmdird (vmware-vmdird)directory service (SSO users, groups, replication)/var/log/vmware/vmdird/
stsd (vmware-stsd)Security Token Service (SAML tokens)/var/log/vmware/sso/
vmafdd (vmware-vmafdd)authentication framework, VECS certificate store/var/log/vmware/vmafdd/
vsphere-uivSphere Client (https://<vcenter>/ui)/var/log/vmware/vsphere-ui/logs/
eam (vmware-eam)ESX Agent Manager (NSX and vSAN agents, Supervisor)/var/log/vmware/eam/
vapi-endpoint (vmware-vapi-endpoint)REST and vAPI endpoint/var/log/vmware/vapi/endpoint/
vpostgres (vmware-vpostgres)VCDB/var/log/vmware/vpostgres/
vsan-health (vmware-vsan-health)vSAN health and Skyline Health/var/log/vmware/vsan-health/

Service control:

  service-control --status --all
  service-control --restart vmware-vpxd
  service-control --stop --all ; service-control --start --all    (vmon resolves dependency order)
  vmon log: /var/log/vmware/vmon/vmon.log (why a service failed to start)

Certificates & SSO Troubleshooting

vmafd hosts VECS, the certificate store, and the connection to vmdir; VMCA (vmware-vmca) is the built-in CA. In VCF the lifecycle of these certificates is driven from VCF Operations > Fleet Management > Certificates (CA choice, CSR, import, renew, auto-renewal 60 days before expiry); the appliance tools are for diagnosis.

Check certificates:

  /usr/lib/vmware-vmafd/bin/vecs-cli store list
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A2 Validity
  /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | grep -E 'Alias|Not After'
  STS signing certificate: the checksts.py script from the Broadcom KB (an expired STS certificate blocks every login)
  Manual replacement outside VCF: /usr/lib/vmware-vmca/bin/certificate-manager (menu-driven); in VCF use VCF Operations so SDDC Manager's inventory stays consistent

Directory service (vmdir):

  service-control --status vmware-vmdird
  /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator   (replication partners and lag in Enhanced Linked Mode)
  /usr/lib/vmware-vmdir/bin/vdcadmintool                                                   (reset the administrator@vsphere.local password: option 3)
  Logs: /var/log/vmware/vmdird/

Identity sources:

  vCenter 9.0 in VCF uses VCF SSO through the VCF Identity Broker (an embedded service in the management-domain vCenter, or a dedicated appliance); legacy direct AD over LDAP(S) remains possible. Test reachability of the identity source (DNS, TCP 636/389), the service account password and the certificate chain of the LDAPS endpoint before suspecting SSO itself.

PostgreSQL (VCDB) Basics

The appliance uses only its embedded PostgreSQL. Connect read-only for diagnosis:

  /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB
  \l+                                           database size
  SELECT relname, pg_size_pretty(pg_total_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC LIMIT 10;

Do not update or delete rows. Growth of the event and task tables is controlled in the vSphere Client: vCenter > Configure > Settings > General > Database (task and event retention days, statistics levels); vCenter purges on that schedule. If /storage/seat is already full, grow the partition (above) and then lower retention.

Backup and Availability

Protection of vCenter in VCF (VCF 9.0.1 Table 670): vSphere HA restarts the vCenter VM on another host; file-based backup from the VAMI (Backup > Configure: schedule, FTPS/SFTP/NFS/SMB target; in VCF the schedule is managed from VCF Operations) restores a vCenter of the same build through the installer's Restore option; vCenter High Availability (active/passive/witness) and Fault Tolerance are not supported for a vCenter managed by VCF. Check the backup state in the VAMI and with VCF Operations' backup status before any upgrade.

Key Takeaways

  • vCenter appliance partitions: /storage/db (VCDB), /storage/seat (Stats, Events, Alarms, Tasks), /storage/log, /storage/core; grow a full partition from the VAMI or autogrow.sh after enlarging its VMDK, and control SEAT growth with the task/event retention and statistics settings, never by editing VCDB.
  • Certificates: VECS stores (vecs-cli) and the STS signing certificate decide whether anyone can log in; in VCF renew from VCF Operations > Fleet Management > Certificates (auto-renewal 60 days before expiry) and use certificate-manager only outside VCF.
  • Services run under vMon: service-control --status --all and vmon.log, not systemctl. Availability in VCF is vSphere HA plus file-based backup; vCenter HA and FT are not supported for VCF-managed vCenter (Table 670).

vSAN Troubleshooting at Scale#

Object-Level Analysis

vSAN objects:
Each VM home, virtual disk, swap and snapshot delta is a vSAN object made of components (RAID-1 mirrors or RAID-5/6 segments plus witness components) placed by the storage policy; an object is available while more than 50 percent of its votes are reachable.

Cluster and object commands (ESX shell; the same data is in vSphere Client > Monitor > vSAN):

  esxcli vsan cluster get                           → membership (master, backup, agents), sub-cluster UUID, health state
  esxcli vsan debug object health summary get       → objects per health state (healthy, reduced availability, inaccessible ...)
  esxcli vsan debug object list [-u <uuid>]         → components, their state (ACTIVE, ABSENT, DEGRADED), host and device per component
  esxcli vsan debug object overview                 → per-object policy and used space (ESA used-capacity figures may be wrong: VCF 9.0.1 known issue)
  esxcli vsan debug disk list / esxcli vsan debug disk summary get   → device health as vSAN sees it
  esxcli vsan storage list (OSA disk groups) / esxcli vsan storagepool list (ESA)

Interpreting: an object with one ABSENT component and a mirror ACTIVE is accessible and waiting on the 60-minute repair timer; DEGRADED components rebuild at once; an object with 50 percent or fewer votes reachable is inaccessible and the VM stalls.

CMMDS, CLOM and DOM:

  CMMDS (cluster membership and directory) holds the metadata every host agrees on; CLOM (cluster level object manager, clomd.log) decides placement and when to rebuild; DOM (distributed object manager) executes object I/O and resync; LSOM (log-structured object manager) owns the local devices. In day-to-day support you read their effects through the vSphere Client and the esxcli vsan debug commands rather than by driving the daemons directly; the Skyline Health findings and VCF Operations' vSAN dashboards (and Operations for Logs' vSAN content pack) surface their log lines.

Device State Diagnosis

Device states as reported by vSAN:

  Healthy (OK)          device operational; components ACTIVE
  Absent components     device or host temporarily unavailable; repair timer running (default 60 min, Monitor > vSAN > Health > Object repair timer)
  Degraded components   permanent failure detected (I/O errors, device dead); rebuild starts immediately
  Dying disk / high error count   Skyline Health: Physical disk health, or esxcli storage core device smart get -d <device>

Device checks:

  esxcli vsan storage list                                 → OSA: cache and capacity devices per disk group with state and encryption
  esxcli storage core device list | grep -A5 <naa>        → device identity, path state
  esxcli storage core device smart get -d <device>         → SMART health, reallocated sectors, media errors
  Replace: OSA remove the device from its disk group (esxcli vsan storage remove -d <naa>) or the whole group (-s <cache>) with the evacuation mode you can afford, swap, add back (esxcli vsan storage add -d <naa> -s <cache>); ESA: storagepool remove/add. Prefer the vSphere Client (Configure > vSAN > Disk Management) so the evacuation mode is explicit.

Resync Rate & Rebuild Estimation

What rebuilds:

When a device or host fails, every component that lived on it is re-created from its surviving mirror (RAID-1) or reconstructed from parity (RAID-5/6) on other hosts; the data to move is the used capacity of the failed device, not a fraction of it.

Estimate:

  data to rebuild = used capacity on the failed device
  time ≈ data / effective cluster resync throughput
  Example: 2.5 TB used on the failed device at an effective 500 MB/s ≈ 2,500,000 MB / 500 MB/s ≈ 5,000 s ≈ 1.4 h

Effective throughput depends on the number of hosts contributing, the network (10/25 GbE), device write performance and VM I/O contention.

Monitoring and control:

  vSphere Client > Monitor > vSAN > Resyncing Objects: bytes left, ETA, reason (repair, policy change, evacuation, rebalance); esxcli vsan debug resync summary get on a host
  Adaptive resync throttles rebuild traffic automatically when VM I/O is contended; there is no manual throttle in the vSphere Client any more (KB 326830 covers a throttle left configured from older releases)
  Rebalance: proactive rebalance is automatic (Configure > vSAN > Services > Advanced Options)
  The repair timer can be lowered for a planned device replacement (Repair Objects Immediately in Skyline Health) rather than waiting 60 minutes

Stretched Cluster Partition Scenarios & Recovery

Topology (VCF 9.0 docs): two data sites plus a witness host at a third site.

Stretched cluster 3+3+1:

  Site A (preferred): 3 hosts
  Site B (secondary): 3 hosts
  Witness host: third site, metadata (witness components) only
  Latency: data sites less than 5 ms RTT; witness to each data site less than 200 ms RTT (100 ms one way recommended)

Failure handling (VCF 9.0.1 Table 855, How vSAN Stretched Cluster Handles Failures):

  Inter-site link lost: the witness host and the preferred site continue to service storage operations and keep data available.
    vSphere HA restarts the VMs that were running in Site B on Site A. When the link returns, objects protected by a
    policy with a site tolerance attribute are resynchronised. There is no read-only or fenced mode.
  Secondary site lost or isolated: the witness and the preferred site continue; site-tolerant objects resync afterwards.
  Preferred site lost or isolated: the secondary site continues storage operations if it still reaches the witness host.
  Witness lost or isolated: objects become noncompliant but data remains available; running VMs are not affected.
    Restore or replace the witness (Configure > vSAN > Fault Domains > Change witness host).

Recovery procedure:

  1. Identify the partition: esxcli vsan cluster get on hosts of both sites shows the membership each side sees
  2. Confirm which side holds quorum (preferred site plus witness in the link-down case) and that its objects are accessible
  3. Restore the inter-site link (switch, fibre, routing); the isolated hosts rejoin automatically
  4. Monitor the resync: Monitor > vSAN > Resyncing Objects (or esxcli vsan debug resync list on a host)
     (hours for large site-tolerant objects)
  1. Validate: all objects compliant and healthy in Monitor > vSAN > Virtual Objects and Skyline Health

vSAN Performance Service

The vSAN performance service collects and analyses performance statistics and shows them as charts for the cluster, each host, disk group or device and VM (vSAN docs, About the vSAN Performance Service). It is not vSAN Observer, which was a separate legacy tool.

Access in the vSphere Client:

  Cluster > Monitor > vSAN > Performance

Real-time metrics:

  - Cluster IOPS, throughput (MB/s), latency
  - Per-disk IOPS, outstanding I/Os
  - Resync queue depth, rebuild rate
  - Component state summary

Interpretation:

  IOPS spike → application load increase or rebuild in progress
  Latency spike >50ms → storage congestion (add disks) or network latency
  Queue depth >20 → potential I/O bottleneck (rebuild too slow)

Historical data:

  Performance Service retains 7 days (default); can be configured to 30 days
  Useful for capacity planning, trend analysis

ESX vSAN Debug Commands

Advanced read-only commands for a support case (ESX shell):

  esxcli vsan debug object health summary get          → object health counts
  esxcli vsan debug object list -u <uuid>              → component placement and state for one object
  esxcli vsan debug resync summary get / list           → resync backlog
  esxcli vsan debug disk list                           → device health per host
  esxcli vsan debug limit get                           → component and capacity limits in use
  esxcli vsan debug controller list                     → storage controllers and their queue depth
  esxcli vsan network list                              → vSAN VMkernel interfaces and multicast/unicast mode
  esxcli vsan health cluster list                       → health test results as the host sees them

Collect them with vm-support (vSAN manifest) or the SoS --esx-logs run rather than by hand when opening a case.

Key Takeaways

  • Resync: the data to rebuild is the used capacity of the failed device, moved at the cluster's effective resync throughput (Monitor > vSAN > Resyncing Objects gives bytes left and ETA); adaptive resync throttles automatically under VM I/O pressure; the old manual throttle is no longer in the vSphere Client (KB 326830).
  • CMMDS holds the cluster metadata, CLOM decides placement and rebuild, DOM performs object I/O and resync, LSOM owns the devices; read them through esxcli vsan debug object/disk/resync and Skyline Health; an object needs more than 50 percent of its votes reachable to stay accessible.
  • Stretched cluster witness: a witness host at a third site (less than 200 ms RTT to each data site) is the tie-breaker; on an inter-site partition the preferred site plus the witness keep serving data; if the witness itself fails, objects become noncompliant but data stays available and running VMs are unaffected (VCF 9.0.1 Table 855).

NSX Troubleshooting Deep Dive#

Control Plane Diagnostics: NSX Manager Cluster and CCP

NSX Manager (management plane plus central control plane):

SSH into NSX Manager (NSX CLI, admin user):

  ssh admin@<nsx-mgr-fqdn>

Check the Manager cluster:

  get cluster status
  → every service group STABLE, all three nodes UP (DATASTORE, CLUSTER_BOOT_MANAGER, CONTROLLER, MANAGER, POLICY, HTTPS, MONITORING, CORFU_NONCONFIG, SITE_MANAGER ...)
  get cluster config
  → node UUIDs, IPs and the cluster VIP
  get managers
  → the Manager nodes this node knows and their connectivity
  get version / get services / get service http (restart service http if the UI or API is down)

Central control plane (CCP):

  The CCP runs inside each NSX Manager node (no controller appliances since NSX-T 2.4; VCF 9.0.1 docs: "The NSX Management and Control Plane Models are contained within the NSX Manager nodes"). It computes logical switching, routing and DFW state and pushes it to the transport nodes.
  On a transport node: nsxcli -c get controllers (ESX) or get controllers (Edge CLI)
  → the Manager nodes acting as controllers, controller status connected, session state up
  CCP log on each Manager node: /var/log/cloudnet/nsx-ccp.log; management plane API log: /var/log/proton/nsxapi.log; system log: /var/log/syslog (get log-file syslog)

If a Manager node is down:

  1. get cluster status on a surviving node: the cluster keeps quorum with two of three nodes (read-only with one)
  2. Restore the node: power it on, or System > Appliances > redeploy / replace the failed node (same version, same cluster)
  3. Hosts re-establish controller sessions automatically; confirm with nsxcli -c get controllers on a few hosts
  4. Never remove two nodes at once; if the cluster is lost, restore from the NSX configuration backup (System > Backup & Restore) onto a new Manager with the same FQDN or VIP

Data Plane Diagnostics: Segments & Forwarding

On an ESX transport node (nsxcli):

  nsxcli -c get logical-switches                 → segments realised on this host (VNI, admin state)
  nsxcli -c get logical-switch <vni> arp-table / mac-table / vtep   → learned ARP, MACs and remote TEPs for a segment
  nsxcli -c get logical-routers                  → DR instances on the host
  nsxcli -c get logical-router <uuid> forwarding → distributed router forwarding table
  summarize-dvfilter | grep <vm-name>            → the VM's switch port and DFW filter name
  nsxcli -c get firewall <vif-id> ruleset rules  → DFW rules realised on that port, with hit counts

On an NSX Edge (NSX CLI):

  get logical-routers                            → SR and DR instances with their VRF ids
  vrf <n>; get interfaces; get route; get forwarding   → interfaces, routing table and forwarding of that router
  get high-availability status                   → active/standby role per SR
  get tunnel-ports; get bfd-sessions             → Edge TEPs and tunnels to hosts

Traceflow (Plan & Troubleshoot > Traffic Analysis > Traceflow):

  Source VM and vNIC, destination VM or IP, protocol and port. Observations show Injected, Forwarded, Received, Delivered and Dropped with the component (DFW rule id, routing, tunnel) that dropped it.
  If dropped by the DFW: read the rule id, check the group membership of source and destination (Inventory > Groups > View Members), the Applied To scope and the category order; the default rule is Allow after host preparation, so a Drop here comes from an explicit rule. Avoid "temporary allow-all" rules in production; use Traceflow plus the rule hit counts.

Packet Capture

On an ESX host (pktcap-uw; not available on the Edge CLI):

  net-stats -l                                   → switch port ids per VM
  pktcap-uw --switchport <port-id> --dir 2 -o /tmp/vm.pcap            → both directions at the VM port
  pktcap-uw --uplink vmnic0 --capture UplinkRcvKernel --ng -o /tmp/uplink.pcap   → uplink receive path (VCF 9.0.1 docs)
  pktcap-uw --vmk vmk10 -o /tmp/tep.pcap         → Geneve traffic at the TEP interface

On an NSX Edge (NSX CLI):

  start capture interface <fp-ethX> file <name>.pcap [expression "..."]   → fastpath interface capture; stop with stop capture session <id>
  Files are under /var/vmware/nsx/file-store on the Edge; copy with scp and open in Wireshark.

Reading: no SYN-ACK = destination down or filtered; RST = reset by the host or a firewall; retransmissions = loss or MTU; Geneve on UDP 6081 between TEPs.

Edge Service Diagnostics: BGP & Routing

Tier-0 BGP health (Edge CLI, in the Tier-0 SR VRF):

  get logical-routers; vrf <n>
  get bgp neighbor summary                       → state Established per neighbour; Idle/Active/Connect = no session
  get bgp neighbor <ip>                          → timers (VCF design default: keepalive 4 s, hold 12 s; BFD optional), prefixes received/advertised, last error
  get bgp neighbor <ip> advertised-routes / routes
  get route bgp                                  → learned BGP routes

If a neighbour is down:

  1. Uplink reachability from the SR: ping <neighbor-ip> source <uplink-ip> (uplink VLAN, interface address, MTU)
  2. Session parameters on both sides: local/remote AS, neighbour address, password, timers, BFD
  3. Route maps and prefix lists (Networking > Tier-0 > Routing > Route Filters), and Tier-0 route redistribution (connected, Tier-1 connected/LB/NAT must be ticked to be advertised)
  4. Logs: Edge /var/log/syslog (routing daemon messages); NSX UI alarms for BGP down
  Clear a session: clear bgp neighbor <ip> (soft options available)

Tier-1 routing:

  A Tier-1 with no services is distributed (DR only); with NAT, LB or gateway firewall it also has an SR on the Edge cluster.
  Route advertisement from Tier-1 to Tier-0: Networking > Tier-1 Gateways > Route Advertisement (connected segments, NAT IPs, LB VIPs); Tier-0 redistribution then exports them to BGP.
  On the Edge: vrf <tier-1-sr-vrf>; get route — connected segments and the default to the Tier-0 via the router link.

TEP Tunnel State & Geneve Diagnostics

TEP: Tunnel Endpoint — each ESX host and Edge has one or more TEP IPs for the Geneve overlay.

Check TEP status (on ESXi):

  nsxcli -c get vteps
  → Shows TEP IPs, MACs and the host switch they belong to

Check tunnel status:

  nsxdp-cli bfd sessions list
  → BFD sessions to peer TEPs with state (up/down) on the ESX host (get bfd-sessions is the Edge command); System > Fabric > Hosts > Tunnels shows the same per host

If tunnel down:

  1. Verify TEP IP reachable: vmkping -I vmk-nsx <remote-tep-ip>
  2. Check MTU: must be ≥1600 (Geneve minimum; 1700 recommended and the NSX default Global TEP MTU)
     vmkping -I vmk-nsx -s 1572 -d <remote-tep-ip>   # -d = do not fragment
     (1572 payload + 28 ICMP/IP headers = 1600 on the wire)

Key Takeaways

  • Manager cluster health: get cluster status must show every group STABLE on all three nodes; two nodes keep quorum, one node is read-only; the CCP runs inside the Manager nodes and hosts report it with nsxcli -c get controllers.
  • BGP neighbour state: get bgp neighbor summary in the Tier-0 SR VRF; check uplink reachability, AS numbers, passwords, timers/BFD, route filters and Tier-0 redistribution before clearing the session.
  • Packet capture: pktcap-uw at the VM switch port, uplink or TEP vmk on ESX; start capture interface <fp-eth> on the Edge CLI; open in Wireshark to separate DFW drops (no SYN-ACK or RST) from routing and MTU problems.

VCF Upgrade & vSphere-to-VCF Conversion (Objectives 5.2–5.3)#

VCF 5.x to 9.0 Upgrade Path (Obj 5.2)

A VCF 5.x to 9.0 upgrade moves the management planes first and then the core SDDC components of each VCF Instance in a fixed order, orchestrated from VCF Operations with SDDC Manager executing the instance steps (VCF 9.0.1 docs, Upgrading to VCF 9.0: 'You must upgrade the management planes first, followed by core SDDC components').

Upgrade Sequence (Broadcom KB 390634, VCF 9.0; the numbering is the KB's):

  1. VCF Download Tool (binaries for the upgrade)
  2. VMware Aria Suite Lifecycle 8.18 patch (no upgrade path to 9; patched so it can hand over)
  3. VCF Operations and the VCF Operations fleet management appliance (Aria Operations becomes VCF Operations)
  4. VCF Automation; VCF Operations for networks (in parallel)
  5. VCF Operations for logs (new installation; Aria Operations for Logs has no upgrade path)

5-8. Third-party and adjacent: VADP-based backup software, vSphere Replication, Site Recovery Manager / VMware Live Site Recovery, Avi Load Balancer

  1. SDDC Manager and VCF services
  2. HCX
  3. NSX Manager cluster (then edges and hosts by the NSX upgrade coordinator)
  4. vCenter (reduced downtime upgrade)
  5. VCF Identity Broker (new installation; VMware Identity Manager has no upgrade path); Supervisor (in parallel)
  6. vSAN witness host
  7. ESX hosts (rolling, through vSphere Lifecycle Manager images) with NSX finalize
  8. VMware Tools
  9. Virtual machine hardware
  10. vSAN on-disk format / VMFS
  11. vSAN File Services

Within an Instance the core order is therefore SDDC Manager, NSX, vCenter, ESX; workload domains can be upgraded before or after the management domain as long as every component in them is compatible.

Pre-Upgrade Checks:

├── Run the update precheck: VCF Operations > Fleet Management > Lifecycle (or SDDC Manager > Inventory > Workload Domains > <domain> > Updates > Precheck)
├── Verify bundle compatibility: ensure all component bundles match target VCF version
├── Check vSAN health: all health checks must pass before host upgrades
├── Verify NTP synchronization across all components
├── Take the documented prerequisites: SFTP backup and snapshot of SDDC Manager, backups of vCenter and NSX Manager
├── Ensure N+1 host capacity in each cluster for rolling upgrade
└── Review VMware Interoperability Matrix for component version compatibility

Upgrade Binaries:

SDDC Manager downloads the binaries (called bundles before 9.0) from the Broadcom online depot (dl.broadcom.com, with a download token) or from an offline depot populated by the VCF Download Tool for air-gapped sites:

├── Binary types: VCF (SDDC Manager), vCenter, ESX (vLCM image), NSX, vSAN witness, plus the VCF Operations components handled by fleet management
├── Validation: checksums verified on download; the Lifecycle UI shows Available, Downloading and Downloaded states
├── Staging: an upgrade can be planned and scheduled; binaries are pushed to the components at run time
└── No automatic rollback of the upgrade as a whole: a failed step is retried after the cause is fixed (vCenter reduced-downtime upgrade is the exception: Cancel while running, automatic rollback on failure); recovery is from the pre-upgrade backups and snapshot

Common Upgrade Issues:

├── Pre-check failures: resolve ALL pre-check warnings before proceeding
├── Bundle download timeout: verify depot URL reachability and proxy settings
├── ESXi host stuck in maintenance mode: check vSAN data evacuation status
├── NSX upgrade compatibility: NSX version must be compatible with target vCenter version
└── Post-upgrade vSAN resync: monitor resync completion before upgrading next host

vSphere-to-VCF 9.0 Conversion (Obj 5.2)

Converging an existing vSphere environment into a VCF 9.0 management domain (VCF 9.0.1 docs, Converging Existing Virtual Infrastructure to a VCF or a vSphere Foundation Platform):

Conversion Process:

  1. Upgrade the existing components to the scenario minimums: VCF 9.0.0 requires vCenter 9.0 and ESX 9.0; VCF 9.0.1 and 9.0.2 require vCenter 8.0 Update 3 and ESX 8.0 Update 1 (with NSX 4.2.1 if an existing NSX is to be converged; 9.0.0 cannot converge an existing NSX and deploys a new one)
  2. Deploy the VCF Installer appliance and download the binaries (online depot with a token, or offline depot from the VCF Download Tool)
  3. Run the converge workflow in the VCF Installer: the existing vCenter and its hosts become the management domain; the missing components are deployed (SDDC Manager, NSX Manager cluster or simple node, VCF Operations and fleet management, optionally VCF Automation)
  4. Validate the fleet in VCF Operations; license from VCF Operations
  5. Import further vCenter instances as workload domains afterwards from VCF Operations (Objective 5.3), not with the Installer

Conversion Requirements (Supported Configurations table):

├── vSphere Distributed Switch (version 8.0 or later) with the management, vMotion and storage networks; standard switches are not supported
├── Any supported vSphere storage type; when several datastore types exist VCF picks the principal datastore by priority (vSAN first)
├── Host count for the new components: simple deployments need at least 3 ESX hosts with vSAN or 2 with external storage; high-availability deployments need at least 4
├── Enough free capacity for the full VCF deployment per the Planning and Preparation Workbook; a datastore shared, accessible and writable by every host in the cluster
├── DNS forward and reverse records and NTP for every existing and new component
└── Existing NSX: only 9.0.1 and later converge it, and only without Enhanced Linked Mode; a Federation local manager converges as a local manager only

The VCF Import Tool belongs to VCF 5.2; in 9.0 the VCF Installer converges the management domain and VCF Operations imports workload domains.

VCF Workload Domains — Create, Scale, Import (Obj 5.3)

Creating Workload Domains (VCF 9.0.1 docs, Create a New Workload Domain Using VCF Operations):
VCF Operations > Inventory > Detailed View > VCF Instances > <instance> > Add Workload Domain > Create New:

├── Prerequisites: a vSphere Lifecycle Manager cluster image imported for the instance, hosts commissioned with the matching principal storage type, a network pool with free IPs, a static IP pool or DHCP for the host overlay (TEP) VLAN, DNS for the new vCenter and NSX FQDNs
├── General: domain name, vSphere Supervisor on or off (on = Supervisor with NSX VPC networking; an NSX Edge cluster is needed afterwards), SSO domain (join the instance's or create an isolated one), passwords
├── vCenter: FQDN (the workload domain vCenter is deployed into the management domain); cluster name and image
├── NSX: create a new NSX Manager cluster or share one with another workload domain (never the management domain's)
├── Storage and hosts: vSAN (OSA or ESA), NFS, FC or vVols principal storage; select the commissioned hosts (vSAN needs at least three per standard cluster)
└── Validation runs before deployment; the domain appears Active in Inventory when done

Scaling Workload Domains:

Add Cluster / Add Hosts:

├── Commission hosts first: vSphere Client > Global Inventory Lists > Hosts > Unassigned Hosts > Commission Hosts (ESX version per BOM, DNS, NTP, SSH; storage type and network pool chosen at commissioning)
├── Add a cluster or add hosts to a cluster from the vSphere Client inventory of the domain (VCF-aware actions on the domain and cluster objects); the vLCM image is applied and NSX transport node configuration is pushed automatically
├── vSAN cluster: minimum three hosts (two-host clusters need an external witness); LACP-enabled VDS domains are created and scaled through the SDDC Manager API
└── Network pool capacity (vMotion, vSAN, NFS, iSCSI ranges) must have free addresses for the new hosts

Remove Cluster / Remove Hosts:

├── Migrate or power off the VMs; for vSAN choose the evacuation mode
├── Remove the host or cluster from the vSphere Client VCF actions; the host leaves the cluster, NSX transport node configuration is removed and the host returns to the unassigned pool
└── Decommission from Unassigned Hosts to drop a host from the VCF inventory entirely

Import an Existing vSphere Environment as a Workload Domain (VCF 9.0.1 docs, Import an Existing vCenter to Create a Workload Domain):

├── From VCF Operations: Inventory > VCF Instance > Add Workload Domain > Import; all clusters of that vCenter are imported (no subset)
├── Minimums: vCenter 8.0 Update 1 or later, ESX 8.0 Update 1 or later, NSX Manager 4.1.0.2 or later if present (otherwise a new NSX is deployed during the import); SSH enabled on the vCenter
├── Non-destructive for running VMs; NSX is activated on the distributed port groups, which also activates the DFW with allow-all default rules (can be deactivated afterwards)
├── Supported configurations table: shared storage on all hosts, VDS 8.0 or later, no standard switches, no unsupported features in use (see Table 822)
└── Isolated workload domain: the imported vCenter keeps its own SSO domain (objective 5.3 'import ... as a VCF Workload Domain (isolated)')

VCF Operations — Observability Workbench & Log Assist (Obj 5.9)

The Observability Workbench is a VCF 9.0 feature in VCF Operations that provides correlated troubleshooting across metrics, logs, and events in a single view.

Observability Workbench:

├── Unified view: Select any VCF object (host, VM, cluster) → see metrics timeline + event timeline + log entries
├── Correlation: Click a metric anomaly → see concurrent logs and events → identify root cause
├── Cross-component: Correlates vCenter events, ESXi logs, vSAN health, NSX alerts in one timeline
├── Time range: Adjustable time window for investigation (last 1h, 6h, 24h, custom)
└── Export: Export correlated data for support case attachment

Log Assist:

├── Purpose: generate support log bundles from VCF Operations and upload them straight to a Broadcom support case (VCF 9.0.1 docs p.3909, 'Use Log Assist to Accelerate Your Support Cases') — it is not an AI log-explanation or root-cause engine
├── Attaches the Diagnostic Findings relevant to the case to the bundle
├── Set up Log Assist before first use: licenses registered in Connected Mode, Log Assist activated on a unified cloud proxy (Administration > Control Panel > Cloud Proxies; the Classic collector does not work with Log Assist), inventory objects assigned under Infrastructure Operations > Configurations > Log Assist Assignment (VCF 9.0 docs, Set up Log Assist)
├── Log bundle generation: open Log Assist from the Diagnostics Findings tab, select the inventory object, and enter the Support case ID on the Transfer to Support Portal tab
└── Upload: Log Assist uploads the bundle to the Broadcom case directly; the manual alternative is the Broadcom support portal

VCF Operations for Networks:

├── Network topology visualization: See NSX segments, gateways, DFW rules graphically
├── Flow analysis: Identify VM-to-VM traffic patterns, detect anomalies
├── Path analysis: Trace packet path from source to destination through NSX overlay
├── Intent verification: Compare intended network policy with actual forwarding state
└── Troubleshooting: Identify dropped flows, misconfigured rules, routing loops

Key Takeaways

  • VCF 9.0 upgrade sequence (KB 390634): management planes first (VCF Download Tool, Aria Suite Lifecycle patch, VCF Operations and fleet management, VCF Automation and Operations for networks, Operations for logs new install, backup/VR/SRM/Avi), then per instance SDDC Manager, HCX, NSX, vCenter, Identity Broker and Supervisor, vSAN witness, ESX with NSX finalize, VMware Tools, VM hardware, vSAN on-disk format, vSAN File Services.
  • Pre-upgrade pre-checks are mandatory — resolve ALL warnings before proceeding. Common blocker: vSAN health check failures.
  • vSphere-to-VCF 9.0 conversion: upgrade to the scenario minimums first (9.0.0: vCenter 9.0 and ESX 9.0; 9.0.1+: vCenter 8.0 U3 and ESX 8.0 U1), then the VCF Installer converges the management domain (VDS 8.0+, any supported storage, 3 hosts simple / 4 hosts HA); further vCenters are imported as workload domains from VCF Operations.
  • Workload domains are created in VCF Operations (Add Workload Domain > Create New or Import) with commissioned hosts, a vLCM image and a network pool with free IPs; a standard vSAN cluster needs three hosts.
  • Hosts are commissioned from the vSphere Client (Global Inventory Lists > Hosts > Unassigned Hosts > Commission Hosts) with validation of version, DNS, NTP and storage type; clusters and hosts are added or removed from the domain's vSphere Client inventory.

HCX Workload Migration for VCF (Objective 5.11)#

HCX Overview & Architecture

HCX (Hybrid Cloud Extension) enables large-scale workload migration between sites — on-premises to cloud, VCF to VCF, or vSphere to VCF. Key for Obj 5.11.

HCX Components:

├── HCX Manager: management plane, deployed as OVA on source and destination
├── HCX Connector: source-side component (on-premises or legacy site)
├── HCX Cloud: destination-side component (target VCF or VMC environment)
├── Service Mesh: network tunnel between sites for migration traffic
├── IX (Interconnect): the migration appliance pair — encrypted tunnel carrying vMotion, Bulk and RAV traffic
└── NE (Network Extension): L2 stretch appliance pair (extends VLANs across sites)

WAN Optimization (HCX-WAN-OPT) is discontinued in VCF Operations HCX 9.0: an upgrade from earlier HCX releases is blocked while any Service Mesh or Compute Profile uses it, so remove it before upgrading (VCF 9.0 release notes, VCF Operations product support notes).

HCX Configuration (Obj 5.11)

Deployment Steps:

  1. Deploy HCX Manager OVA on source vCenter
  2. Licensing: VCF Operations HCX 9.0 has no separate license key; it is licensed by the VCF license assigned to the vCenter instance (VCF 9.0 release notes, What's New)
  3. Configure HCX Connector: pair with destination HCX Cloud
  4. Site Pairing: source HCX ↔ destination HCX (requires network connectivity on port 443)
  5. Create Compute Profile: define source/destination resources (clusters, datastores, networks)
  6. Create Service Mesh: select services/appliance types (IX, NE) and uplink networks
  7. Deploy Service Mesh appliances (auto-deployed as VMs on both sites)
  8. Verify tunnel status: Service Mesh → Tunnel Status → all tunnels green

Network Requirements:

├── Site to site: TCP 443 (HCX Manager pairing and management), UDP 4500 (IX and NE tunnel uplinks)
├── Local only (ESX host → local IX appliance, never across the WAN): TCP 8000 vMotion, TCP 31031 replication
├── Underlay minimums (VCF 9.0.1 docs p.5327-5328, Network Underlay Minimum Requirements): MTU 1150 for IX (1500 for HCX vMotion direct; 1150–9000 valid), bandwidth 250 Mbps for vMotion / Replication Assisted vMotion / Bulk (50 Mbps for Assisted vMotion and OS Assisted Migration), latency ≤150 ms, packet loss ≤0.1 % (1 % for the 50 Mbps services)
└── MTU is set in the Site Resource Profile before the IX appliances are deployed; changing it means redeploying the appliances

HCX Workload Migration Types (Obj 5.11)

Bulk Migration (VCF 9.0.1 docs, Migration Types):

├── Uses the vSphere Replication protocols to copy VMs to the destination, many in parallel
├── The VM keeps running at the source until the switchover; the switchover can be scheduled to a window
├── Service interruption at switchover is equivalent to a reboot (the VM is powered on at the destination)
├── Concurrency limits are published in VMware Configuration Maximums for the HCX release, not fixed here
└── Best for: waves of VMs where a reboot-length interruption is acceptable

HCX vMotion:

├── Uses the vMotion protocol over the Service Mesh, one VM at a time
├── Live: the VM state moves with the VM, no service interruption
└── Best for: single sensitive VMs; no re-IP needed when the network is extended

RAV (Replication Assisted vMotion):

├── Combines Bulk Migration's parallel replication, resiliency and scheduling with vMotion's zero-downtime switchover
├── Phase 1: disks replicated in parallel while the VM runs; Phase 2: a vMotion of the running state at the switchover time
└── Best for: large waves that still need zero downtime

Cold Migration:

├── Uses the NFC protocol; selected automatically when the source VM is powered off
├── Downtime: the VM is off for the whole copy
└── Best for: powered-off VMs, templates, and VMs whose configuration excludes live migration

OS Assisted Migration (OSAM):

├── Agent-based migration for physical-to-virtual (P2V) or cross-platform
├── Installs HCX Sentinel agent on source machine
├── Replicates at block level, converts to VM on destination
├── Supports: Windows, Linux source machines (physical or virtual)
└── Best for: legacy workloads on non-VMware platforms

HCX Troubleshooting for Support

Common Issues:

├── Service Mesh deployment fails: check network connectivity, port 443 open, DNS resolution
├── Tunnel status shows 'Degraded': verify MTU settings, WAN latency (<150ms recommended)
├── Migration fails at 80%: usually disk I/O bottleneck — check storage latency on both sides
├── vMotion switchover timeout: insufficient bandwidth or high memory change rate on VM
├── L2 Extension MAC conflict: duplicate MAC addresses across sites — use HCX mobility-optimized networking (MON)
└── License expiry: HCX requires valid subscription — check Broadcom portal for renewal

Migration Planning Best Practices:

├── Wave planning: group VMs by application affinity (migrate app tiers together)
├── Pre-migration validation: run HCX Migration Validation to check VM compatibility
├── Network dependency mapping: identify VM-to-VM network flows to avoid breaking application chains
├── Rollback plan: HCX supports reverse migration if issues detected post-cutover
├── Performance monitoring: use HCX Dashboard to track migration progress, errors, bandwidth
└── Post-migration validation: verify VM health, network connectivity, application functionality

Key Takeaways

  • HCX Service Mesh creates IX/NE appliance pairs between sites (WAN Optimization is discontinued in HCX 9.0) — the sites need TCP 443 (manager pairing) and UDP 4500 (tunnels) open; TCP 8000 and 31031 are host-to-local-IX flows, not WAN openings.
  • RAV combines Bulk Migration's parallel, schedulable replication with vMotion's zero-downtime switchover; Bulk Migration alone interrupts service for the length of a reboot at switchover; HCX vMotion moves one VM at a time live.
  • Concurrent migration limits come from VMware Configuration Maximums for the HCX release; plan waves by application affinity and by what each VM tolerates (reboot-length switchover for Bulk, none for vMotion/RAV).
  • L2 Network Extension stretches VLANs across sites — enables migration without re-IP. Mobility Optimized Networking (MON) prevents tromboning.
  • HCX troubleshooting: check tunnel status, underlay MTU (min 1150 for IX, set in the Site Resource Profile), bandwidth (≥250 Mbps for vMotion/Bulk/RAV), WAN latency (≤150 ms) and UDP 4500 / TCP 443 connectivity before migration.

Exam Mapping: 2V0-15.25 — VMware Cloud Foundation 9.0 Support (2V0-15.25)

  • 5.1 VCF fleet deployment and adding/removing VCF Instances (VCF Troubleshooting Methodology; Log Collection Deep Dive (SoS Utility))
  • 5.2 VCF 5.x to 9.0 upgrade and vSphere to VCF 9.0 conversion (VCF Lifecycle Troubleshooting; VCF Upgrade & vSphere-to-VCF Conversion)
  • 5.3 VCF workload domains: create, scale clusters and hosts, import a vSphere environment (VCF Upgrade & vSphere-to-VCF Conversion — Workload Domains)
  • 5.4 VCF Operations fleet management: certificates, passwords, Identity Broker IAM (VCF Lifecycle Troubleshooting — Certificate Management; vCenter / VCSA Troubleshooting)
  • 5.5 License management: assignment, entitlements, expiry (VCF Lifecycle Troubleshooting — version and license)
  • 5.6 VCF compute: ESX hosts, vCenter, VMs, cluster configuration (vSphere Troubleshooting; ESXi & Hardware Troubleshooting; vCenter / VCSA Troubleshooting; Performance Troubleshooting)
  • 5.7 VCF storage: vSAN, stretched clusters, supplemental storage (vSAN Troubleshooting; vSAN Troubleshooting at Scale)
  • 5.8 Networking: VDS, NSX configuration, routing, gateways, services, VPC (NSX Troubleshooting; NSX Troubleshooting Deep Dive)
  • 5.9 VCF Operations — logs, networks, Observability Workbench, Log Assist, dashboards/reports/alerts (Log Collection Deep Dive (SoS Utility); VCF Upgrade & vSphere-to-VCF Conversion — VCF Operations)
  • 5.10 Identity Broker, VCF Automation, Supervisor, provider management and organizations (no dedicated topic in this section; partial coverage in vCenter / VCSA Troubleshooting)
  • 5.11 HCX configuration and workload migrations (HCX Workload Migration for VCF)
  • Cross-objective: Support Engineering Fundamentals (Broadcom support process, severity, case preparation) supports every objective
  • Objective ids and titles follow the 2V0-15.25 exam guide (last updated 15 Aug 2025), Section 5; Broadcom publishes no objective weights

Labs in This Section

VMware Cloud Foundation Overview
BeginnerVCF 9.060 min
vSphere Foundation Overview
BeginnerVCF 9.075 min
Compute Core Concepts — ESXi Host Architecture
BeginnerVCF 9.060 min
Storage Core Concepts — vSAN for VCF
BeginnerVCF 9.060 min
Networking Core Concepts — NSX Architecture
BeginnerVCF 9.060 min
VMware Cloud Foundation Installation — Deployment Wizard & JSON Spec
IntermediateVCF 9.090 min
vSphere Foundation Installation
IntermediateVCF 9.090 min
Gate: vcffts9-03
VMware Cloud Foundation Upgrades — 5.x to 9.0 & Conversion
IntermediateVCF 9.075 min
Gate: vcffts9-03
License Management — VCF 9.0 Licensing Model
BeginnerVCF 9.030 min
Creating Workload Domains & Network Pools
IntermediateVCF 9.060 min
VCF Operations Console — Monitoring Storage, Compute & Security
IntermediateVCF 9.075 min
Gate: vcffts9-07
VCF Operations for vSphere Foundation
BeginnerVCF 9.060 min
VCF Automation — Purpose & Functionality
BeginnerVCF 9.045 min
Technical Support Fundamentals — Broadcom Wolken Case Management
BeginnerVCF 9.045 min
VCF Operations HCX — Workload Mobility & Migration Troubleshooting
IntermediateVCF 9.030 min
vCenter Service Recovery
IntermediateVCF 9.0120 min
vSAN Disk Failure Recovery
IntermediateVCF 9.0120 min
DFW Rule Debugging & Traceflow
IntermediateVCF 9.0120 min
Upgrade Failure & Remediation
AdvancedVCF 9.0150 min
Performance Analysis with esxtop
IntermediateVCF 9.090 min
vSAN Failure & Recovery Scenario
AdvancedVCF 9.0120 min
NSX Connectivity Troubleshooting
AdvancedVCF 9.090 min
Upgrade Troubleshooting Scenario
IntermediateVCF 9.090 min
Advanced esxtop Performance Analysis
AdvancedVCF 9.090 min
Avi Load Balancer Troubleshooting
IntermediateVCF 9.090 min

📝 Quiz — VCF 9.0 Support

0/50 correct

Section 1 — Troubleshooting Methodology and Tools

Q1
Which command generates an SOS diagnostic bundle from SDDC Manager for a specific workload domain?
  • python /opt/vmware/sddc-support/sos --domain-name <name>
  • sddc-manager collect --logs
  • esxcli sos bundle create
  • vcf-support-bundle --domain <name>
The SoS diagnostic bundle is generated via: python /opt/vmware/sddc-support/sos --domain-name <name>. This collects logs from all VCF components in that domain. esxcli sos is not a valid command. The other options use incorrect syntax.
Q2
Where are SDDC Manager operation logs located on the appliance?
  • /var/log/vmware/vcf/
  • /var/lib/vmware/sddc/
  • /var/log/messages
  • /opt/vmware/logs/
SDDC Manager operation logs are located in /var/log/vmware/vcf/ on the appliance. This includes domainmanager.log, operationsmanager.log, and other VCF-specific service logs. /var/log/messages is general syslog. The other paths don't exist.
Q3
A support engineer needs to correlate logs from vCenter, NSX, and ESXi during a failed upgrade. Which VCF tool is purpose-built for this?
  • SDDC Manager UI Dashboard
  • vSphere Client Tasks & Events tab
  • VCF Operations for Logs with unified indexing
  • esxtop
VCF Operations for Logs (formerly vRealize Log Insight) provides unified log indexing across vCenter, NSX, and ESXi for cross-component correlation during troubleshooting. The SDDC Manager dashboard shows status, not log correlation. Tasks & Events is per-component. esxtop is for real-time metrics.
Q4
When opening a Broadcom support case for a VCF fleet issue, which artifact is the most important to attach FIRST?
  • The vCenter configuration backup
  • A screenshot of the alert
  • The SOS diagnostic bundle
  • A list of installed VMware Tools versions
The SOS diagnostic bundle is the most critical artifact for Broadcom support — it contains comprehensive logs, configs, and health data from all VCF components. Screenshots lack detail. VMware Tools versions are supplementary. vCenter backup is for recovery, not troubleshooting.
Q5
Which ESXi CLI command is used to test VMkernel-level IP connectivity with a specific MTU to validate jumbo frames on the vSAN network?
  • vmkping -I vmk2 -s 8972 -d <ip>
  • tcpdump -n host <ip>
  • esxcli network diag ping --host <ip>
  • ping -s 8972 <ip>
vmkping -I vmk2 -s 8972 -d <ip> tests VMkernel connectivity with a specific interface (-I vmk2), large packet size (-s 8972 for MTU 9000 minus headers), and do-not-fragment flag (-d). Regular ping doesn't use VMkernel stack. esxcli network diag ping exists but doesn't specify the VMkernel adapter. tcpdump captures packets but doesn't test connectivity.

Section 2 — vSphere and Compute Troubleshooting

Section 3 — vSAN Storage Troubleshooting

Section 4 — NSX Networking Troubleshooting

Section 5 — VCF Operations and Platform Troubleshooting

🃏 Flashcards — VCF 9.0 Support

50 cards
Card 1 of 50
Structured Troubleshooting Loop
A repeatable methodology — identify symptoms, collect data, hypothesize, test, fix, verify, document — that VCF support engineers follow for every case. Following the loop prevents guess-based changes and produces an auditable record for root-cause analysis. Skipping steps typically extends outages and invites regressions.
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.