Academy/vSphere Foundation 9.0 Support (2V0-18.25)
VCF

vSphere Foundation 9.0 Support (2V0-18.25)

VCF 9.0vcp-foundation

Troubleshooting, diagnosis, and remediation for VVF environments. Focus on support escalation, log analysis, and remedial procedures. 60 questions, 135 minutes.

2V0-18.25
VCP
60
Questions
135m
Duration
300 (scaled)
Pass Score
45
Objectives

Exam Blueprint Weights

Section titles, groupings and weights below are VCDX Academy study groupings, NOT the official Broadcom blueprint structure. Broadcom publishes no section weights. Always cross-check the official exam guide. Official exam guide ↗
Section 1 — Troubleshooting Methodology
~12%
Section 2 — vSphere Compute Issues
~25%High Weight
Section 3 — Storage Issues
~22%
Section 4 — Networking Issues
~18%
Section 5 — Upgrade and Migration Issues
~23%

Version Evolution

VVF (vSphere Foundation) support is a subset of full VCF support. VVF excludes NSX, vSAN advanced features, and SDDC Manager. Support focus is on vSphere compute, basic networking (vDS), and standard storage. Understanding VVF scope helps candidates distinguish between VVF-only and full VCF troubleshooting scenarios.

Learning Outcomes

  • Obj 5.1: Troubleshoot VVF deployment — ESXi installation, vCenter deployment, initial configuration
  • Obj 5.2: VVF upgrade — vSphere to VVF 9.0 conversion, ESXi/vCenter/vSAN upgrade paths
  • Obj 5.3: VVF clusters — create, scale (add/remove hosts/clusters), import existing vSphere environments
  • Obj 5.4: License management — assign licenses to vCenter instances from VCF Operations, understand the cores + vSAN TiB entitlements and the 180-day usage reporting, handle expiry and renewal
  • Obj 5.5: Compute troubleshooting — ESXi boot, hardware compatibility, drivers, esxtop performance
  • Obj 5.6: Storage troubleshooting — vSAN Skyline Health, disk replacement (OSA disk groups vs ESA storage pool), capacity reserves, resync, Performance Service and I/O Insight
  • Obj 5.7: Network troubleshooting — VDS/VSS health check, port group issues, VMkernel adapters, NIOC, MTU
  • Obj 5.8: VCF Operations — adapter config, dashboards, Ops for Logs (syslog, Explore Logs), Observability Workbench, Log Assist
  • Obj 5.9: VCF Operations Orchestrator — configuration, pre-built and custom workflows, scheduling, event triggers

VVF Support Fundamentals#

VVF support scope: vSphere (ESX and vCenter), vDS/vSS networking, vSAN, the Supervisor/VKS on VDS networking, VCF Operations and VCF Operations for Logs — no NSX, no SDDC Manager. The 2V0-18.25 objectives in Section 5 test the support workflow (deployment, upgrade, clusters, licensing), compute/storage/network troubleshooting and the VCF Operations tooling.

Support workflow and escalation

  • Reproduce and scope: which object (host, cluster, VM, vCenter service), since when, what changed (patch, hardware, configuration, certificate expiry).
  • Knowledge base: knowledge.broadcom.com — search the exact error string in quotes, then the component and version; note the KB number in the case.
  • Collect evidence before changing anything: support bundles (below), screenshots of alarms, the time window in UTC.
  • Open a Broadcom support case with the bundle attached; in 9.0 VCF Operations Log Assist can gather and upload the bundles for you. Severity follows business impact (production down → highest).

Support bundles (VCF 9.0.1 — Export System Logs; KB 313542)

# vSphere Client: right-click the host (or the vCenter object) > Export System Logs
#   → host bundle = vm-support package; vCenter bundle = vc-support package

# ESX shell: write the host bundle to a datastore and add performance snapshots
vm-support -w /vmfs/volumes/<datastore>
vm-support -p -d 60 -i 5 -w /vmfs/volumes/<datastore>     # 60 s of perf data at 5 s intervals

# vCenter appliance shell
vc-support.sh                                               # writes the bundle under /var/tmp (or /storage/log)
# vCenter Management Interface (VAMI, https://<vcsa>:5480) > Actions > Create Support Bundle

# VCF Operations (9.0): Diagnostics > Log Assist — collect and upload bundles to the Broadcom case

Bundle size depends on log retention and host count; compress nothing manually — the tools already produce .tgz/.zip. Do not send bundles through unencrypted channels.

Diagnostics in 9.0

Skyline Advisor and Skyline Health Diagnostics are retired in favour of VCF Operations Diagnostics: Diagnostics Findings (known issues, VMSA security exposures, best-practice recommendations with remediation steps; export to CSV), the Findings Catalog (every signature) and Log Assist (VCF 9.0 What's New — Diagnostics). vSAN Skyline Health stays in the vSphere Client under Cluster > Monitor > vSAN > Skyline Health.

Key log locations

# vCenter appliance
/var/log/vmware/vpxd/vpxd.log          # vCenter server (vpxd)
/var/log/vmware/vpxd/vpxd-alert.log    # vpxd alerts
/var/log/vmware/vsphere-ui/logs/       # vSphere Client UI
/var/log/vmware/sso/                   # SSO / STS
/var/log/vmware/vmafdd/                # VMware Authentication Framework daemon
/var/log/vmware/vmdird/                # directory service
/var/log/vmware/vpostgres/             # database
/var/log/vmware/applmgmt/              # appliance management (VAMI)
/var/log/vmware/vmware-updatemgr/      # vSphere Lifecycle Manager service
/var/log/vmware/vsan-health/           # vSAN health service

# ESX host
/var/log/vmkernel.log    # VMkernel: storage, network, devices
/var/log/vmkwarning.log  # VMkernel warnings only
/var/log/hostd.log       # host agent (VM power operations, host configuration)
/var/log/vpxa.log        # vCenter agent on the host
/var/log/fdm.log         # vSphere HA agent (Fault Domain Manager)
/var/log/vobd.log        # VMkernel Observation (VOB) events, e.g. esx.problem.vob.vsan.*
/var/log/syslog.log      # system messages
/var/log/vsanmgmt.log    # vSAN management agent (ESX)
/var/log/clomd.log       # vSAN CLOM (placement)
/var/log/lifecycle.log   # vLCM image operations (VCF 9.0.1 docs p.246)
/var/log/esxupdate.log   # VIB installation

Useful shell habits: tail -f /var/log/vmkernel.log | grep -i -E "error|warn|fail", grep -B2 -A2 "<timestamp>" to see context, and esxcli system syslog config get to confirm remote syslog (point hosts at VCF Operations for Logs so the evidence survives a host reboot).

Key Takeaways

  • Support bundles: Export System Logs in the vSphere Client (vm-support / vc-support packages), vm-support -w <datastore> (-p for performance data) on ESX, vc-support.sh or the VAMI on vCenter, and Log Assist in VCF Operations for upload to Broadcom.
  • 9.0 diagnostics live in VCF Operations (Diagnostics Findings, Findings Catalog, Log Assist) — Skyline Advisor/Health Diagnostics are retired; vSAN Skyline Health remains in the vSphere Client.
  • Know the log map: vpxd.log and the SSO/vmafd/vmdir/vpostgres directories on vCenter; vmkernel.log, hostd.log, vpxa.log, fdm.log (HA), vobd.log (VOB events), clomd/vsanmgmt on ESX; ship them to VCF Operations for Logs.

ESXi Host Troubleshooting#

ESX host troubleshooting for the VVF support track.

Boot and hardware

  • Boot failures: check the DCUI/console for the stage (loader, VMkernel, hostd), the boot device health (boot from USB/SD is unsupported for the OS-data partition in 9.0 — use a persistent device), and the BIOS settings (VT-x/AMD-V, NX/XD, VT-d/IOMMU for passthrough).
  • Hardware compatibility: the Broadcom Compatibility Guide (BCG) is the source of truth for server, I/O devices and vSAN ReadyNodes; driver and firmware pairs must match the certified combination. In vLCM, the vendor add-on and the hardware support manager (OEM plug-in) deliver the certified driver/firmware set.
esxcli hardware platform get                 # vendor, model, serial
esxcli hardware pci list                      # devices and the driver bound to each
esxcli software vib list | grep -i -E "nic|hba|driver"   # driver VIB versions
esxcli network nic get -n vmnic0              # driver, firmware, link state, speed/duplex
esxcli storage core adapter list              # HBAs and their drivers
esxcli software profile get                   # image profile on the host

Network connectivity

esxcli network ip interface ipv4 get          # VMkernel interfaces and addresses
esxcli network ip netstack list               # TCP/IP stacks (default, vmotion, provisioning)
esxcli network nic list                       # pNIC link/speed/duplex
esxcli network vswitch standard list          # standard switches
esxcli network vswitch dvs vmware list        # distributed switch membership, uplinks, MTU
vmkping -I vmk2 <target-ip> -s 1472 -d        # path MTU test at 1500 (1472 + 28)
vmkping -I vmk2 <target-ip> -s 8972 -d        # jumbo-frame test at 9000 (VCF 9.0.1 vSAN networking)
nc -z <ip> <port>                             # TCP port reachability from the host

vmkping takes a VMkernel interface name (vmk0, vmk2), not a port-group name; -d sets don't-fragment so an MTU mismatch shows as a failure rather than fragmentation.

Storage paths and multipathing

esxcli storage core device list               # devices, display names, naa IDs
esxcli storage core path list                 # paths: vmhbaN:C0:T0:L0 with state active/dead/standby
esxcli storage nmp device list                 # SATP and PSP per device (e.g. VMW_SATP_ALUA / VMW_PSP_RR)
esxcli storage nmp satp list                   # claim rules per SATP
esxcli storage core path set --state off -p vmhba1:C0:T0:L0    # take a path offline for testing
esxcli storage core path set --state active -p vmhba1:C0:T0:L0
esxcli storage nmp device set -d naa.6xxxx -P VMW_PSP_RR        # change the path selection policy
esxcli storage nmp psp roundrobin deviceconfig set -d naa.6xxxx --iops=1 --type=iops

Paths are named vmhba (adapter):C (channel):T (target):L (LUN) — never vmnic. Dead paths with an active device usually mean fabric zoning, array port or cabling faults; all paths dead (APD) versus permanent device loss (PDL) is distinguished in vmkernel.log and decides whether HA VM Component Protection reacts.

Core dumps and PSOD

esxcli system coredump partition get           # local dump target
esxcli system coredump file list               # dump files on VMFS
esxcli system coredump network set -v vmk0 -i <netdump-collector-ip> -o 6500
esxcli system coredump network set -e true
esxcli system coredump network check            # verify the network dump target

After a PSOD, capture the screen, collect vm-support (the dump is included when the target is configured) and search the KB for the backtrace's top function; do not reboot into production without the dump. Common root causes: faulty memory (check IPMI/iDRAC/iLO logs), driver/firmware mismatch, storage driver faults under APD.

Services on the host

services.sh restart                             # restarts all host agents (brief loss of management)
/etc/init.d/hostd restart ; /etc/init.d/vpxa restart
esxcli system maintenanceMode set -e true       # before invasive work (DRS evacuates VMs)

Key Takeaways

  • Hardware first: BCG-certified driver/firmware pairs delivered through vLCM vendor add-ons and the OEM hardware support manager; esxcli hardware/network/storage commands show what is actually bound.
  • vmkping -I vmkN with -s 1472/-s 8972 and -d is the MTU test; paths are vmhbaN:C:T:L with esxcli storage core path set --state to toggle them and esxcli storage nmp device set -P to change the PSP.
  • Configure a core dump target (partition, file or network) before you need it; a PSOD without a dump is a guess, not a root cause.

vCenter Troubleshooting#

vCenter (VCSA) troubleshooting.

Shell access and services

ssh root@<vcsa>                                 # Bash shell (enable in VAMI > Access if the default is the appliance shell)
service-control --status --all                  # every service with state
service-control --restart vmware-vpxd           # vCenter server
service-control --restart vmware-vsphere-ui     # vSphere Client
service-control --restart vmware-stsd           # Security Token Service (SSO)
service-control --stop --all ; service-control --start --all   # ordered restart of the whole stack
vmon-cli -l                                     # service list from the service monitor
df -h /storage/log /storage/db /storage/core    # appliance disks

Disk space: vCenter stops services or raises VAMI alarms when /storage/log, /storage/db or /storage/core fill. Fix: purge rotated logs under /storage/log (never delete live log files or database files), clear old core dumps in /storage/core, or grow the VMDK and then the partition from the VAMI (Administration > Disk management).

Database (vPostgres)

service-control --status vmware-vpostgres
du -sh /storage/db/vpostgres /storage/seat     # database and stats/events/alarms/tasks partitions
/opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB -c "select now();"   # read-only connectivity check

Supported recovery for a corrupt or unstartable database is a file-based restore of the appliance (VAMI Backup and Restore) or a Broadcom support case. Never delete the vPostgres data directory: it does not re-initialise a clean vCenter, it destroys the inventory, SSO configuration, vDS and licensing state. Statistics growth is managed with the statistics level and the SEAT retention settings, not by deleting rows.

SSO and identity

  • 9.0 identity sources: Active Directory over LDAPS, OpenLDAP, identity federation (SAML/OIDC through the VCF Identity Broker or an external IdP). Integrated Windows Authentication is removed in vCenter 9.0; a host that was joined for IWA must be migrated.
  • Login failures: check the time skew between vCenter, the LDAP server and the IdP (Kerberos/SAML tolerate minutes, not more), the LDAPS certificate chain, and the STS signing certificate validity (vSphere Client > Administration > Certificates > Certificate Management).
/opt/likewise/bin/domainjoin-cli query          # legacy domain-join state (if present from an upgrade)
/usr/lib/vmware-vmafd/bin/dir-cli service list  # solution users registered in the directory
/usr/lib/vmware-vmafd/bin/vecs-cli store list   # certificate stores (MACHINE_SSL_CERT, TRUSTED_ROOTS, ...)

Certificates (vSphere 9.0 Authentication — Certificate Manager)

/usr/lib/vmware-vmca/bin/certificate-manager
#  1  Replace Machine SSL certificate with a custom certificate
#  2  Replace VMCA root certificate with a custom signing certificate and replace all certificates
#  3  Replace Machine SSL certificate with a VMCA certificate
#  4  Regenerate a new VMCA root certificate and replace all certificates
#  5  Replace solution user certificates with custom certificates
#  6  Replace solution user certificates with VMCA certificates
#  7  Revert last performed operation by re-publishing old certificates
#  8  Reset all certificates   ← destructive: every certificate becomes VMCA-signed, custom chains are lost
  • Expiry is checked in the vSphere Client (Administration > Certificates > Certificate Management) and raised by the vCenter certificate-expiry alarms; VMCA-issued certificates are valid for two years by default, and 9.1 adds auto-renewal windows. Trusted roots are added with dir-cli trustedcert publish, not with option 8.
  • Expired STS signing certificate: services fail to start with SSO/token errors; follow the Broadcom KB for replacing the STS certificate before anything else.

Inventory and connectivity

  • Hosts disconnected: vpxa on the host must reach vpxd on 443 and vpxd must reach the host on 902 (heartbeats, UDP) — check tail /var/log/vpxa.log on the host and vpxd.log on vCenter for the host name.
  • vSphere Client slow or blank: restart vmware-vsphere-ui, clear the browser cache, check /storage/log; large inventories need the correct appliance size.
  • Backups: VAMI file-based backup to SFTP/NFS/SMB/HTTPS on a schedule is the supported vCenter protection in VVF (vCenter HA is not part of a VVF design and is unsupported under VCF).

Key Takeaways

  • service-control --status/--restart and vmon-cli manage the vCenter services; disk-full on /storage/log, /storage/db or /storage/core is the most common cause of a sick vCenter — purge rotated logs/dumps or grow the disk, never the live data.
  • Database recovery is a file-based restore (VAMI) or a support case; deleting /storage/db/vpostgres destroys the inventory, SSO configuration and licensing state.
  • Certificate Manager lives in /usr/lib/vmware-vmca/bin; option 8 resets every certificate to VMCA-signed (destructive). Check expiry in the vSphere Client; IWA is removed in vCenter 9.0 — use AD over LDAPS or identity federation.

vSAN Troubleshooting in VVF#

vSAN troubleshooting in a VVF estate (ESA by default on 9.0 greenfield, OSA on carried-over hardware).

Health first

  • vSphere Client > Cluster > Monitor > vSAN > Skyline Health: categories cover hardware compatibility, network, physical disk, data, cluster, capacity utilisation, performance service, stretched-cluster and online health; every finding links to a KB. Re-test after a fix.
  • VCF Operations also surfaces vSAN health, capacity and performance alerts and the Diagnostics Findings signatures.
esxcli vsan cluster get                         # membership, leader/backup/agent role, cluster UUID
esxcli vsan health cluster list                  # health tests and status from the host
esxcli vsan network list                          # vSAN VMkernel interfaces per host
esxcli vsan storage list                          # claimed devices (ESA storage pool / OSA disk groups)
esxcli vsan debug object health summary get       # object health counts
esxcli vsan debug resync summary get              # bytes left to resync, ETA
esxcli vsan debug object list --uuid=<object>     # placement of one object
vmkping -I vmk2 <peer-vsan-ip> -s 1472 -d         # vSAN network reachability and MTU

Resync and rebuilds

  • Monitor > vSAN > Resyncing Objects shows components, bytes left, ETA and the reason (rebuild, policy change, rebalance, evacuation). The default object repair timer is 60 minutes after a component goes absent; a decommissioned (failed) component rebuilds immediately.
  • Slow resync: check network saturation on the vSAN VMkernel, host CPU, and whether multiple hosts are in maintenance mode. If a VM stops responding because of resync latency, the 9.0 docs say resync IOPS can be throttled and point to KB 326830 — a PowerCLI procedure (vSAN 6.7.x/7.0.x) that notes the GUI throttle option was removed; no Resyncing Objects throttle control is documented for 9.0. Treat any throttle as temporary and record it.
  • Slack space: keep the Operations Reserve and Host Rebuild Reserve (ESA/OSA capacity reserve settings) enabled where supported so rebuilds have room; vSAN warns at the configured thresholds and stops new writes when the datastore is full.

Device failures and replacement

  • OSA: a failed cache device takes its disk group offline — replace the device, recreate the disk group, let the resync run. A failed capacity device is handled by component rebuild on remaining devices.
  • ESA: devices belong to a single storage pool per host; remove the failed device from the pool (Cluster > Configure > vSAN > Disk Management), replace it, claim the new one. Check the ESA device compatibility (NVMe TLC on the ReadyNode list).
  • Always use Maintenance Mode with the right data evacuation option: Ensure accessibility (fastest, FTT degraded for the duration), Full data migration (for permanent removal or long maintenance), No data migration (only when every object has another copy and you accept the risk).

Capacity

  • Watch Monitor > vSAN > Capacity: usage breakdown, reserved capacity, what-if for policies. Deduplication exists on OSA only; ESA compression is per storage policy.
  • Out of space: delete or move data, add capacity (hosts or devices), or relax a storage policy (e.g. RAID-1 → RAID-5 on ESA at 6+ hosts) and let the resync complete.

Performance

  • Turn on the Performance Service (Cluster > Configure > vSAN > Services); then Monitor > vSAN > Performance shows cluster/host/VM/disk metrics, and I/O Insight records per-VM traces. VCF Operations adds trend and capacity analytics. The old vSAN Observer (RVC) is not a 9.0 tool.

Common issues

IssueSymptomRemediation
Quorum/partitionHosts show different cluster members; objects inaccessibleFix the vSAN network (VLAN, MTU, multicast is not required since 6.6 — unicast), confirm esxcli vsan cluster get shows one leader; rejoin isolated hosts
Absent componentsObjects non-compliant, resync pendingWait for the repair timer or repair now from Skyline Health; check the host or device that went absent
Slow resyncResync ETA grows, VM latency risesReduce concurrent maintenance, check network saturation, consider temporary resync throttling (KB 326830, PowerCLI)
Device media errorsHealth alerts on a disk, DDH (dying disk handling) eventsReplace the device; review vendor firmware; check Skyline Health hardware tests
Capacity thresholdCapacity alarm, policy non-complianceAdd capacity or relax policy; keep reserves enabled; delete unneeded snapshots
Witness unreachable (2-node/stretched)Site disaster tolerance warningsVerify witness traffic VMkernel, routing to the third site, witness appliance version matches the cluster

Key Takeaways

  • Start at Cluster > Monitor > vSAN > Skyline Health and re-test after each fix; on the host, esxcli vsan cluster get / health cluster list / debug object and resync summaries give the same picture without the UI.
  • Resync: 60-minute repair timer for absent components, immediate rebuild on failed ones; keep the capacity reserves enabled; resync throttling is a temporary lever via KB 326830 (PowerCLI), not a Resyncing Objects control.
  • Device replacement differs by architecture — OSA cache device failure takes the disk group offline, ESA devices leave the storage pool individually; choose the maintenance-mode evacuation option deliberately.

VM Troubleshooting#

Virtual machine troubleshooting.

VM will not power on

  • Resource issues: HA admission control or a reservation that no host can satisfy (check the power-on task error and the cluster's failover capacity); insufficient datastore space for swap or thin growth.
  • Lock files: a VM's files are locked by the host that owns it; a stale lock after a host crash blocks power-on elsewhere. Identify the owner before touching anything:
vmkfstools -D /vmfs/volumes/<datastore>/<vm>/<vm>.vmdk      # prints the lock owner (MAC address of the host) and mode
# If the owner host is dead, a vSphere HA restart or rebooting the owner releases the lock; do not delete .lck files on a live VM
ls -la /vmfs/volumes/<datastore>/<vm>/                       # look for -delta.vmdk, .lck, orphaned files

Snapshots (KB 318825 — best practices)

  • Supported maximum is 32 snapshots in a chain; keep 2–3 at most and no snapshot older than 72 hours. Long chains slow I/O and make consolidation long and risky.
  • Needs Consolidation warning: run Snapshots > Consolidate; if it fails for space, free datastore space or Storage vMotion the VM; check vmware.log for the error.
  • vSAN snapshots are vSAN objects with their own storage-policy overhead; delete old snapshots before capacity alarms, not after.

Disk corruption and datastore checks

voma -m vmfs -f check -d /vmfs/devices/disks/naa.xxxx:1    # VMFS metadata check (datastore must be unmounted or quiesced)
vmkfstools -e /vmfs/volumes/<ds>/<vm>/<vm>.vmdk             # check a descriptor/chain for consistency
vmkfstools -i source.vmdk clone.vmdk -d thin                 # clone a disk out of a suspect chain

vmkfstools -D shows lock ownership; it is not a corruption check — VOMA is. Guest file-system errors inside the VM are fixed with the guest OS tools after the virtual disk is healthy.

Guest and tools

  • VMware Tools: out of date or not running stops guest heartbeats (HA VM monitoring), customisation and quiescing; upgrade from Content Library or the guest.
  • Hung guest: check vmware.log in the VM folder, the host's vmkernel.log for the VM's world, and esxcli vm process list / esxcli vm process kill --type=soft --world-id=<id> as a last resort.

VM networking

  • Port group VLAN, security policy and teaming must match the uplinks; esxcli network vm list and esxcli network vm port list -w <world-id> show the port the VM uses.
  • MTU: a guest at 1500 over a 9000 uplink is normal; problems arise only when a path hop has a smaller MTU than the sender. Test with ping -f/-M do from the guest and vmkping -d from the host.
  • Use the VM's vNIC type consistently (VMXNET3), and check the DVS port state (blocked, link down) in the vSphere Client when a VM loses connectivity after vMotion.

Key Takeaways

  • Power-on failures are resources or locks: check admission control and datastore space, then vmkfstools -D to find the lock owner — never delete lock files on a live VM.
  • Snapshots: 32 maximum per chain, keep 2–3, none older than 72 hours (KB 318825); consolidate from the UI and free space first when it fails.
  • VOMA checks VMFS metadata; vmkfstools -e checks a disk chain; guest MTU 1500 over a 9000 uplink is fine — only a smaller hop in the path breaks traffic.

Standalone vSphere Support Tool Chest (Support Track)#

Standalone vSphere support tool chest.

esxtop (KB 344099 and the esxtop reference)

esxtop                      # interactive: c CPU, m memory, d disk adapter, u disk device, v disk VM, n network, i interrupts, p power
esxtop -b -d 5 -n 60 > esxtop.csv     # batch mode: 60 samples at 5 s for offline analysis (perfmon / esxplot)
PanelCounterMeaning / threshold
CPU%USEDCPU time used by the world
CPU%RDYtime ready to run but waiting for a pCPU — sustained > 10% per vCPU indicates contention
CPU%CSTPco-stop for multi-vCPU VMs — > 3–5% means too many vCPUs
CPU%MLMTDtime held back by a CPU limit — subtract from %RDY when limits are set
CPU%WAITblocked or idle (includes %IDLE); %SWPWT is swap wait
MemoryMCTLSZballoon size — non-zero means memory pressure
MemorySWCUR / SWR/s / SWW/scurrent swap and swap read/write rates — any sustained swap is a problem
MemoryZIP/s, UNZIP/smemory compression activity
DiskDAVG/cmddevice (array) latency — sustained > 10–15 ms is a storage problem
DiskKAVG/cmdkernel latency — should be ~0–1 ms; higher means queueing in the host
DiskGAVG/cmdguest-observed latency = DAVG + KAVG
DiskQUED, ACTVqueued and active commands; QUED > 0 with high KAVG = queue depth limit
DiskREADS/s, WRITES/sIOPS; MBREAD/s and MBWRTN/s are throughput
NetworkMbTX/s, MbRX/sthroughput per vmnic (physical) or per VM port
Network%DRPTX, %DRPRXdropped packets — receive drops point at a saturated vNIC/VM, transmit drops at the uplink

Logs and bundles

  • Host: vm-support (see the Support Fundamentals block) — -w <datastore> sets where the bundle is written, -p -d <secs> -i <interval> adds performance snapshots; the bundle contains /var/log, /etc, esxcli dumps and core dumps when present.
  • vCenter: Export System Logs from the vSphere Client or Create Support Bundle in the VAMI; vc-support.sh in the shell. Bundles contain the vCenter service logs (vpxd, vsphere-ui, SSO, vmafd/vmdir, vpostgres logs — not the database itself).
  • Case upload: Broadcom Support Portal case attachments or VCF Operations Log Assist (9.0).

Other tools

  • vSphere Client Monitor tabs: Tasks and Events, Performance charts (real-time and historical), Issues and Alarms, vSAN views.
  • esxcli system syslog to confirm remote logging; VCF Operations for Logs Explore Logs for cross-host search and the vSphere content pack dashboards.
  • PowerCLI for repeatable collection: Get-VMHost | Get-EsxTop, Get-Log, Get-VMHostDiagnosticPartition.
  • VCF Operations: Diagnostics Findings, Troubleshooting Workbench (correlates alerts, events and metrics around an object and a time; the exam blueprint calls it the Observability Workbench).

Key Takeaways

  • esxtop thresholds: %RDY > 10% per vCPU (minus %MLMTD) = CPU contention, %CSTP > 3–5% = too many vCPUs, MCTLSZ/SWCUR > 0 = memory pressure, DAVG > 10–15 ms = storage, KAVG > 1 ms = host queueing, %DRPRX/%DRPTX = network drops.
  • vm-support -w <datastore> writes the host bundle, -p adds performance data; vCenter bundles come from Export System Logs / VAMI / vc-support.sh and contain service logs, not the database.
  • Correlate in VCF Operations (Troubleshooting Workbench, Diagnostics Findings) and VCF Operations for Logs (Explore Logs) before opening a case; attach bundles via Log Assist or the support portal.

VVF Upgrade, Clusters & License Management (Objectives 5.2–5.4)#

Blueprint Objectives 5.2, 5.3, 5.4 — VVF Upgrade, Clusters & Licensing

VVF Upgrade — vSphere to VVF 9.0 Conversion (Obj 5.2)

Converting a standalone vSphere environment to VVF involves licensing changes and optional component additions. Unlike VCF, VVF conversion does NOT require SDDC Manager or VCF Instance fleet management (VVF does use VCF Operations).

Conversion Workflow:

├── Pre-Conversion Assessment:
│   ├── Verify hardware in the Broadcom Compatibility Guide for ESX 9.0
│   ├── Convergence minimums (VCF 9.0.1 docs p.1344): VVF 9.0.0 needs vCenter 9.0 and ESX 9.0; VVF 9.0.1/9.0.2 accept vCenter 8.0 U3 and ESX 8.0 U1 or later without upgrading first
│   ├── Clusters still on vLCM baselines must move to images (baselines are removed in vCenter 9.0, p.226)
│   └── Check the ESX upgrade path from the current version in the interoperability data (p.1791)
├── Convergence (VCF Installer):
│   ├── Deploy the VCF Installer, run the vSphere Foundation convergence workflow for the existing vCenter and hosts; it deploys VCF Operations (or deploy VCF Operations manually)
│   └── Existing vCenter, ESX and Aria Operations at older versions are upgraded manually to 9.0.x first where the target version requires it, then licensed from VCF Operations
├── ESX upgrade:
│   ├── vLCM image remediation, host by host in maintenance mode, after compliance check and pre-check
│   └── Revert a failed host with Shift+R at boot (KB 316592)
├── vCenter upgrade:
│   └── 9.0 offers reduced-downtime upgrade, integrated with vCenter HA deployments (p.80); take a file-based backup first
├── vSAN:
│   └── Update the on-disk format after all hosts run 9.0 for the full feature set (p.1454); a format upgrade cannot be rolled back
├── Licensing (no keys in 9.0):
│   ├── Add the vSphere Foundation (cores) license and the vSAN (TiB) add-on in VCF Operations, from the VCF Business Services console (connected) or a license file (disconnected) (p.1457, p.1463)
│   └── Assign to the vCenter instance; its hosts are licensed through that assignment
└── Post-Conversion Validation:
    ├── All hosts on ESX 9.0 and compliant with the cluster image
    ├── vSAN Skyline Health all green
    ├── VMs running without issues
    └── VCF Operations shows the vCenter licensed and usage reporting scheduled (at least every 180 days)

Troubleshooting VVF Upgrade:

├── ESXi upgrade fails on host: Check vLCM pre-check results, verify VCG compliance
├── Convergence pre-check fails: compare vCenter/ESX versions with the minimums for the target VVF version (p.1344)
├── vSAN disk format upgrade hangs: Ensure no active resyncs, verify all hosts upgraded
└── Hosts unlicensed after upgrade: check the license is assigned to the vCenter instance in VCF Operations and that usage reporting is current

VVF Clusters — Create, Scale, Import (Obj 5.3)

Creating VVF Workload Clusters:

├── vCenter → New Cluster → Name, DRS (Fully Automated), HA (enabled)
├── Add hosts: Right-click cluster → Add Host → enter ESXi IP, credentials
├── Enable vSAN: Cluster → Configure → vSAN → Turn On → select disks
├── Configure vDS: Add hosts to distributed switch, migrate vmk adapters
├── Storage policies: Create or assign default vSAN storage policy (FTT, RAID)
└── Validation: vSAN Health Check green, DRS recommendations applied

Scaling VVF — Add/Remove Clusters:

├── Add Cluster: vCenter → Datacenter → New Cluster → configure DRS/HA/vSAN
├── Remove Cluster: Migrate all VMs out → Enter maintenance mode on all hosts → Remove
├── Cross-cluster vMotion: Supported between clusters with shared storage or vSAN
├── Resource sharing: Clusters are independent — no cross-cluster DRS
└── Considerations: Each cluster requires minimum 3 hosts for vSAN quorum

Scaling Cluster — Add/Remove ESXi Hosts:

├── Add Host:
│   ├── Right-click cluster → Add Host → enter IP/credentials
│   ├── Host auto-joins vSAN (disks claimed automatically based on disk group config)
│   ├── vDS configuration applied automatically (if host added to vDS)
│   ├── DRS rebalances workloads to include new host
│   └── Validation: vSAN Health Check, host compliance in vLCM
├── Remove Host:
│   ├── Enter Maintenance Mode: Right-click host → Enter Maintenance Mode
│   ├── vSAN data migration options:
│   │   ├── Ensure Accessibility: Move data to maintain accessibility (fastest)
│   │   ├── Full Data Migration: Evacuate all data to remaining hosts (slowest, safest)
│   │   └── No Data Migration: Host removed without data migration (data loss risk)
│   ├── DRS evacuates VMs to other hosts
│   ├── After maintenance mode: Right-click → Remove from Cluster
│   └── Warning: Removing host below minimum (3 for vSAN) breaks quorum

Import Existing vSphere Environment as VVF Cluster:

├── Scenario: Customer has standalone vSphere and purchases VVF license
├── Process:
│   ├── Apply VVF license to existing vCenter and hosts
│   ├── If vSAN already in use: Enable VVF vSAN features via license
│   ├── If no vSAN: Enable vSAN on existing cluster (claim local disks)
│   ├── Optional: Deploy VCF Operations for monitoring (included in VVF)
│   └── Optional: Enable Supervisor for Kubernetes workloads
├── No re-deployment needed: VVF is a licensing overlay, not a separate product
└── Validation: License status shows VVF entitlement, all features available

License Management in VVF 9.0 (Obj 5.4)

Licensing model (VCF 9.0 What's New — Licensing):

├── Licenses are managed in VCF Operations, connected to the VCF Business Services console (vcf.broadcom.com) or in disconnected mode
├── Two licenses — "VMware Cloud Foundation (cores)" and "VMware vSAN (TiBs)"; vSphere Foundation follows the same pattern (cores plus vSAN TiB); all subscriptions pool into one license file that is imported once
├── Assign the license to vCenter instances in VCF Operations; every host and component connected to that vCenter is licensed from the assignment
├── Usage reporting: license usage must be submitted from VCF Operations every 180 days, or hosts disconnect from vCenter (connected mode reports automatically)
└── No Standard/Advanced/Enterprise tiers and no 30-day grace period in 9.0; the 90-day VCF Operations evaluation applies only until a license is added

Troubleshooting:

├── Hosts disconnecting with a licensing message → check the last usage submission date in VCF Operations > Licensing and submit or reconnect
├── vCenter shows unlicensed hosts after adding a cluster → confirm the vCenter instance has the license assignment in VCF Operations and that the host is connected to that vCenter
├── Capacity warnings → licensed cores versus physical cores (minimum 16 counted per CPU) and vSAN TiB versus consumed capacity; add subscription or reduce footprint
└── Disconnected mode → export the usage report from VCF Operations and upload it in the Business Services console within the 180-day window

Holodeck Lab Note: the nested lab runs on evaluation licensing in VCF Operations; practise the Licensing view, the usage report export and the vCenter assignment screen rather than entering a key in vCenter.

Key Takeaways

  • Exam: vSphere → VVF conversion = licensing overlay + optional component additions. No SDDC Manager needed. Upgrade path: ESXi 7.x → 8.x → 9.0 (sequential).
  • Exam: Add host to vSAN cluster → disks claimed automatically, DRS rebalances. Remove host → Enter Maintenance Mode with data migration option (Ensure Accessibility vs Full Migration).
  • Exam: VVF licensing = per-core subscription (16-core minimum per CPU) plus vSAN TiB, managed and assigned to vCenter instances in VCF Operations, usage reported every 180 days; no edition tiers in 9.0.

VVF Network Troubleshooting — VDS and VSS (Objective 5.7)#

Blueprint Objective 5.7 — VVF Networks: VDS and VSS Troubleshooting

VDS (vSphere Distributed Switch) Troubleshooting

VDS Health Check:

├── vCenter → Networking → Distributed Switch → Health Check
├── Checks: VLAN trunking, MTU, NIC teaming consistency
├── Common findings:
│   ├── VLAN not trunked on physical switch → VMs on that VLAN can't communicate
│   ├── MTU mismatch → jumbo frames fail (vSAN, vMotion degraded)
│   └── NIC teaming inconsistency → some hosts have different uplink config
└── Resolution: Fix physical switch config or vDS port group settings

VDS Port Group Issues:

├── VM on wrong port group → no network connectivity
│   ├── Check: VM → Edit Settings → Network adapter → verify port group name
│   ├── Fix: Change port group assignment, verify VLAN ID matches intent
│   └── Bulk check: vCenter → Networking → vDS → Ports → filter by VM
├── Port blocked by security policy:
│   ├── MAC Address Changes: Reject → VMs with custom MACs blocked
│   ├── Forged Transmits: Reject → nested ESXi (Holodeck!) cannot communicate
│   ├── Promiscuous Mode: Reject → packet capture VMs get no data
│   └── Fix: vDS → Port Group → Edit → Security → set to Accept as needed
├── VLAN mismatch:
│   ├── Port group VLAN ID doesn't match physical switch trunk
│   ├── Diagnosis: vmkping works on management VLAN but fails on VM VLAN
│   ├── Fix: Verify physical switch trunk allows the VLAN (show vlan brief)
│   └── VDS PVLAN: Private VLAN misconfiguration isolates VMs unintentionally
└── NIOC (Network I/O Control) Contention:
    ├── Symptom: vSAN or vMotion slow despite healthy physical links
    ├── Cause: NIOC shares too low for the affected traffic type
    ├── Diagnosis: esxtop → 'n' (network) → check %DRPTX (and %DRPRX) per vmnic
    ├── Fix: vDS → Resource Allocation → increase shares for affected pool
    └── Production: Reserve bandwidth for vSAN (100 shares) and vMotion (50+)

VSS (vSphere Standard Switch) Troubleshooting:

VSS is host-local — issues must be diagnosed per-host:

├── NIC teaming failover not working:
│   ├── Check: esxcli network vswitch standard policy failover get -v vSwitch0
│   ├── Verify: Active/standby NIC assignment matches intent
│   ├── Fix: esxcli network vswitch standard policy failover set -v vSwitch0 -a vmnic0,vmnic1
│   └── Test: Disconnect physical cable → verify failover to standby NIC
├── Port group VLAN not set:
│   ├── Check: esxcli network vswitch standard portgroup list
│   ├── Fix: esxcli network vswitch standard portgroup set -p "VM Network" -v 100
│   └── Verify: VMs on port group can reach hosts on VLAN 100
├── VSS to VDS Migration Issues:
│   ├── Pre-migration: Ensure VDS has same port groups as VSS
│   ├── Migration order: Create VDS port groups → migrate vmk adapters → migrate VM network adapters → remove VSS
│   ├── Risk: Losing management connectivity if vmk0 migration fails
│   ├── Rollback: Keep VSS active until VDS verified, then remove VSS
│   └── Holodeck: Nested ESXi requires Forged Transmits = Accept on outer VDS

VMkernel Adapter Troubleshooting:

├── vmk0 (Management) down:
│   ├── Symptom: Host disconnected from vCenter
│   ├── Cause: IP conflict, VLAN change, NIC failure
│   ├── Fix: DCUI console (direct console) → reset management network
│   └── Alternatively: esxcli network ip interface ipv4 set -i vmk0 -I <ip> -N <mask> -t static
├── vmk (vMotion) not working:
│   ├── Symptom: vMotion fails with "timed out" or "migration error"
│   ├── Check: vmkping -I vmk1 <dest-vmk1-ip> (verify reachability)
│   ├── Common cause: vMotion vmk on wrong TCP/IP stack
│   ├── Fix: Remove and re-add vmk with correct TCP/IP stack = "vmotion"
│   └── MTU: If using jumbo frames, verify: vmkping -d -s 8972 <dest-ip>
├── vmk (vSAN) issues:
│   ├── Symptom: vSAN health shows "Network partition detected"
│   ├── Check: vmkping -I vmk2 <other-host-vmk2-ip>
│   ├── Common cause: VLAN mismatch, MTU mismatch, physical switch issue
│   ├── Fix: Verify VLAN ID and MTU (9000) match on all hosts and physical switches
│   └── vSAN inter-node latency must be less than 1 ms RTT (stretched clusters: less than 1 ms within each site, 5 ms RTT between data sites) — VCF 9.0.1 docs p.1608

Holodeck Lab Note: In Holodeck nested environments, VSS-to-VDS migration is a common lab exercise. Remember: outer ESXi VDS must have Forged Transmits = Accept for nested ESXi networking to work. If Holodeck VMs lose connectivity after VDS migration, check the outer VDS security policy first.

Key Takeaways

  • Exam: VDS Health Check validates VLAN trunking, MTU, and NIC teaming consistency. Run before troubleshooting network issues.
  • Exam: VSS is host-local — each host configured independently. VDS is centralized. Know the migration path: create VDS → migrate vmk → migrate VM NICs → remove VSS.
  • Exam: vmkping with -d -s 8972 tests jumbo frames (MTU 9000). vSAN needs less than 1 ms RTT between hosts (5 ms between stretched-cluster data sites, p.1608).

VCF Operations, Operations for Logs & Orchestrator Troubleshooting (Objectives 5.8–5.9)#

Blueprint Objectives 5.8, 5.9 — VCF Operations & Orchestrator in VVF

VCF Operations Configuration & Troubleshooting (Obj 5.8)

VCF Operations (included in VVF licensing) provides monitoring, alerting, and compliance. Support engineers must know how to troubleshoot it.

VCF Operations Configuration Issues:

├── Adapter Connection Failures:
│   ├── Symptom: vCenter adapter shows "Not Collecting" or "Error"
│   ├── Causes: Incorrect credentials, certificate changed, network blocked
│   ├── Diagnosis: Administration > Integrations → open the vCenter account → re-test the connection
│   ├── Fix: Re-enter credentials, accept new certificate, verify port 443 open
│   └── Log: /var/log/vmware/vcops/vcops-adapter-vCenter.log
├── Data Collection Gaps:
│   ├── Symptom: Missing metrics for some hosts/VMs
│   ├── Causes: Host disconnected from vCenter, adapter overwhelmed
│   ├── Diagnosis: Check adapter collection state, verify host connectivity
│   ├── Fix: Restart adapter, increase collection interval for large environments
│   └── Verification: Environment → select object → verify recent data points
├── Performance Issues:
│   ├── Symptom: VCF Operations UI slow, reports timing out
│   ├── Causes: Undersized deployment, too many objects, disk I/O bottleneck
│   ├── Diagnosis: SSH to VCF Ops → top (check CPU/memory), df -h (check disk)
│   ├── Fix: Scale up (add data nodes) or scale out (additional collectors)
│   └── Threshold: >4,000 objects on Small → upgrade to Medium deployment
└── Certificate Issues:
    ├── Symptom: "Certificate not trusted" warnings in UI
    ├── Fix: Replace self-signed cert with CA-signed, or add root CA to trust store
    └── Location: Administration → Appliance → SSL Certificate → Import

Troubleshoot VVF Using VCF Operations (Obj 5.8):

├── Performance Troubleshooting Workflow:
│   ├── Start: Dashboard → identify problem area (high CPU host, slow VM)
│   ├── Drill-down: Click object → Metrics tab → review time-series data
│   ├── Correlate: Check related objects (host → VMs on host → datastore)
│   ├── Root cause: Use "Root Cause Analysis" widget for automated correlation
│   └── Remediate: Follow recommendation attached to alert
├── Capacity Troubleshooting:
│   ├── Capacity dashboard: Shows days remaining before resource exhaustion
│   ├── What-if analysis: Model adding/removing hosts or VMs
│   ├── Reclaimable resources: Identify oversized VMs (unused CPU/memory)
│   └── Action: Right-size recommendations with before/after comparison
└── Health Troubleshooting:
    ├── Health badges: Green/Yellow/Red per object
    ├── Risk badges: Predicted issues (capacity, performance trending)
    ├── Efficiency badges: Resource waste identification
    └── All three badges together provide holistic object health assessment

VCF Operations for Logs Troubleshooting (Obj 5.8):

├── Log Ingestion Issues:
│   ├── Symptom: Logs not appearing from ESXi hosts
│   ├── Causes: Syslog not configured, firewall blocking UDP 514
│   ├── Diagnosis: On ESXi host → esxcli system syslog config get (verify loghost)
│   ├── Fix: esxcli system syslog config set --loghost=udp://<ops-logs-ip>:514
│   │   Follow with: esxcli network firewall ruleset set -e true -r syslog
│   │   Then: esxcli system syslog reload
│   └── Test: logger -t TEST "Test message" → verify in Explore Logs
├── Search & Analysis:
│   ├── Explore Logs: Full-text search across all ingested logs
│   ├── Field Extraction: Use regex to extract structured fields from unstructured logs
│   ├── Group By: Source, severity, custom field → identify patterns
│   ├── Saved Queries: Save frequently used searches for quick access
│   └── Alerts: Create log-based alerts (e.g., alert on "SCSI sense code" in vmkernel.log)
├── Troubleshooting Workbench (Infrastructure Operations > Troubleshooting Workbench, VCF 9.0.1 docs p.3912):
│   ├── Start from an alert (Infrastructure Operations > Alerts → alert → TROUBLESHOOT) or search for an object or metric
│   ├── Shows potential evidences for the chosen scope and time window: events, property changes and anomalous metrics
│   ├── Sessions are not saved after logout (they move to Recent Searches)
│   └── Use case: find what changed around the time a problem started
└── Disk Space Management:
    ├── Log retention: Default 30 days, configurable
    ├── Disk full: Log ingestion stops silently if disk full
    ├── Fix: Increase disk, archive old logs to NFS, reduce retention period
    └── Monitor: Administration → System Monitor → Disk Usage

Generate Log Bundle / Log Assist (Obj 5.8):

├── Log Bundle:
│   ├── VCF Operations: Administration > Control Panel > Support Bundles tile > Add (VCF 9.0.1 docs p.4243-4244)
│   ├── Light (last 24 hours of logs) or Full (all logs and configuration files); select nodes or the whole cluster
│   ├── Downloaded as a ZIP
│   └── Attach to the Broadcom support case (for vCenter and ESX components, Log Assist transfers bundles directly)
├── Log Assist (Obj 5.8 — generate a log bundle and upload it):
│   ├── Prerequisites (VCF 9.0.1 docs p.3910-3912): VCF Operations licensing in Connected Mode; Log Assist activated on a unified cloud proxy (Administration > Control Panel > Cloud Proxies > Activate Log Assist; Classic collectors are not supported); objects assigned under Infrastructure Operations > Configurations > Log Assist Assignment. A VVF deployment has no cloud proxy by default (Table 812, p.1265), so add one first
│   ├── Workflow: select inventory objects → generate support bundles (one task per object) → authenticate to the Broadcom Support Portal → enter the support case ID → transfer
│   ├── Bundles are per component support bundles, not a hand-picked list of log files
│   └── Troubleshoot: case ID validation, Log Assist Assignment of the object, cloud proxy reachability (443 to the components), retry timed-out transfers

VCF Operations Orchestrator — Configuration & Workflows (Obj 5.9)

Orchestrator Configuration (DOCUMENTED, VCF 9.0.1 docs):

├── Deployment: a separate VCF Operations orchestrator appliance deployed from its OVA — 4 CPUs, 12 GB memory, 200 GB disk; do not reduce the default memory, the server needs at least 8 GB free (p.1306). The "embedded" orchestrator exists only inside VCF Automation (p.939), which is not a VVF component — in VVF it is always the standalone appliance.
├── Access: Orchestrator client at https://<orchestrator-fqdn>/orchestration-ui/ (appliance health: https://<orchestrator-fqdn>/vco/api/healthstatus)
├── Authentication: register vSphere as the authentication provider on the appliance as root — vracli vro authentication set -p vsphere -u <username> -hn <vsphere_hostname> --fqdn <deployment_fqdn> (p.1310); log in with a vSphere SSO user in the orchestrator admin group
├── Integration: vCenter plug-in — Library > Workflows > "Add a vCenter Server instance"; VCF Operations sees the appliance only through the Management Pack for Orchestrator (p.4537), not as a built-in page
└── Content: plug-in workflow libraries under Library > Workflows, plus your own workflows and actions in the designer

Running Workflows & Actions:

├── Plug-in Workflow Library (vCenter plug-in — finding the support-relevant workflows):
│   ├── Find workflows in Library > Workflows by name or tag — for example the vcenter and configuration tags locate "Add a vCenter Server instance" (p.6709)
│   ├── Host workflows named in the docs: "Enter maintenance mode" and "Reprovision a host by a simple reboot" (p.6654)
│   ├── Virtual machine, cluster and datastore workflows (snapshots, power, clone, relocate, DRS/HA): search by name and record the grouping and tags you see on your appliance
│   └── Scheduled library runs: any library workflow can be scheduled (Activity > Scheduled, p.6618)
├── Custom Workflow Creation:
│   ├── Designer: schema editor in the orchestrator client (workflow elements, bindings, inputs/outputs)
│   ├── Scripting: JavaScript natively; Python, Node.js and PowerShell runtimes for polyglot scripts and actions (p.6626)
│   ├── Inputs: workflow input parameters bound at run time (VM, host, strings, numbers, arrays)
│   ├── Scheduling: Run > Schedule; scheduled workflows appear under Activity > Scheduled (p.6618)
│   └── Event-driven runs: through the orchestrator REST API or a caller such as VCF Operations (Management Pack for Orchestrator "Run Action")
├── Troubleshooting Orchestrator:
│   ├── Workflow fails: Activity > Workflow Runs → select the run → Logs and Variables tabs
│   ├── Connection issues: re-run "Add a vCenter Server instance" / check the vCenter plug-in inventory under Inventory
│   ├── Resource limits: Python/Node.js/PowerShell scripts can exceed default memory and timeout limits — raise them per the docs (p.6626 area)
│   └── Permissions: the vCenter account registered with the plug-in must hold the privileges the workflow needs
└── Support Use Cases:
    ├── Scheduled hygiene: weekly snapshot report or cleanup across all VMs
    ├── Batch operations: apply one change to many hosts or VMs in a single run
    ├── Guided remediation: wrap a documented fix in a workflow with inputs so it is repeatable and logged
    └── Self-healing: restart a service or re-enter maintenance mode from a VCF Operations alert via the Management Pack action (with safeguards)

Holodeck Lab Note: VCF Operations Orchestrator can be deployed in Holodeck as the standalone appliance (4 vCPU / 12 GB / 200 GB) — the same appliance lab vvf-support-13 uses. Create a simple workflow: "Collect support bundle from all ESXi hosts" — this demonstrates the automation capability and is directly exam-relevant. Schedule the workflow to run weekly as a proactive support measure.

Key Takeaways

  • Exam: VCF Operations adapter issues — test connection, check credentials, verify port 443. Logs in /var/log/vmware/vcops/.
  • Exam: Ops for Logs — syslog config on ESXi (esxcli system syslog), firewall ruleset 'syslog' must be enabled. Explore Logs for full-text search.
  • Exam: Troubleshooting Workbench (p.3912; exam guide: Observability Workbench) = potential evidence — events, property changes, anomalous metrics — around an object and a time window. Launch-in-context for correlation. Log Assist = support bundles generated through a Log Assist-enabled cloud proxy and transferred to the Broadcom case ID.
  • Exam: Orchestrator — standalone appliance (4 vCPU/12 GB/200 GB, p.1306), vSphere as authentication provider (vracli vro authentication set -p vsphere), plug-in workflow library under Library > Workflows, custom workflows in the designer (JavaScript; Python/Node.js/PowerShell runtimes), scheduling under Activity > Scheduled, runs under Activity > Workflow Runs.

Exam Mapping: 2V0-18.25 — vSphere Foundation 9.0 Support

  • 5.1 Deployment troubleshooting — ESXi installation, vCenter deployment, initial config
  • 5.2 VVF Upgrade — vSphere to VVF 9.0 conversion, ESXi/vCenter/vSAN upgrade paths
  • 5.3 VVF Clusters — create, scale (add/remove hosts/clusters), import existing vSphere
  • 5.4 License Management — assign from VCF Operations, entitlements (cores + vSAN TiB), usage reporting, expiry/renewal
  • 5.5 Compute — ESXi boot issues, hardware compatibility, driver/firmware, esxtop
  • 5.6 Storage — vSAN Skyline Health, disk replacement, capacity management, Performance Service / I/O Insight
  • 5.7 Networks — VDS/VSS troubleshooting, VMkernel adapters, NIOC, MTU/jumbo frames
  • 5.8 VCF Operations — adapter config, dashboards, Ops for Logs, Observability Workbench, Log Assist/bundle
  • 5.9 VCF Operations Orchestrator — configuration, workflows, actions, scheduling

Labs in This Section

📝 Quiz — VVF 9.0 Support

0/50 correct

Section 1 — Troubleshooting Methodology

Q1
A VVF customer calls with a 'VM is slow' complaint. Which FIRST step aligns with structured troubleshooting methodology?
  • Immediately reboot the ESXi host
  • Delete the VM snapshot
  • Open a Broadcom support case
  • Gather specific symptoms, affected scope, and timeline before forming any hypothesis
Structured troubleshooting starts with gathering specific symptoms, affected scope, and timeline before forming a hypothesis. Rebooting, opening cases, and deleting snapshots are premature actions before understanding the problem.
Q2
Which VMware resource provides authoritative step-by-step remediation articles for known VVF issues?
  • Reddit r/vmware
  • Random blog posts
  • Vendor marketing whitepapers
  • Broadcom/VMware Knowledge Base (KB) articles
Broadcom/VMware KB articles are the authoritative source for step-by-step remediation. Reddit, blogs, and whitepapers are unofficial and may contain outdated or incorrect information.
Q3
Which feature in VVF is NOT present (compared to VCF) and therefore OUT OF SCOPE for a VVF support call?
  • ESXi host troubleshooting
  • vCenter service failures
  • NSX overlay and HCX migration troubleshooting
  • vSAN troubleshooting (when add-on is licensed)
NSX overlay and HCX migration are VCF-only features, out of scope for VVF support. ESXi troubleshooting, vSAN (when licensed), and vCenter failures are all in VVF scope.
Q4
A support engineer needs the most commonly requested diagnostic bundle for a single ESXi host. Which command should be run?
  • ps -ef
  • df -h
  • vm-support
  • esxcli diag
vm-support is the standard command for collecting an ESXi host-level diagnostic bundle. esxcli diag, df, and ps are not bundle collection tools.
Q5
For initial triage of a cluster-wide issue, which vCenter tab is MOST valuable for a time-correlated view of events?
  • Tags & Custom Attributes
  • Profiles tab
  • Permissions tab
  • Tasks & Events history sorted by time range
Tasks & Events history sorted by time range provides a time-correlated view across cluster operations. Permissions, Tags, and Profiles tabs serve different purposes.

Section 2 — vSphere Compute Issues

Section 3 — Storage Issues

Section 4 — Networking Issues

Section 5 — Upgrade and Migration Issues

🃏 Flashcards — VVF 9.0 Support

49 cards
Card 1 of 49
VVF Support Scope
The intentionally narrow troubleshooting surface for VVF engineers: vSphere, vCenter, VCF Operations, and optional vSAN — with NSX, HCX, and VCF Automation explicitly out of scope. Recognizing scope avoids blind alleys in unrelated products. Issues outside scope are escalated to the appropriate VCF or networking team.
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.