vSphere Foundation 9.0 Support (2V0-18.25)

VVF Support Fundamentals#

Support Tiers & Escalation

Tier 1 (L1):

Initial diagnostics, knowledge base search, basic troubleshooting (reboot, restart services)

Tier 2 (L2):

Deep log analysis, vLCM/vSAN health diagnostics, CLI commands, potential code bugs

Tier 3 (L3):

VMware engineering, code fixes, custom escalation handling

Escalation criteria:

PAC (Priority & Severity) scoring — Critical (P1 = production down), High (P2 = feature unavailable), Medium (P3 = degraded), Low (P4 = cosmetic)

VMware Knowledge Base Navigation

KB search strategy:

1. Error message exact phrase search: "PSEC error 12345"

2. Component-based: "vSAN quorum loss"

3. Version-specific: "ESXi 9.0 coredump"

KB Portal: https://knowledge.broadcom.com

• Full-text search
• Filter by product version, component
• Solutions tagged with "2V0-18" = Support exam relevant
• Rate solution usefulness (feedback improves ranking)

Skyline Advisor for Support Diagnostics

Proactive health analysis; can be manually triggered for troubleshooting:

Data collection:

vCenter logs, vSAN health snapshots, performance metrics (24-hour rolling window)

Cloud comparison:

Configuration compared against millions of VMware deployments; flags anomalies

Export for support case:

vCenter UI → Skyline → Download diagnostic bundle → Attach to VMware support ticket

Limitations:

Requires internet connectivity and VMware account. Air-gapped environments use local export + manual upload to support portal.

Support Bundle Collection & Compression

Collect diagnostic data for support case

Method 1: vCenter UI

vCenter UI → Administration → Bundles → Request Support Bundle

(Bundles all logs from vCenter, ESXi hosts, vSAN if present)

Method 2: CLI

ssh root@esxi_host
esxcli system supportrequest interactive
(Creates file /tmp/esx-support-bundle-timestamp.tgz)

Method 3: vCenter SSH shell

• ssh root@vcsa_hostname
• /usr/lib/vmware-vpx/bin/logbundle.sh
• (Creates compressed tarball in /tmp/vc_support_bundle-*.tgz)

Typical size: 2-5GB (vCenter + 3x ESXi)

Compression: gzip or 7-zip for upload (<500MB after compression)

Log File Structure & Key Paths

vCenter VCSA

• /var/log/vmware/vpxd/vpxd.log # Main vCenter process log
• /var/log/vmware/vpxd/vpxd-alert.log # Critical events
• /var/log/vmware/vpxd/vpxd.*.log # Rotated logs (100MB each)
• /var/log/vmware/vsan/vsanmgmtd.log # vSAN management daemon
• /var/log/vmware/vpostgres/pg_log/ # Database logs (if enabled)
• /var/log/messages # System messages (OS-level)
• /var/log/audit/audit.log # Authentication/authorization

Access:

ssh root@vcsa_hostname
cd /var/log/vmware/vpxd
tail -f vpxd.log | grep "error\|warning"

ESXi Hosts

• /var/log/hostd.log # vSphere hostd daemon (HA, heartbeat)
• /var/log/vmkernel.log # Kernel-level events (storage, network, HA)
• /var/log/vobd.log # Virtual Object Base daemon (vSAN agent)
• /var/log/boot.log # System boot messages
• /var/log/esxupdate.log # vLCM/patch application log

Real-time monitoring:

ssh root@esxi_hostname
tail -f /var/log/vmkernel.log | grep "vsan\|scsi\|iscsi"

Log Levels & Verbosity

Common error codes for diagnosis:

vpxd:
Error codes 100-999 (authentication), 1000-1999 (inventory), 2000-2999 (clustering)
vmkernel:
"PSEC" (platform security), "NvmXX" (NVMe driver), "xvmfs" (vSAN filesystem)
vobd:

"Cluster" (quorum loss), "Disk" (media errors), "Network" (partition detection)

Key Takeaways

• VVF scope: vSphere (ESXi + vCenter) + vDS + vSAN basic + no NSX/NSX-T; support exam focuses on foundational troubleshooting.
• Tier 1 escalation: knowledge base search, reboot/restart services; Tier 2 = log analysis, vSAN health diagnostics.
• Support case workflow: vSphere-specific issues escalate to L2 after initial triage; vSAN problems require full cluster snapshot and CMMDS data.

ESXi Host Troubleshooting#

Boot Issues & POST Failures

BIOS POST error codes:

Check OEM documentation (Dell: codes 0xc0-0xef in diagnostic panel). Common: Memory ECC errors (0xc1), USB initialization failure (0xc4).

ESXi boot hang:

BIOS → Security settings may block boot. Disable Secure Boot initially, verify ESXi can load, then re-enable with signed images.
UEFI vs BIOS:
Modern servers UEFI only; ESXi 9.0 supports both. Check firmware mode: ssh root@host → esxcli hardware firmware get
No console output:
Serial console (COM1/COM2) may be required if video disabled in BIOS. Configure vCenter → Hosts → Edit → Settings → Advanced → Logging

Hardware Compatibility & Driver/Firmware Mismatches

VCG (VMware Compatibility Guide) Verification:

Check HW compatibility before deployment

1. Visit VMware Compatibility Guide: hcl.vmware.com
2. Select ESXi version (9.0)
3. Filter by server model (Dell PowerEdge, HP ProLiant, Lenovo)
4. Verify ALL components listed:

• Server model certified
• RAID controller firmware version approved
• NIC driver version matches VCG recommendation
• SSD/HDD models on supported list

On host, verify actual versions:

ssh root@esxi_host
esxcli hardware pci list | grep -i "raid\|ethernet\|hba"
esxcli system module get -m bnx2x | grep Version  # Broadcom driver example

Driver & Firmware Update Procedure

Critical: Firmware updates often require host reboot; coordinate with HA/vMotion:

RAID controller firmware:

Cannot update online. Evacuate host → Enable Maintenance Mode → Update via iDRAC/iLO web UI → Reboot → Exit Maintenance Mode

NIC firmware:

Updates via vLCM (preferred) or manual ethtool command. SSH method: ethtool -f ethX path/to/firmware.bin (requires driver reload)

SSD firmware:

Some models support in-place firmware updates; others require power-cycle. Verify with manufacturer before attempting.

Never update firmware during I/O operations. Evacuate host to Maintenance Mode, wait for all VMs to vMotion, then proceed.

Network Troubleshooting: vmkping, esxcli network

Network diagnostics on ESXi

vmkping -I vmk_portgroup_name destination_ip -c 4

Example: vmkping -I "Management Network" 10.0.1.1 -c 4

Good: 4 replies, <5ms latency, 0% loss

Bad: "ICMP host unreachable" = network isolation

Check network stack binding:

esxcli network ip netstack list

Output: default, vsan (if vSAN enabled), vmotion (if configured)

Interface status:

esxcli network nic list

Fields: Name (vmnic0), LinkStatus (Up/Down), Speed, Duplex (Full)

VLAN verification:

esxcli network vswitch dvport list | grep -i "VLAN ID"

Should show VLAN tagging for each port group

TCP/IP stack check (for vMotion/vSAN isolation):

esxcli network ip interface ipv4 get

List all vmk interfaces and their assigned TCP/IP stacks

Storage Path Failover & PSP/SATP Issues

PSP (Path Selection Policy) determines which LUN path active; SATP (Storage Array Type Plugin) detects array type:

View active paths:

esxcli storage nmp device list

Output: Device naa.6xxxxx, Status OK, SATP= VMW_SATP_SPC, PSP=VMW_PSP_RR (Round-Robin)

Failing path diagnosis:

esxcli storage nmp path list

Status: active, enabled, disabled, dead

If status=dead: Physical SAN switch issue, cable fault, or array port disabled

Manual path enable (if incorrectly disabled):

esxcli storage nmp path set -p vmnic2:0:0:0 -e true

Change PSP (requires LUN not in use):

esxcli storage nmp device set -d naa.6xxxxx -P VMW_PSP_FIXED -A vmnic0:0:0:0

Coredump Configuration & Analysis

ESXi coredumps (purple screen) essential for deep debugging:

Dump target:

Must be NFS, ISCSI, or local disk ≥1GB. Network-based preferred (ISCSI/NFS) to preserve on-host data.

Configuration:

vCenter → Host → Manage → System → Dump Partition → Select active partition

Trigger:

Coredump generated on kernel panic (ESXi crash); automatically uploaded to /scratch/log/vmkernel-coredump* on restart

Analysis:

VMware analyzer tool can process dumps (contact support for analysis); manual inspection requires gdb + symbol files

Enable network coredump (ISCSI):

ssh root@esxi_host
esxcli system coredump network set -v vmk0 -i 192.168.1.200 -o 6500

Verify:

esxcli system coredump network get

Should show "Configured: true"

Test dump (initiates panic, host will reboot):

esxcli system coredump network test

Don't run in production without HA configured!

Key Takeaways

• Host connectivity: vmkping, network path redundancy, and NIC speed/duplex are first diagnostics; check esxcli network commands.
• Driver/firmware mismatches: VCG (VMware Compatibility Guide) is source of truth; RAID adapter firmware must match supported version.
• Coredump troubleshooting: ESXi coredumps require proper disk/network target; PSOD BugCheck codes identify hardware or driver issues.

vCenter Troubleshooting#

VCSA Shell Access & Service Management

SSH into VCSA:

ssh root@vcsa_hostname

Default shell: /bin/bash

Check resource usage:

• top -b -n 1 | head -20 (CPU/memory usage)
• df -h | grep -i "/" (disk space)
• free -m (RAM summary)

Service-control commands:

service-control --status --all

Lists all vCenter services and their state (stopped/started)

Restart specific service (if hanging):

• service-control --restart vmafdd (Authentication service)
• service-control --restart vpxd (vCenter server process)
• service-control --restart vsphere-ui (Web UI)

Stop all services (maintenance, DB backup):

service-control --stop --all

Database Issues (VPOSTGRES)

vCenter uses embedded PostgreSQL (VPOSTGRES) as default database:

Database location:

/storage/db/vpostgres (VMFS datastore on VCSA)

Disk full issue:

If /storage runs out, VCSA becomes read-only. Check: df -h /storage. Free space = 20% of partition minimum.

Corruption diagnosis:

vpxd.log shows errors like "Database connection timeout" or "Query failed". Restart service-control --restart vpxd often resolves.

Backup/restore:

vCenter appliance backup includes DB; restore via vCenter management interface (vCenter >> Backup & Restore)

Check VPOSTGRES status:

service-control --status vpostgres

Should show "running"

Verify DB size:

du -sh /storage/db/vpostgres

If database corrupted (rare), initiate recovery:

NOTE: This erases DB and re-initializes. Requires restore from backup.

• service-control --stop vpostgres
• rm -rf /storage/db/vpostgres/*
• service-control --start vpostgres

(vCenter will re-initialize empty DB; re-register ESXi hosts)

SSO & Identity Source Problems

vCenter uses vSphere Single Sign-On (SSO) for authentication to multiple vCenter instances:

SSO domain:

Default "vsphere.local" (local users) or external AD domain. Users login with domain\username

Trust issues:

If AD authentication fails, check: vCenter >> Administration >> Users and Groups >> Active Directory. Verify domain controller reachable (ping, LDAP port 389).

Certificate mismatch:

AD cert validation fails if root CA cert not in VCSA trusted store. Import: vCenter appliance shell >> certificate-manager (option 8 = trust external certs)

Certificate Management & Renewal

vCenter certificates expire after 365 days; renewal required before expiration:

Check certificate expiry:

ssh root@vcsa_hostname

Option 1: Check via appliance shell

/usr/lib/vmware-vpx/bin/certificate-manager

Menu: Option 6 = View certificate expiry

Option 2: OpenSSL command-line

openssl x509 -in /etc/vmware-vpx/ssl/rui.crt -text -noout | grep "Not After"

Renewal process (auto-renewal if enabled):

vCenter >> Administration >> System Configuration >> View Certificate >> Issue New Certificate

(vCenter will generate new CSR, sign, and restart services)

Manual renewal (if auto-renewal disabled):

/usr/lib/vmware-vpx/bin/certificate-manager

Option 1 = Generate self-signed cert

Option 2 = Generate CSR (submit to CA, import signed cert)

vCenter HA Troubleshooting (Not included in VVF 9.0 - Single VCSA standard)

Note:

VVF does not include vCenter HA (requires 3 VCSA + passive nodes). Single VCSA standard. For HA, upgrade to VCF.

Inventory Sync Issues

vCenter loses connection to ESXi hosts (often temporary):

Host disconnected:

vCenter UI shows "Disconnected" state. Check: vCenter >> Hosts → Right-click host → Reconnect

Network isolation:

vCenter cannot reach host IP. Verify network connectivity: From vCenter SSH session, ping host_ip

Slow inventory sync:

>500 VMs or >50 hosts may cause slow inventory updates. Monitor: vCenter >> Performance >> vCenter Server >>> "Inventory Service Duration" metric

Key Takeaways

• VCSA filesystem: /storage/db and /storage/log must have sufficient free space; event database vacuum reclaims space for old events.
• Service management: service-control --status/stop/start commands control vpxd, vmdir, sso services; order matters for HA clusters.
• SSO and certificates: vmdir replication breaks if vmafd certificate expires; renew via certificate-manager before expiry window.

vSAN Troubleshooting in VVF#

vSAN Health Check Dashboard

Automated multi-category diagnostic; run before/after changes:

vSAN Health Check categories:

1. Cluster: Quorum, unicast agent connectivity, split-brain prevention
2. Storage: Disk claims, controller firmware, encryption certs
3. Network: Latency <5ms, jitter <2ms, MTU validation (9000), packet loss <0.1%
4. Limits: Max objects (vSAN limit = millions; practical ~100k), max hosts (64)
5. vSAN FSA: Namespace health (if File Services enabled)
6. Physical disk: Media errors, predictive failures, rebuild status

Access:

vCenter UI → Clusters → Cluster_Name → vSAN → Health Check

Result: Green (pass), Yellow (warning), Red (fail)

Disk Replacement Procedures

vSAN automatically rebalances after disk removal/replacement:

Non-disk-group disk failure:

Single capacity disk fails → vSAN marks disk offline. Data resyncs to remaining capacity disks (called "autosync"). Replacement: Power down host → Replace disk → vSAN auto-rebalances.
Cache device failure:
Whole disk group goes offline (cache is critical). Replacement: Mark disk group for rebalance (vCenter UI) → Replace SSD → vSAN re-creates disk group with existing capacity disks.
Timeline:
Resync rate depends on workload and policy (default: FTT=1 = 2-way mirror). 2TB disk → 8-24 hours typical resync.

Monitor resync progress:

vCenter UI → Cluster → vSAN → Performance

Graph: "Resync throughput (MB/s)" shows active rebalance rate

CLI verification:

ssh root@esxi_host
esxcli vsan cluster get | grep "Resync"

If resync too slow (impacting IOPS):

vCenter UI → Cluster → Edit → vSAN → Advanced options
→ Set "resync_throttle" to higher value (1-64, default=1)
→ Higher = faster resync but increased I/O contention

Capacity Management & Out-of-Capacity Recovery

When vSAN cluster approaches 100% full, write I/O fails (hard stop):

Warning threshold:

70% capacity triggered yellow alarm; 85% = red alarm

Prevention:

Scale: Add new disk groups or hosts; OR shrink: Delete large VMs, compress snapshots, enable dedup

Emergency recovery (100% full):

Cluster becomes read-only. Remove non-essential VMs to restore capacity, then resume operations.

vSAN Observer Tool (Performance Diagnostics)

Deep performance analysis; installed on ESXi host as rvcagent module:

Enable vSAN Observer on each host:

vCenter UI → Host → Manage → vSAN Observer → Enable Observer Agent

Access observer web UI:

https://esxi_hostname:8307

Login: root / esxi_root_password

Graphs: Latency (per disk/component), throughput, IOPS, queue depth

Typical use: Identify slow disk, component bottlenecks before they impact VMs

vSAN Common Issues & Remediation

• Issue
• Symptom
• Remediation
• Quorum loss
• 3-node cluster down to 1-2 nodes; all objects unavailable
• Restore failed host; if hardware issue, rebuild/replace
• Network partition (split-brain)
• 2+ host groups isolated; conflicting writes possible
• Restore network link; vSAN uses witness quorum to pick canonical side
• Slow resync
• VM latency spikes during rebuild
• Increase resync_throttle parameter; temporarily pause other I/O
• Dedup stuck/hung
• High CPU but no capacity freed; no dedup progress
• Disable dedup, restart vobd, re-enable dedup after recovery
• Disk media errors
• Physical disk reports SMART errors; failing status
• Replace disk; vSAN resyncs data off failing disk

Run vSAN Health Check and interpret results

Key Takeaways

• vSAN health check: cluster, storage, network, and limits categories must all pass; failures prevent new object placement.
• Disk replacement: non-disk-group disk failure triggers autosync; cache device failure requires disk group rebalance (no autosync).
• Capacity thresholds: 70% yellow alarm, 85% red alarm, 100% = read-only mode; delete snapshots or migrate VMs to restore capacity.

VM Troubleshooting#

VM Won't Start: Lock Files & Resource Issues

VM registration fails or fails to power on; often transient on vSAN:

vmx lock file:

.vmx.lck file prevents concurrent access; persists after crash. Remedy: SSH to ESXi → Find /vmfs/volumes -name "*.vmx.lck" → Delete stale lock (verify no other vCenter has VM open first)
Insufficient CPU:
Admission control denies power-on if reserved CPU not available. vCenter UI → Cluster → Edit → HA → Admission control → Relax % reserved or disable (not recommended production)
Insufficient memory:
Same as CPU. Memory balloons existing VMs to free space. Check vCenter → Cluster → Resource allocation → Available memory
Storage not accessible:
VM's primary disk on offline datastore. Check: vCenter → VM → Edit Settings → Check datastore status

Snapshot Issues & Cleanup

Snapshots consume storage; old snapshots can degrade I/O:

Snapshot chain depth:

Limit to 5-10 snapshots. >20 snapshots = high consolidation risk and slow delta tracking.

Delete snapshot hung:

Consolidation process ongoing; can take hours for large disks. Monitor: vCenter → Recent Tasks. Interrupt only if hung >24 hours.

Snapshot undo log full:

vSAN snapshots use dedup; undo log (transient data for rollback) can fill. Free space via snapshot deletion or VM migration to different vSAN datastore.

VMDK Corruption Detection & Recovery

VMDK corruption symptoms:

• VM I/O errors in guest OS logs
• vmkernel.log shows "Device or resource busy" errors
• vCenter VMDK properties show "inconsistent snapshot state"

Detection method 1: vCenter UI

VM → Edit Settings → Check all disks listed + snapshot chain

Detection method 2: CLI check

ssh root@esxi_host
vmkfstools -D /vmfs/volumes/datastore_name/VM_name/VM_disk.vmdk

Output: Analyzes disk metadata; flags corruption

Recovery: Restore from snapshot or backup

If no snapshot: Use vmkfstools -J check-uuid to verify disk header

Guest OS Hangs & vCenter Response

Guest unresponsive:

vCenter can't reach VMware Tools. Remedy: Force power off (hard shutdown) via vCenter UI → VM → Power → Power Off. Last resort; may corrupt guest filesystem.

vSphere HA response:
If HA enabled and host hangs, HA isolates host + restarts VM on different host (configurable restart policy).

Debugging:

Guest logs rarely accessible during hang; power down + check guest disk via offline disk analysis or backup recovery.

VM Networking Issues

VM cannot reach network:

Check: vCenter → VM → Edit → Network adapter → Port group assignment. Verify port group VLAN tagged correctly on vDS.
Packet loss/latency:
Check: VM vNIC MTU (default 1500) matches physical uplink MTU (should be 9000 for vSAN traffic). Mismatch causes fragmentation/retransmission.
vDS port blocked:
Port security filtering (vDS profile) may block traffic. Check: vCenter → Distributed Switch → Port group → Edit → Security settings (MAC spoofing, forged transmits, etc.)

Key Takeaways

• VM lock files: .vmx.lck persists after crashes; identify and delete stale locks after verifying no other vCenter manages the VM.
• Snapshot issues: snapshot chain depth >20 degrades I/O; old snapshots consume storage; consolidation can take hours on large disks.
• Network issues: vNIC MTU mismatch (1500 vs 9000) causes packet loss; check port group VLAN tagging and distributed switch configuration.

Standalone vSphere Support Tool Chest (Support Track)#

Supporting standalone vSphere (without SDDC Manager) requires different tools and approaches. Troubleshooting happens at host level, using esxtop, logs, and CLI diagnostics.

esxtop Command Reference (Complete)

esxtop is the primary performance troubleshooting tool on ESXi. Master its columns and filtering:

esxtop Basic Usage:
---

Access: SSH to ESXi host → esxtop command

Interactive mode: Realtime metrics, updateable

Batch mode: esxtop -b -n 10 > esxtop.csv (10 iterations to file)

Key Screens (press key to switch):

• c = CPU view
• m = Memory view
• d = Disk view
• n = Network view
• v = VM view
• p = Power view
• f = Frame buffer/video

---

CPU View (press 'c'):

Column Explanation:

PCPU# Physical CPU number (0-15 for 16-core host)

• %USED Cumulative CPU time in use (should be < 100 for good perf)
• %RUN Waiting to run (idle time = 100 - %RUN)
• %WAIT Waiting for I/O (disk/network blocked)
• %CSTP Co-stop (time VM threads couldn't run simultaneously)

Example:

PCPU0: %USED=85, %RUN=5, %WAIT=10

Interpretation: CPU busy (85%), some I/O wait (10%), headroom (15%)

Problem: %USED=100, %WAIT=0

  → CPU maxed out, no I/O waiting (compute-bound bottleneck)

---

Memory View (press 'm'):

Column Explanation:

• PMEM Physical memory installed
• VMKMEM VMkernel memory usage
• HOSTCACHES vSAN, NFS cache
• VMPHYSMEM VM physical memory
• BALLOONED Memory reclaimed by balloon driver
• SWAPPED VM memory paged to disk (very bad)
• MEMCTL Compression (memory zipping)

Example:

PMEM=256GB, VMKMEM=12GB, VMPHYSMEM=200GB, BALLOONED=5GB, SWAPPED=0

Interpretation: 200GB allocated to VMs, 5GB reclaimed, no swapping (healthy)

Problem: SWAPPED>0

  → VMs running out of memory, host is paging (performance impact)

---

Disk View (press 'd'):

Column Explanation:

• DISK Disk name (vmhba0:C0:T0:L0 = LUN address)
• READ Read I/Os per second
• WRITE Write I/Os per second
• READS/s Read throughput (MB/s)
• WRITES/s Write throughput (MB/s)
• LATENCY Average read latency (ms)
• QREAD Queue depth for read (0-N)
• QWRITE Queue depth for write

Example:

DISK vmhba1:vSAN, READ=5000, WRITE=1000, LATENCY=3ms

Interpretation: High I/O rate, healthy latency, vSAN active

Problem: LATENCY>20ms

  → Storage bottleneck (slow disk, oversubscribed LUN, or array issue)

---

Network View (press 'n'):

Column Explanation:

• VMNIC# Virtual NIC (vmnic0, vmnic1)
• PKTTX/s Packets transmitted
• PKTRX/s Packets received
• MBITTX/s Megabits transmitted (Mbps)
• MBITRX/s Megabits received (Mbps)
• %DROPTX Dropped packets on transmit
• %DROPRX Dropped packets on receive

Example:

vmnic0: MBITRX/s=8000, MBITTX/s=6000 (on 10GbE NIC)
Interpretation: ~8Gbps input, 6Gbps output (80% utilization, good)

Problem: %DROPRX>0.1%, %DROPTX>0.1%

  → Network congestion (packets dropped, increase NIC count or speed)

---

Interactive Commands:

• f = Add/remove columns (customize view)
• s = Sort by column
• g = Select group (VMs, hosts, etc.)
• e = Toggle between realtime and reset stats
• q = Quit
• h = Help

vm-support Bundle & vSphere Log Bundle

• vm-support (Command-Line Tool):
• ---
• Purpose: Collect diagnostic data for troubleshooting
• Used by: ESXi hosts, vCenter (via UI)

Execution (on ESXi host, SSH):

vm-support [-w] /tmp/host-support-bundle.tgz

Options:

-w = Include performance stats (large, takes time)
(default: lightweight, just logs and config)

Output:

/tmp/host-support-bundle.tgz (~100-500MB)

Contains:

• /var/log/*.log (VMkernel, hostd, vpxa logs)
• /proc/vmware/config (ESXi configuration)
• Hardware inventory, network config
• Performance snapshots (if -w flag)

Usage: Upload to TAC for analysis

• ---
• vSphere Log Bundle (UI Method):
• ---

Access: vCenter UI → Administration → System Configuration → Logs

Download: Generate support bundle (Web UI)

Output: support-bundle.zip (~200MB)

Contains:

• All vCenter logs (catalina, vmware-vcloud-director, etc.)
• vCenter configuration
• Event history (last 100k events)
• Performance statistics
• DB snapshot (vCenter database state)

Timeline: Bundle can take 10-15 minutes to generate

---

Extracting & Analyzing:

• tar -xzf host-support-bundle.tgz
• ls -la log/
• tail -1000f log/vmkernel.log (follow live logs)
• grep ERROR log/vmkernel.log (search errors)

vSphere Skyline Advisor (Proactive Health)

Skyline Advisor (Cloud-Connected Recommendation Engine):

---
Purpose: Analyze infrastructure, recommend improvements

Data: Anonymized logs + config → VMware cloud

Output: Risk assessments, security recommendations, upgrade readiness

Setup (in vCenter):

  1. vCenter UI → Administration → Skyline Advisor

2. Collect and upload configuration (14 days of data)
3. Recommendations generated within hours

Example Recommendations:

• "Driver firmware outdated: update recommended"
• "Disk array latency trending high: plan upgrade"
• "vSphere 8.x EOL in 6 mo

Key Takeaways

• esxtop metrics: %RDY (CPU ready) >10% = overallocation, %CSTP (co-stop) >5% = vCPU topology issue, SWCUR >0 = memory crisis.
• Log files: vmkernel.log on ESXi, vpxd.log on vCenter; grep for ERROR/WARN and timestamps near reported issue.
• vm-support bundle: lightweight logs + hardware config; -w flag adds performance stats; upload to support for TAC analysis.

VVF Upgrade, Clusters & License Management (Objectives 5.2–5.4)#

Blueprint Objectives 5.2, 5.3, 5.4 — VVF Upgrade, Clusters & Licensing

VVF Upgrade — vSphere to VVF 9.0 Conversion (Obj 5.2)

Converting a standalone vSphere environment to VVF involves licensing changes and optional component additions. Unlike VCF, VVF conversion does NOT require SDDC Manager or VCF Operations Manager.

Conversion Workflow:

├── Pre-Conversion Assessment:
│   ├── Verify hardware in VMware Compatibility Guide (VCG) for vSphere 9.0
│   ├── Confirm vCenter version ≥ 8.0 U3 (minimum for VVF 9.0 upgrade path)
│   ├── Inventory current licenses: vSphere Standard/Enterprise → VVF license tier
│   ├── Check vSAN readiness: existing vSAN cluster compatible with 9.0?
│   └── Network: vDS recommended (vSS to vDS migration may be needed)
├── ESXi Upgrade Path:
│   ├── ESXi 7.x → 8.x → 9.0 (sequential upgrades required, no skip)
│   ├── Method: vLCM image-based upgrade (cluster-wide, rolling)
│   ├── Pre-check: vLCM compatibility check before remediation
│   ├── Rollback: Snapshot ESXi boot bank before upgrade (automatic)
│   └── Duration: 30-45 minutes per host (rolling via DRS/HA)
├── vCenter Upgrade:
│   ├── VCSA 8.x → 9.0 (migration-based upgrade — new appliance deployed)
│   ├── Stage 1: Deploy new VCSA 9.0 appliance alongside existing
│   ├── Stage 2: Migrate data (inventory, history, configuration)
│   ├── Downtime: 15-30 minutes (vCenter unavailable during switchover)
│   └── Rollback: Revert to old VCSA from snapshot
├── vSAN Upgrade:
│   ├── vSAN on-disk format upgrade: triggered after all hosts on 9.0
│   ├── Cannot downgrade vSAN disk format once upgraded
│   ├── New features unlocked: ESA (if NVMe-only), improved dedup
│   └── Duration: 5-15 minutes per host (online, non-disruptive)
├── License Application:
│   ├── Remove old vSphere licenses from vCenter
│   ├── Add VVF subscription license key
│   ├── Assign to cluster and hosts
│   └── Verify: vCenter → Administration → Licensing → all hosts show VVF license
└── Post-Conversion Validation:
    ├── All hosts on ESXi 9.0 and reporting correct version
    ├── vSAN health check all green
    ├── VMs running without issues
    └── vLCM baseline shows no pending updates

Troubleshooting VVF Upgrade:

├── ESXi upgrade fails on host: Check vLCM pre-check results, verify VCG compliance
├── vCenter migration fails at Stage 2: Check network (new VCSA must reach old vCenter DB)
├── vSAN disk format upgrade hangs: Ensure no active resyncs, verify all hosts upgraded
└── License mismatch after upgrade: Re-apply VVF license key, verify entitlement count

VVF Clusters — Create, Scale, Import (Obj 5.3)

Creating VVF Workload Clusters:

├── vCenter → New Cluster → Name, DRS (Fully Automated), HA (enabled)
├── Add hosts: Right-click cluster → Add Host → enter ESXi IP, credentials
├── Enable vSAN: Cluster → Configure → vSAN → Turn On → select disks
├── Configure vDS: Add hosts to distributed switch, migrate vmk adapters
├── Storage policies: Create or assign default vSAN storage policy (FTT, RAID)
└── Validation: vSAN Health Check green, DRS recommendations applied

Scaling VVF — Add/Remove Clusters:

├── Add Cluster: vCenter → Datacenter → New Cluster → configure DRS/HA/vSAN
├── Remove Cluster: Migrate all VMs out → Enter maintenance mode on all hosts → Remove
├── Cross-cluster vMotion: Supported between clusters with shared storage or vSAN
├── Resource sharing: Clusters are independent — no cross-cluster DRS
└── Considerations: Each cluster requires minimum 3 hosts for vSAN quorum

Scaling Cluster — Add/Remove ESXi Hosts:

├── Add Host:
│   ├── Right-click cluster → Add Host → enter IP/credentials
│   ├── Host auto-joins vSAN (disks claimed automatically based on disk group config)
│   ├── vDS configuration applied automatically (if host added to vDS)
│   ├── DRS rebalances workloads to include new host
│   └── Validation: vSAN Health Check, host compliance in vLCM
├── Remove Host:
│   ├── Enter Maintenance Mode: Right-click host → Enter Maintenance Mode
│   ├── vSAN data migration options:
│   │   ├── Ensure Accessibility: Move data to maintain accessibility (fastest)
│   │   ├── Full Data Migration: Evacuate all data to remaining hosts (slowest, safest)
│   │   └── No Data Migration: Host removed without data migration (data loss risk)
│   ├── DRS evacuates VMs to other hosts
│   ├── After maintenance mode: Right-click → Remove from Cluster
│   └── Warning: Removing host below minimum (3 for vSAN) breaks quorum

Import Existing vSphere Environment as VVF Cluster:

├── Scenario: Customer has standalone vSphere and purchases VVF license
├── Process:
│   ├── Apply VVF license to existing vCenter and hosts
│   ├── If vSAN already in use: Enable VVF vSAN features via license
│   ├── If no vSAN: Enable vSAN on existing cluster (claim local disks)
│   ├── Optional: Deploy VCF Operations for monitoring (included in VVF)
│   └── Optional: Enable Supervisor for Kubernetes workloads
├── No re-deployment needed: VVF is a licensing overlay, not a separate product
└── Validation: License status shows VVF entitlement, all features available

License Management in VVF (Obj 5.4)

Assigning Licenses:

├── vCenter → Administration → Licensing → Licenses → Add License Key
├── Select hosts or cluster → Assign License
├── VVF license covers: vSphere (vCenter + ESXi) + vSAN + VCF Operations
├── Per-core licensing: VVF license is per physical CPU core
└── Verification: Each host shows correct license with features enabled

License Entitlements:

├── VVF Standard: vSphere Standard + vSAN Standard + basic VCF Operations
├── VVF Advanced: vSphere Advanced (DRS, SIOC) + vSAN Advanced (dedup, stretched)
├── VVF Enterprise: Enterprise editions + Skyline Advisor + Workload Optimization
├── Feature Comparison:
│   ├── Standard: No DRS (manual placement only), basic vSAN, no stretched cluster
│   ├── Advanced: Full DRS/HA, vSAN dedup/compression, stretched cluster
│   └── Enterprise: All features + proactive support + advanced analytics
└── Check entitlements: vCenter → Administration → Licensing → Features per license

License Expiry & Renewal:

├── Subscription: VVF licenses expire annually (subscription model)
├── Notification: vCenter alerts 90/60/30 days before expiry
├── Grace Period: 30 days after expiry (functionality retained, warning banners)
├── Expired: After grace period, new VM power-on blocked, management restricted
├── Renewal: Apply new license key → old key automatically replaced
└── Audit: Export license usage report for compliance (vCenter → Licensing → Export)

Holodeck Lab Note: In Holodeck, VVF license keys can be evaluation licenses (60-day trial). Practice license assignment workflow by adding/removing evaluation keys. Observe the behavior when license expires — VMs continue running but new operations are blocked.

Key Takeaways

• Exam: vSphere → VVF conversion = licensing overlay + optional component additions. No SDDC Manager needed. Upgrade path: ESXi 7.x → 8.x → 9.0 (sequential).
• Exam: Add host to vSAN cluster → disks claimed automatically, DRS rebalances. Remove host → Enter Maintenance Mode with data migration option (Ensure Accessibility vs Full Migration).
• Exam: VVF license covers vSphere + vSAN + VCF Operations. Per-core subscription. Standard/Advanced/Enterprise tiers. 30-day grace after expiry.

VVF Network Troubleshooting — VDS and VSS (Objective 5.7)#

Blueprint Objective 5.7 — VVF Networks: VDS and VSS Troubleshooting

VDS (vSphere Distributed Switch) Troubleshooting

VDS Health Check:

├── vCenter → Networking → Distributed Switch → Health Check
├── Checks: VLAN trunking, MTU, NIC teaming consistency
├── Common findings:
│   ├── VLAN not trunked on physical switch → VMs on that VLAN can't communicate
│   ├── MTU mismatch → jumbo frames fail (vSAN, vMotion degraded)
│   └── NIC teaming inconsistency → some hosts have different uplink config
└── Resolution: Fix physical switch config or vDS port group settings

VDS Port Group Issues:

├── VM on wrong port group → no network connectivity
│   ├── Check: VM → Edit Settings → Network adapter → verify port group name
│   ├── Fix: Change port group assignment, verify VLAN ID matches intent
│   └── Bulk check: vCenter → Networking → vDS → Ports → filter by VM
├── Port blocked by security policy:
│   ├── MAC Address Changes: Reject → VMs with custom MACs blocked
│   ├── Forged Transmits: Reject → nested ESXi (Holodeck!) cannot communicate
│   ├── Promiscuous Mode: Reject → packet capture VMs get no data
│   └── Fix: vDS → Port Group → Edit → Security → set to Accept as needed
├── VLAN mismatch:
│   ├── Port group VLAN ID doesn't match physical switch trunk
│   ├── Diagnosis: vmkping works on management VLAN but fails on VM VLAN
│   ├── Fix: Verify physical switch trunk allows the VLAN (show vlan brief)
│   └── VDS PVLAN: Private VLAN misconfiguration isolates VMs unintentionally
└── NIOC (Network I/O Control) Contention:
    ├── Symptom: vSAN or vMotion slow despite healthy physical links
    ├── Cause: NIOC shares too low for the affected traffic type
    ├── Diagnosis: esxtop → 'n' (network) → check %DROPTX per vmnic
    ├── Fix: vDS → Resource Allocation → increase shares for affected pool
    └── Production: Reserve bandwidth for vSAN (100 shares) and vMotion (50+)

VSS (vSphere Standard Switch) Troubleshooting:

VSS is host-local — issues must be diagnosed per-host:

├── NIC teaming failover not working:
│   ├── Check: esxcli network vswitch standard policy failover get -v vSwitch0
│   ├── Verify: Active/standby NIC assignment matches intent
│   ├── Fix: esxcli network vswitch standard policy failover set -v vSwitch0 -a vmnic0,vmnic1
│   └── Test: Disconnect physical cable → verify failover to standby NIC
├── Port group VLAN not set:
│   ├── Check: esxcli network vswitch standard portgroup list
│   ├── Fix: esxcli network vswitch standard portgroup set -p "VM Network" -v 100
│   └── Verify: VMs on port group can reach hosts on VLAN 100
├── VSS to VDS Migration Issues:
│   ├── Pre-migration: Ensure VDS has same port groups as VSS
│   ├── Migration order: Create VDS port groups → migrate vmk adapters → migrate VM network adapters → remove VSS
│   ├── Risk: Losing management connectivity if vmk0 migration fails
│   ├── Rollback: Keep VSS active until VDS verified, then remove VSS
│   └── Holodeck: Nested ESXi requires Forged Transmits = Accept on outer VDS

VMkernel Adapter Troubleshooting:

├── vmk0 (Management) down:
│   ├── Symptom: Host disconnected from vCenter
│   ├── Cause: IP conflict, VLAN change, NIC failure
│   ├── Fix: DCUI console (direct console) → reset management network
│   └── Alternatively: esxcli network ip interface ipv4 set -i vmk0 -I <ip> -N <mask> -t static
├── vmk (vMotion) not working:
│   ├── Symptom: vMotion fails with "timed out" or "migration error"
│   ├── Check: vmkping -I vmk1 <dest-vmk1-ip> (verify reachability)
│   ├── Common cause: vMotion vmk on wrong TCP/IP stack
│   ├── Fix: Remove and re-add vmk with correct TCP/IP stack = "vmotion"
│   └── MTU: If using jumbo frames, verify: vmkping -d -s 8972 <dest-ip>
├── vmk (vSAN) issues:
│   ├── Symptom: vSAN health shows "Network partition detected"
│   ├── Check: vmkping -I vmk2 <other-host-vmk2-ip>
│   ├── Common cause: VLAN mismatch, MTU mismatch, physical switch issue
│   ├── Fix: Verify VLAN ID and MTU (9000) match on all hosts and physical switches
│   └── vSAN network requires <5ms latency, <1% packet loss

Holodeck Lab Note: In Holodeck nested environments, VSS-to-VDS migration is a common lab exercise. Remember: outer ESXi VDS must have Forged Transmits = Accept for nested ESXi networking to work. If Holodeck VMs lose connectivity after VDS migration, check the outer VDS security policy first.

Key Takeaways

• Exam: VDS Health Check validates VLAN trunking, MTU, and NIC teaming consistency. Run before troubleshooting network issues.
• Exam: VSS is host-local — each host configured independently. VDS is centralized. Know the migration path: create VDS → migrate vmk → migrate VM NICs → remove VSS.
• Exam: vmkping with -d -s 8972 tests jumbo frames (MTU 9000). vSAN requires <5ms latency, <1% packet loss on vmk2.

VCF Operations, Operations for Logs & Orchestrator Troubleshooting (Objectives 5.8–5.9)#

Blueprint Objectives 5.8, 5.9 — VCF Operations & Orchestrator in VVF

VCF Operations Configuration & Troubleshooting (Obj 5.8)

VCF Operations (included in VVF licensing) provides monitoring, alerting, and compliance. Support engineers must know how to troubleshoot it.

VCF Operations Configuration Issues:

├── Adapter Connection Failures:
│   ├── Symptom: vCenter adapter shows "Not Collecting" or "Error"
│   ├── Causes: Incorrect credentials, certificate changed, network blocked
│   ├── Diagnosis: Administration → Solutions → vSphere → Test Connection
│   ├── Fix: Re-enter credentials, accept new certificate, verify port 443 open
│   └── Log: /var/log/vmware/vcops/vcops-adapter-vCenter.log
├── Data Collection Gaps:
│   ├── Symptom: Missing metrics for some hosts/VMs
│   ├── Causes: Host disconnected from vCenter, adapter overwhelmed
│   ├── Diagnosis: Check adapter collection state, verify host connectivity
│   ├── Fix: Restart adapter, increase collection interval for large environments
│   └── Verification: Environment → select object → verify recent data points
├── Performance Issues:
│   ├── Symptom: VCF Operations UI slow, reports timing out
│   ├── Causes: Undersized deployment, too many objects, disk I/O bottleneck
│   ├── Diagnosis: SSH to VCF Ops → top (check CPU/memory), df -h (check disk)
│   ├── Fix: Scale up (add data nodes) or scale out (additional collectors)
│   └── Threshold: >4,000 objects on Small → upgrade to Medium deployment
└── Certificate Issues:
    ├── Symptom: "Certificate not trusted" warnings in UI
    ├── Fix: Replace self-signed cert with CA-signed, or add root CA to trust store
    └── Location: Administration → Appliance → SSL Certificate → Import

Troubleshoot VVF Using VCF Operations (Obj 5.8):

├── Performance Troubleshooting Workflow:
│   ├── Start: Dashboard → identify problem area (high CPU host, slow VM)
│   ├── Drill-down: Click object → Metrics tab → review time-series data
│   ├── Correlate: Check related objects (host → VMs on host → datastore)
│   ├── Root cause: Use "Root Cause Analysis" widget for automated correlation
│   └── Remediate: Follow recommendation attached to alert
├── Capacity Troubleshooting:
│   ├── Capacity dashboard: Shows days remaining before resource exhaustion
│   ├── What-if analysis: Model adding/removing hosts or VMs
│   ├── Reclaimable resources: Identify oversized VMs (unused CPU/memory)
│   └── Action: Right-size recommendations with before/after comparison
└── Health Troubleshooting:
    ├── Health badges: Green/Yellow/Red per object
    ├── Risk badges: Predicted issues (capacity, performance trending)
    ├── Efficiency badges: Resource waste identification
    └── All three badges together provide holistic object health assessment

VCF Operations for Logs Troubleshooting (Obj 5.8):

├── Log Ingestion Issues:
│   ├── Symptom: Logs not appearing from ESXi hosts
│   ├── Causes: Syslog not configured, firewall blocking UDP 514
│   ├── Diagnosis: On ESXi host → esxcli system syslog config get (verify loghost)
│   ├── Fix: esxcli system syslog config set --loghost=udp://<ops-logs-ip>:514
│   │   Follow with: esxcli network firewall ruleset set -e true -r syslog
│   │   Then: esxcli system syslog reload
│   └── Test: logger -t TEST "Test message" → verify in Explore Logs
├── Search & Analysis:
│   ├── Explore Logs: Full-text search across all ingested logs
│   ├── Field Extraction: Use regex to extract structured fields from unstructured logs
│   ├── Group By: Source, severity, custom field → identify patterns
│   ├── Saved Queries: Save frequently used searches for quick access
│   └── Alerts: Create log-based alerts (e.g., alert on "SCSI sense code" in vmkernel.log)
├── Observability Workbench:
│   ├── Unified view combining metrics (VCF Ops) and logs (Ops for Logs)
│   ├── Side-by-side correlation: Metric spike + corresponding log entries
│   ├── Launch-in-context: Click metric anomaly → jump to logs from same timeframe
│   └── Use case: Identify root cause by correlating performance degradation with log events
└── Disk Space Management:
    ├── Log retention: Default 30 days, configurable
    ├── Disk full: Log ingestion stops silently if disk full
    ├── Fix: Increase disk, archive old logs to NFS, reduce retention period
    └── Monitor: Administration → System Monitor → Disk Usage

Generate Log Bundle / Log Assist (Obj 5.8):

├── Log Bundle:
│   ├── VCF Operations → Administration → Support → Create Support Bundle
│   ├── Includes: Application logs, configuration, system state
│   ├── Size: 200-500MB typical
│   └── Upload to VMware support case for analysis
├── Log Assist:
│   ├── Automated log collection from connected vCenter and ESXi hosts
│   ├── Triggered via: VCF Operations → Troubleshooting → Log Assist
│   ├── Collects: vpxd.log, vmkernel.log, vobd.log from specified time range
│   ├── Formats: Compressed bundle ready for support case
│   └── Advantage: No SSH needed — centralized log collection via adapter

VCF Operations Orchestrator — Configuration & Workflows (Obj 5.9)

Orchestrator Configuration:

├── Deployment: Standalone appliance (2 vCPU, 6GB RAM) or embedded in VCF Operations
├── Access: https://<orchestrator-ip>/orchestration-ui/
├── Integration: Connect to vCenter (REST API), NSX (optional in VVF), ESXi (SSH)
├── Authentication: Same SSO domain as vCenter (administrator@vsphere.local)
└── Content: Pre-built workflow library + custom workflow designer

Running Workflows & Actions:

├── Pre-Built Workflows (Support-Relevant):
│   ├── VM Snapshot Management: Automated cleanup of snapshots > N days
│   ├── Host Maintenance Mode: Orchestrated entry/exit with VM migration
│   ├── Certificate Renewal: Automated certificate rotation for vCenter/ESXi
│   ├── Log Collection: Automated support bundle generation across hosts
│   ├── Compliance Remediation: Apply Host Profile fixes to non-compliant hosts
│   └── Scheduled Reports: Generate and email VCF Operations reports
├── Custom Workflow Creation:
│   ├── Designer: Drag-and-drop workflow editor
│   ├── Actions: JavaScript or Python code blocks
│   ├── Inputs: Parameters passed at runtime (VM name, host IP, etc.)
│   ├── Scheduling: Run on schedule (daily snapshot cleanup, weekly reports)
│   └── Event Triggers: Run on event (alert fired, VM created, host disconnected)
├── Troubleshooting Orchestrator:
│   ├── Workflow fails: Check workflow execution log (Orchestrator → Runs → select run → Logs)
│   ├── Connection issues: Verify vCenter endpoint credentials in Orchestrator config
│   ├── Timeout: Increase workflow timeout for long-running operations
│   └── Permissions: Workflow service account must have sufficient vCenter privileges
└── Support Use Cases:
    ├── Automated incident response: Alert triggers → Orchestrator collects diagnostics
    ├── Batch operations: Apply configuration fix to multiple hosts simultaneously
    ├── Scheduled health checks: Run vSAN health check weekly, email results
    └── Self-healing: Restart hung services automatically (with appropriate safeguards)

Holodeck Lab Note: VCF Operations Orchestrator can be deployed in Holodeck as a standalone appliance. Create a simple workflow: "Collect support bundle from all ESXi hosts" — this demonstrates the automation capability and is directly exam-relevant. Schedule the workflow to run weekly as a proactive support measure.

Key Takeaways

• Exam: VCF Operations adapter issues — test connection, check credentials, verify port 443. Logs in /var/log/vmware/vcops/.
• Exam: Ops for Logs — syslog config on ESXi (esxcli system syslog), firewall ruleset 'syslog' must be enabled. Explore Logs for full-text search.
• Exam: Observability Workbench = unified metrics + logs view. Launch-in-context for correlation. Log Assist = centralized bundle collection via adapter.
• Exam: Orchestrator — pre-built workflows (snapshot cleanup, cert renewal, log collection). Custom workflows via designer. Schedule or trigger on events.