Academy/vSphere Foundation 9.0 Support (2V0-18.25)/Lab: Diagnose Performance Issue with esxtop & Logs
This lab targets VCF 9.0

Lab: Diagnose Performance Issue with esxtop & Logs

VCF 9.0Intermediatevcp-foundation⏱ 135 min

VVF 9.0 support lab (2V0-18.25). Built for a Holodeck VVF instance.

Objectives

  • Read CPU (%RDY, %CSTP), memory (MCTLSZ, SWCUR, SWR/s) and disk (DAVG, KAVG, GAVG) counters in esxtop (Obj 5.5)
  • Capture esxtop batch output and a vm-support bundle to a datastore path
  • Name the bottleneck layer and a fix at that layer

Prerequisites

Holodeck VVF 9.0 instance; a load-generator VM (stress-ng or fio) on the vSAN datastore; SSH enabled on its host

Lab Environment

Holodeck vSphere Foundation 9.0 instance (New-HoloDeckInstance -VVF; -vSANMode OSA by default, ESA where the physical layer supports it): vCenter managing one vSAN cluster of nested ESX hosts, plus VCF Operations. VVF has no SDDC Manager and no NSX.

Credentials

SystemUsernamePassword
vCenteradministrator@vsphere.localHolodeck documented default (Post Deployment table)
VCF OperationsadminHolodeck documented default (Post Deployment table)
Nested ESX hostsrootHolodeck documented default (Post Deployment table)

Tasks

Task 1 Find the bottleneck with esxtop

esxtop is the essential real-time performance diagnostic tool for ESXi. Understanding the key metrics in each view (CPU, memory, network, disk) and what constitutes abnormal values is the foundation of performance troubleshooting.

Simulate performance issue, use esxtop to identify bottleneck, collect logs, analyze and recommend fix.

Step 1

SSH to ESXi host, run esxtop in CPU view

Step 2

Note current utilization (baseline)

Step 3

Initiate heavy I/O workload (stress-ng or fio on VM)

Step 4

Return to esxtop, observe latency and queue depth increasing

Step 5

Switch to disk view, identify bottleneck disk

Step 6

Switch to memory view, check for ballooning/swapping

Step 7

Generate vm-support bundle: vm-support -w /vmfs/volumes/<datastore> (-w sets the output directory; the .tgz is named esx-<host>-<date>.tgz — KB 313542)

Step 8

Extract and review vmkernel.log for errors during workload

Step 9

Document findings: bottleneck cause, recommendation (add disk, upgrade NIC, etc.)

Validation Gate

Check: Using esxtop, identify: which VM has highest CPU ready time, whether any host memory pressure exists (ballooning/swapping), and current disk latency per datastore

Expected: CPU: %RDY <5% for all VMs (or identified high-RDY VMs for investigation). Memory: no active swapping (SWR/s and SWW/s at 0) and MCTLSZ=0; SWCUR can stay above 0 after past pressure. Disk: DAVG/KAVG/GAVG not above ~10 ms sustained (KB 344099), or the high-latency devices identified.

Common Errors

Reading esxtop %USED as VM CPU utilization
Fix: In esxtop CPU view: %USED = physical CPU time used by the VM (can exceed 100% on multi-vCPU VMs). %RDY (ready time) is more important — it shows time the VM wanted CPU but couldn't get it. %RDY > 5% indicates CPU contention. %CSTP (co-stop) > 3% indicates vSMP scheduling issues (reduce vCPU count).
Ignoring memory metrics beyond active/consumed
Fix: esxtop memory view shows: MCTLSZ (balloon driver reclaimed — host is under memory pressure), SWCUR (amount currently swapped — it can stay above 0 long after the pressure has passed), SWR/s and SWW/s (active swap-in and swap-out — the sign of current pressure), ZIP/UNZIP (compressed pages). Active swapping means the host is short of memory now. MCTLSZ > 0 indicates ballooning — review memory allocation.
Not capturing esxtop batch mode output for historical analysis
Fix: Real-time esxtop shows current state only. For performance issues that are intermittent, use batch mode: 'esxtop -b -d 5 -n 720 > /vmfs/volumes/<datastore>/esxtop.csv' (5-second samples for 1 hour; /tmp is a small ramdisk). Read the CSV in Windows Performance Monitor or a spreadsheet; vscsiStats is a separate histogram collector, not a CSV analyser. Batch mode captures the data you need for Broadcom support cases.
Looking at only one resource dimension
Fix: Performance issues are often multi-dimensional: a VM that looks CPU-starved may really be waiting on storage — check %VMWAIT/%WAIT and GAVG alongside %RDY, because I/O waits accrue as wait time, not ready time (%RDY counts time a world was ready to run but could not be scheduled on a physical CPU, Broadcom KB 304594). Check all four dimensions: CPU (%RDY, %CSTP, %VMWAIT), Memory (SWCUR, MCTLSZ), Disk (GAVG/DAVG/KAVG), Network (dropped packets). The root cause is often in a different dimension than the symptom.

Final Validation

Gate: every check below passes with the evidence recorded in the lab log.

✓ esxtop CPU view → %RDY per VM read; contention flagged above ~5% (KB 304594)

✓ esxtop memory view → Active swapping (SWR/s, SWW/s) and ballooning (MCTLSZ) read correctly

✓ esxtop disk view → DAVG/KAVG/GAVG compared with ~10 ms sustained (KB 344099)

✓ Bundle → vm-support .tgz written to the datastore directory given to -w

Cleanup / Restore

• Stop the load generator

• Delete the esxtop CSV and the bundle from the datastore when finished

Design Reflection (VCDX)

Performance troubleshooting methodology is tested in VCDX defense. Panelists may present a performance scenario and expect you to describe your diagnostic approach — esxtop is the first tool you should mention.

Requirements

  • Use esxtop to diagnose CPU, memory, disk, and network performance
  • Capture batch mode output for historical analysis
  • Correlate multi-dimensional metrics for root cause analysis

Constraints

  • esxtop shows real-time data only — use batch mode for intermittent issues
  • Performance diagnosis requires ESXi SSH access
  • esxtop output is complex — know key metrics per dimension

Assumptions

  • Baseline performance metrics are known for comparison
  • ESXi SSH access is available during troubleshooting

Risks

  • Misdiagnosing the symptom dimension as the root cause (application 'slow CPU' complaint with a storage root cause visible in %VMWAIT and GAVG)
  • Missing intermittent issues without batch mode capture

⚠ Known Pitfalls

Treating %USED as the primary CPU health indicator — %RDY (ready time) is far more important for identifying CPU contention.
Checking only one resource dimension — performance issues are often cross-dimensional (e.g. a slow application blamed on CPU when %VMWAIT and GAVG show a storage bottleneck).

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.