Lab: Diagnose Performance Issue with esxtop & Logs
Objectives
- Read CPU (%RDY, %CSTP), memory (MCTLSZ, SWCUR, SWR/s) and disk (DAVG, KAVG, GAVG) counters in esxtop (Obj 5.5)
- Capture esxtop batch output and a vm-support bundle to a datastore path
- Name the bottleneck layer and a fix at that layer
Prerequisites
Holodeck VVF 9.0 instance; a load-generator VM (stress-ng or fio) on the vSAN datastore; SSH enabled on its host
Lab Environment
Holodeck vSphere Foundation 9.0 instance (New-HoloDeckInstance -VVF; -vSANMode OSA by default, ESA where the physical layer supports it): vCenter managing one vSAN cluster of nested ESX hosts, plus VCF Operations. VVF has no SDDC Manager and no NSX.
Credentials
| System | Username | Password |
|---|---|---|
| vCenter | administrator@vsphere.local | Holodeck documented default (Post Deployment table) |
| VCF Operations | admin | Holodeck documented default (Post Deployment table) |
| Nested ESX hosts | root | Holodeck documented default (Post Deployment table) |
Tasks
Task 1 Find the bottleneck with esxtop
Simulate performance issue, use esxtop to identify bottleneck, collect logs, analyze and recommend fix.
SSH to ESXi host, run esxtop in CPU view
Note current utilization (baseline)
Initiate heavy I/O workload (stress-ng or fio on VM)
Return to esxtop, observe latency and queue depth increasing
Switch to disk view, identify bottleneck disk
Switch to memory view, check for ballooning/swapping
Generate vm-support bundle: vm-support -w /vmfs/volumes/<datastore> (-w sets the output directory; the .tgz is named esx-<host>-<date>.tgz — KB 313542)
Extract and review vmkernel.log for errors during workload
Document findings: bottleneck cause, recommendation (add disk, upgrade NIC, etc.)
Validation Gate
Check: Using esxtop, identify: which VM has highest CPU ready time, whether any host memory pressure exists (ballooning/swapping), and current disk latency per datastore
Expected: CPU: %RDY <5% for all VMs (or identified high-RDY VMs for investigation). Memory: no active swapping (SWR/s and SWW/s at 0) and MCTLSZ=0; SWCUR can stay above 0 after past pressure. Disk: DAVG/KAVG/GAVG not above ~10 ms sustained (KB 344099), or the high-latency devices identified.
Common Errors
Final Validation
Gate: every check below passes with the evidence recorded in the lab log.
✓ esxtop CPU view → %RDY per VM read; contention flagged above ~5% (KB 304594)
✓ esxtop memory view → Active swapping (SWR/s, SWW/s) and ballooning (MCTLSZ) read correctly
✓ esxtop disk view → DAVG/KAVG/GAVG compared with ~10 ms sustained (KB 344099)
✓ Bundle → vm-support .tgz written to the datastore directory given to -w
Cleanup / Restore
• Stop the load generator
• Delete the esxtop CSV and the bundle from the datastore when finished
Design Reflection (VCDX)
Performance troubleshooting methodology is tested in VCDX defense. Panelists may present a performance scenario and expect you to describe your diagnostic approach — esxtop is the first tool you should mention.
Requirements
- Use esxtop to diagnose CPU, memory, disk, and network performance
- Capture batch mode output for historical analysis
- Correlate multi-dimensional metrics for root cause analysis
Constraints
- esxtop shows real-time data only — use batch mode for intermittent issues
- Performance diagnosis requires ESXi SSH access
- esxtop output is complex — know key metrics per dimension
Assumptions
- Baseline performance metrics are known for comparison
- ESXi SSH access is available during troubleshooting
Risks
- Misdiagnosing the symptom dimension as the root cause (application 'slow CPU' complaint with a storage root cause visible in %VMWAIT and GAVG)
- Missing intermittent issues without batch mode capture
⚠ Known Pitfalls
References
- Broadcom KB 344099 — esxtop storage performance countersTier 1 — Official
- Broadcom KB 304594 — esxtop CPU countersTier 1 — Official
- Broadcom KB 313542 — vm-support command optionsTier 1 — Official