Performance Analysis with esxtop
Objectives
- Launch and navigate esxtop views (CPU, memory, disk I/O, host-level analysis)
- Interpret CPU metrics (%USED, %RDY, %CSTP, %SYS, %WAIT) and identify contention patterns
- Analyze memory reclamation (balloon, swapping, compression) and detect memory pressure
- Diagnose storage I/O latency and correlate with backend infrastructure
- Perform trend analysis using esxtop batch mode for baseline documentation
- Recommend evidence-based remediation actions based on metric analysis
- Document performance findings in a structured OIAR record (Observation, Interpretation, Action, Rationale) and the design reflection in RCAR form (Requirements, Constraints, Assumptions, Risks)
Prerequisites
VCF lab environment deployed and operational with ESXi hosts
Required skills:
- SSH access to ESXi host
- Basic understanding of CPU scheduling and memory management
- Familiarity with performance baseline concepts
Lab Environment
Standard VCF lab environment for VCF 9.0 Support with multi-tier VM workloads
- Hosts needed: 2
- VMs needed: 3
- Storage: vSAN or iSCSI backend
Tasks
Task 1 CPU Performance Analysis and Contention Detection
Master esxtop CPU view and interpret multi-dimensional contention signals to identify scheduling issues and recommend vCPU optimization.
SSH to an ESXi host in your lab. Launch esxtop by typing 'esxtop' and wait for the interactive interface to load.
Press 'c' to enter the CPU view (should already be the default). Examine the column headers. Identify which rows represent VMs vs. host kernel.
Locate a VM or workload with measurable CPU usage (ideally 30%+). Note its name, current %USED, %RDY, and %CSTP values. Document the current state in your lab notes.
Analyze the scenario: A VM shows %USED=65%, %RDY=20%, %CSTP=2%. Interpret each metric. High %RDY indicates CPU contention at the host level. %CSTP=2% on a multi-vCPU VM suggests scheduling delays for SMP alignment.
Press 'e' and enter the VM's GID to expand the group into its worlds — one row per vCPU plus the helper worlds ('v' switches to the VM disk view and 'U' is a sort key, not a breakdown). Examine whether all vCPUs are equally loaded or if some idle while others busy.
Recommend remediation actions based on your analysis: (a) Reduce the VM's vCPU count by 1-2 cores to lower co-stop, (b) Right-size the VM to fit one NUMA node if it is memory-latency sensitive (check N%L in the memory view) rather than pinning it, (c) Check and remove CPU affinity settings that force scheduling onto specific cores (they also block DRS and vMotion), (d) Review DRS settings—ensure resource pools are not over-committed.
Document your CPU analysis in a structured format: Observation (metric values), Interpretation (what the metrics indicate), Action (specific change or next step), Rationale (why this action addresses the root cause).
Validation Gate
Check: CPU analysis complete with documented metrics and remediation plan
Expected: Lab notes show interpreted esxtop metrics, identified contention pattern, and evidence-based remediation recommendation
Common Errors
Task 2 Memory Performance Analysis and Pressure Detection
Interpret memory reclamation metrics (balloon, swapping, compression) to identify memory pressure and recommend resource allocation adjustments.
While in esxtop, press 'm' to switch to the memory view. Allow the view to render; it may take a few seconds.
Examine the memory view for a VM allocated 64GB. Note MCTLSZ, %ACTV, SWCUR, SWTGT, SWR/s and SWW/s values. Document the current state.
Introduce memory pressure scenario: Assume MCTLSZ=32GB (VM allocated 64GB, balloon reclaimed 32GB), SWCUR=1.2GB, and SWR/s and SWW/s above 0. Interpret: Balloon has reclaimed 32GB (50% of allocation), 1.2GB of this VM is in host swap (SWCUR), and the non-zero SWR/s and SWW/s show the host is swapping now. This indicates severe memory pressure.
Check ZIP and UNZIP metrics. High ZIP (compression) with high UNZIP (decompression) indicates the host is relying on memory compression as a stop-gap, which has CPU overhead. Compression costs CPU: compare the ZIP/s and UNZIP/s trend with the host's %SYS and CPU headroom rather than applying a fixed ratio (no documented CPU-per-ZIP/s rule exists).
Recommend memory remediation: (a) Add physical RAM to the host(s), (b) Reduce the number of VMs, (c) Right-size VM memory from active-memory (%ACTV) trends and use reservations only for workloads that must not balloon or swap, (d) Migrate workloads to less-pressured hosts via DRS, (e) Audit guest OS memory settings for misconfiguration (e.g., huge pages, buffer pools).
Document memory pressure indicators and decision thresholds in your notes: (1) MCTLSZ > 10% of allocation = monitor closely. (2) MCTLSZ > 20% = action required. (3) SWR/s or SWW/s > 0 (active swapping) = emergency; SWCUR alone may be historical. (4) ZIP/s sustained above about 200 MB/s (ZIP/s is MB per second) = investigate; this threshold is a teaching guideline, not a documented value.
Validation Gate
Check: Memory analysis complete with interpreted pressure indicators and ranked remediation plan
Expected: Lab notes document memory metrics, pressure assessment, and actionable remediation with cost/benefit trade-offs
Common Errors
Task 3 Storage and Disk I/O Analysis
Diagnose storage latency using esxtop disk views and correlate metrics with vSAN or backend storage health.
From esxtop, press 'u' to switch to disk device view (physical/logical disk). Wait for the view to render.
Examine your storage device list. Identify the primary data store used by your lab VMs. Note the baseline DAVG. A healthy SSD-backed datastore should show DAVG < 5ms. HDD-backed should show DAVG < 10ms.
Introduce disk I/O stress: Run a heavy I/O workload on a guest VM (e.g., 'fio --name=randrw --ioengine=libaio --iodepth=32 --rw=randrw --bs=8k --runtime=300'). Monitor esxtop disk view in real-time.
Interpret the scenario: DAVG=45ms against the Step 3 baseline of 3.2ms = ~14x storage latency spike. Analyze root cause. In vSAN, check: (a) Is this a vSAN rebuild scenario (rebuilding a failed disk/host)? (b) Are vSAN resync operations active? (c) Is the storage network saturated (vSAN uses dedicated network)?
If your lab uses vSAN, SSH to a vSAN-enabled host and check vSAN with vsantop (per-host vSAN performance), esxcli vsan health cluster list, esxcli vsan debug resync summary get (resync/rebalance activity) and vdq -q (device eligibility); cluster-level latency comes from the vSAN Performance Service in vCenter ('vsanstats' and 'vsan.stats' did not surface as ESX commands; RVC is deprecated in VCF 9.0, p.233).
Press 'v' (VM disk view) to see disk latency from the VM perspective (GAVG). Identify which VM is experiencing the worst latency. GAVG = DAVG + KAVG (KB 344099), so GAVG is always at least DAVG; the gap is KAVG — time spent in the VMkernel.
Recommend storage remediation based on your analysis: (a) For vSAN rebuild scenarios, wait for rebuild to complete and re-measure latency. (b) For persistent high DAVG, increase storage capacity (add disks to vSAN disk groups, upgrade SAN array). (c) Check network for vSAN: ensure the vSAN network VLAN chosen in your design is not congested (there is no default vSAN VLAN number). (d) Consider QoS policies to throttle non-critical I/O during peak periods.
Validation Gate
Check: Storage I/O analysis complete with diagnosed latency spike and remediation recommendation
Expected: Lab notes document esxtop disk metrics, baseline vs. stressed state comparison, vSAN-specific diagnostics (if applicable), and tiered remediation plan
Common Errors
Task 4 Host-Level Analysis and Batch Mode Trending
Analyze host-level resource saturation and establish performance baselines using esxtop batch mode for trend analysis and documentation.
Return to esxtop interactive CPU view ('c'). Observe the host kernel metrics (typically listed first or in a specific row). Note host %USED and %SYS. Host %USED near 85%+ indicates resource saturation; a %SYS well above your baseline (sustained) indicates high system overhead (drivers, interrupts, page-table management) — no fixed threshold is published.
Interpret the scenario: Host %USED=85%, %SYS=8%. The host is near saturation, and system overhead is climbing. Possible causes: (a) Too many VMs for the hardware. (b) Driver inefficiency (NIC driver, storage controller driver). (c) Memory pressure causing high page-table walks. (d) Interrupt storms from overactive devices.
Exit esxtop interactive mode ('q'). Prepare to run esxtop in batch mode for trending. Batch mode captures metrics at regular intervals and outputs CSV data suitable for spreadsheet analysis.
Run esxtop in batch mode with the command: 'esxtop -b -d 5 -n 720 > /tmp/perf.csv'. This captures 720 samples at 5-second intervals (60 minutes of data). Let it run to completion.
Retrieve the CSV file to your workstation via SCP or FTP. Example: 'scp root@your-esxi-ip:/tmp/perf.csv ./perf-baseline.csv'. Open the file in a spreadsheet (Excel, Google Sheets, LibreOffice Calc).
In the spreadsheet, create a chart of key metrics over time: (1) Plot host %USED, %RDY, %SYS as a multi-series line chart. (2) Identify peak utilization and note the timestamp. (3) Look for trends: Is utilization increasing over the hour, or stable? (4) Calculate moving average (15-minute window) to smooth spikes.
Create a performance baseline document in your lab notes (or as a formal write-up). Include: (1) Host specifications (CPU sockets, cores, RAM). (2) Baseline metrics during steady-state (low load): %USED, %RDY, %SYS, DAVG, MCTLSZ. (3) Baseline metrics during peak load: same metrics at 75-85% utilization. (4) Thresholds for alerting (when to escalate).
Document batch mode methodology in your notes: When to run, how long to run, what to measure, and how to interpret results. Include a sample command and sample output interpretation.
Validation Gate
Check: Batch mode trending complete with baseline document and trend analysis interpretation
Expected: Lab notes contain esxtop batch command, CSV trend analysis, performance baseline document with thresholds, and methodology for future trending
Common Errors
Final Validation
Lab completed when all 4 tasks are executed, metrics are documented, and remediation recommendations are justified.
✓ CPU analysis with high-contention scenario analyzed and remediation recommended → Task 1 complete: %RDY, %CSTP metrics interpreted; vCPU adjustment documented with rationale
✓ Memory pressure scenario analyzed with reclamation metrics documented → Task 2 complete: MCTLSZ, SWCUR, ZIP metrics documented; memory pressure assessed; remediation ranked by priority
✓ Storage I/O latency diagnosed with baseline and stress comparisons → Task 3 complete: DAVG, KAVG, GAVG metrics documented; vSAN health checked (if applicable); storage remediation planned
✓ Host-level baseline established via batch mode trending → Task 4 complete: esxtop batch CSV captured, trending chart created, baseline document produced with alert thresholds
Cleanup / Restore
• Remove /tmp/perf.csv from ESXi host (or retain for 30 days as audit trail)
• Revert any VM I/O stress workloads (stop fio, stress-ng)
• Return host CPU/memory to baseline utilization (if VMs were intentionally overloaded)
• Restore baseline snapshots if any changes were made to VM configurations
Design Reflection (VCDX)
This lab emphasizes diagnostic methodology and evidence-based decision-making—core VCDX competencies. Panelists will probe: (1) How do you validate that your esxtop diagnosis matches actual VM performance issues? (2) When do you recommend quick fixes vs. architectural changes? (3) How do you balance performance tuning against operational risk? (4) Can you articulate the trade-offs between CPU overcommit and power efficiency? The RCAR framework below provides structure for panel discussion.
Requirements
- ESXi host(s) with operational VMs and I/O workloads
- SSH/CLI access to ESXi host
- Spreadsheet software for CSV analysis and charting
- Lab VMs running representative workloads (CPU, memory, disk I/O)
- vSAN deployment (optional, for advanced storage metrics) or iSCSI/NFS datastore
- Baseline performance data for the lab environment (to establish 'healthy' thresholds)
Constraints
- esxtop must be run directly on ESXi host; remote monitoring tools (VCF Operations) provide different metrics and context
- Batch mode CSV captures are host-specific; trending across multiple hosts requires aggregation logic outside esxtop
- CPU/memory contention metrics are OS-kernel-dependent; different ESXi versions may report slightly different values
- vSAN-specific metrics (rebuild status, rebalance operations) require vSAN Skyline Health or vSAN Management REST API; not all metrics available in esxtop
- Interactive esxtop view requires SSH terminal session; cannot be automated or scheduled without batch mode
Assumptions
- Lab VMs are configured with realistic resource allocations (not over-provisioned with 100 vCPUs on 16-core host)
- Storage backend is functioning normally during the lab (no storage array failures masking remediation)
- Network connectivity between ESXi hosts and vSAN network is stable (for vSAN trending)
- Lab operator has basic Linux/Windows CLI knowledge to trigger I/O workloads and interpret OS-level metrics
- Remediation recommendations assume you have approval to modify VM configurations (vCPU count, memory allocation, DRS settings)
- Baseline 'healthy' values are organization-specific; lab uses the Broadcom KB 304594 guide (%READY under 5% in normal operation) as reference
Risks
- Risk: Over-aggressive VM scaling during CPU contention analysis may cause application downtime if vCPU is reduced too much — Mitigation: In a VCDX interview, outline the plan and revert immediately if issues emerge. Do not apply changes without supervisor approval in production.
- Risk: Batch mode trending captures can consume significant disk space if run for extended periods (e.g., 24-hour capture = 5-10 MB CSV) — Mitigation: Plan storage; use '-d 30' (30-second intervals) instead of '-d 5' for longer captures. Compress/delete old CSV files after analysis.
- Risk: Memory pressure scenario (MCTLSZ > 20%, SWCUR > 0) may trigger emergency performance degradation or guest OS OOM (Out of Memory) kill — Mitigation: In a lab environment, monitor guest OS closely during memory stress. Have snapshot/revert strategy ready if guest becomes unresponsive.
- Risk: Storage I/O stress (fio) may impact concurrent workloads if run on shared datastore; can cause noisy neighbor contention — Mitigation: Run I/O stress on a dedicated VM on dedicated datastore, or run during off-hours. Document timing in lab notes.
Self-Assessment Discussion Prompts
- Walk me through how you decided what %RDY per vCPU counts as 'concerning contention' (the VM row sums all vCPUs). Where does that number come from? Would you apply the same threshold to a DSS workload vs. an OLTP workload?
- You recommended reducing vCPU from 8 to 6. How would you validate that this change improves performance without harming application throughput? What metrics would you monitor post-change?
- In the memory pressure scenario (MCTLSZ=32GB, SWCUR=1.2GB), you recommended adding RAM as Priority 1. But hardware upgrades take weeks. What is your Priority 2 action to buy time while procurement happens?
- You observed DAVG spiking from 5ms to 45ms during I/O stress. Before blaming the storage array, what hypervisor/network factors would you investigate? How would you distinguish storage latency from network latency in a vSAN environment?
- Batch mode gave you 60 minutes of baseline data. Is that enough to define 'healthy' thresholds for the next quarter? What time window would you recommend for production baselining, and why?
- A VM shows %RDY=15%, DAVG=8ms, but the application team says performance is fine. How do you reconcile esxtop metrics with user-perceived performance? What layer are you missing?
- You're being asked to consolidate 20 more VMs onto this cluster. Based on your baseline, what's the maximum consolidated VMs this cluster can handle before %RDY exceeds the threshold you set for this tier (KB 304594 reference: under 5% under normal operating conditions)?
Extensions
Multi-Host Trending and Anomaly Detection
Extend the lab to capture esxtop batch data from 3+ ESXi hosts simultaneously. Aggregate CSV data into a unified spreadsheet, calculate cluster-level CPU and memory utilization, and identify outlier hosts (e.g., one host consistently near 85% while others idle at 45%). Create alert rules: if one host exceeds 80% while cluster average is 50%, trigger DRS rebalancing. This extension demonstrates cluster-level capacity planning and load-balancing logic.
Performance Regression Testing with VCF Operations
Compare manual esxtop analysis with VCF Operations metrics. Use VCF Operations to generate the same baseline charts as your esxtop CSV analysis, then overlay recommendations. This extension bridges ad-hoc troubleshooting (esxtop) with continuous monitoring (VCF Operations), demonstrating how to design a monitoring strategy based on esxtop findings.
Remediation Validation and A/B Testing
Apply the vCPU reduction (Task 1) or memory balancing (Task 2) changes in a controlled manner. Re-run esxtop batch mode for 60 minutes post-change and compare charts. Quantify the improvement: 'vCPU reduction from 8 to 6 reduced %RDY by 12 percentage points, reducing VM schedule latency by ~300 microseconds per operation.' This extension demonstrates rigor in validating that your diagnosis translates to measurable improvement.
⚠ Known Pitfalls
References
- ESXi Performance Analysis with esxtopTier 1 — Official
- Interpreting CPU Ready (%RDY) and Co-Stop (%CSTP) in vSphereTier 1 — Official
- Memory Balloon, Swap, and Compression in vSphereTier 1 — Official
- Storage Performance Best Practices: DAVG, KAVG, GAVG InterpretationTier 1 — Official
- esxtop Command-Line Reference and Batch Mode UsageTier 1 — Official