Academy/VCF 9.0 Support (2V0-15.25)/Performance Analysis with esxtop
This lab targets VCF 9.0

Performance Analysis with esxtop

VCF 9.0Intermediatevcp-foundationvcp-support⏱ 90 min

Focus on deep metric interpretation and remediation methodology.

Objectives

  • Launch and navigate esxtop views (CPU, memory, disk I/O, host-level analysis)
  • Interpret CPU metrics (%USED, %RDY, %CSTP, %SYS, %WAIT) and identify contention patterns
  • Analyze memory reclamation (balloon, swapping, compression) and detect memory pressure
  • Diagnose storage I/O latency and correlate with backend infrastructure
  • Perform trend analysis using esxtop batch mode for baseline documentation
  • Recommend evidence-based remediation actions based on metric analysis
  • Document performance findings in a structured OIAR record (Observation, Interpretation, Action, Rationale) and the design reflection in RCAR form (Requirements, Constraints, Assumptions, Risks)

Prerequisites

VCF lab environment deployed and operational with ESXi hosts

Required skills:

  • SSH access to ESXi host
  • Basic understanding of CPU scheduling and memory management
  • Familiarity with performance baseline concepts

Lab Environment

Standard VCF lab environment for VCF 9.0 Support with multi-tier VM workloads

  • Hosts needed: 2
  • VMs needed: 3
  • Storage: vSAN or iSCSI backend

Tasks

Task 1 CPU Performance Analysis and Contention Detection

Task progresses from basic metric recognition to scenario-based interpretation. Panelists will probe: How do you distinguish CPU contention from CPU-dependent application performance? What is the significance of %CSTP in a multi-socket VM? How would you validate your diagnosis before making changes?

Master esxtop CPU view and interpret multi-dimensional contention signals to identify scheduling issues and recommend vCPU optimization.

Step 1

SSH to an ESXi host in your lab. Launch esxtop by typing 'esxtop' and wait for the interactive interface to load.

esxtop displays the default CPU view with columns showing %USED, %RDY, %CSTP, %SYS, %WAIT, %IDLE. Screen header shows processor count and uptime. Legend available via 'h'.
If the screen appears garbled, press 'c' to explicitly switch to CPU view. Use 'q' to quit esxtop cleanly.
Step 2

Press 'c' to enter the CPU view (should already be the default). Examine the column headers. Identify which rows represent VMs vs. host kernel.

CPU view displays one row per VM plus rows for kernel domains. %USED should show current CPU usage, %RDY shows waiting-for-scheduler time, %CSTP shows co-stop time (SMP VMs waiting for all vCPUs to align).
Use 'f' to add/remove fields; for this lab ensure %USED, %RDY, %CSTP, %SYS, and %WAIT are visible. Familiarize yourself with the sort order (default is %USED descending).
Step 3

Locate a VM or workload with measurable CPU usage (ideally 30%+). Note its name, current %USED, %RDY, and %CSTP values. Document the current state in your lab notes.

Baseline snapshot captured: e.g., 'web-server-01: %USED=65%, %RDY=18%, %CSTP=0.5%'. Record host %SYS for your baseline; a healthy host shows a low single-digit %SYS, but there is no published band — compare against your own baseline.
If no VMs are running CPU workloads, manually trigger a CPU load: e.g., on the VM, run 'stress-ng --cpu 4 --timeout 300s' or equivalent for your OS.
Step 4

Analyze the scenario: A VM shows %USED=65%, %RDY=20%, %CSTP=2%. Interpret each metric. High %RDY indicates CPU contention at the host level. %CSTP=2% on a multi-vCPU VM suggests scheduling delays for SMP alignment.

Written analysis: 'Host is over-subscribed with CPU. The 20% ready time means this VM is waiting 20% of the time to be scheduled, reducing effective throughput. The 2% co-stop suggests the hypervisor is frequently delaying the VM to align all its vCPUs, typical when vCPU count is high relative to physical cores.'
Remember: Broadcom KB 304594 ('Troubleshooting ESX/ESXi virtual machine performance issues') says %READY should remain under 5% under normal operating conditions; KB 304594 gives no per-vCPU figure. The VM row sums all vCPUs, so an 8-vCPU VM at 20% is 2.5% per vCPU — record both the row and the per-vCPU values before judging contention. %CSTP > 1% on a single-socket VM is unusual; check vCPU count.
Step 5

Press 'e' and enter the VM's GID to expand the group into its worlds — one row per vCPU plus the helper worlds ('v' switches to the VM disk view and 'U' is a sort key, not a breakdown). Examine whether all vCPUs are equally loaded or if some idle while others busy.

Per-vCPU breakdown may show asymmetric load distribution, indicating imbalanced threading or uneven workload distribution within the VM.
Every current esxtop exposes per-vCPU rows through 'e'; the VM-level %RDY you saw before expanding is the SUM across its vCPUs, so divide by vCPU count before judging it.
Step 6

Recommend remediation actions based on your analysis: (a) Reduce the VM's vCPU count by 1-2 cores to lower co-stop, (b) Right-size the VM to fit one NUMA node if it is memory-latency sensitive (check N%L in the memory view) rather than pinning it, (c) Check and remove CPU affinity settings that force scheduling onto specific cores (they also block DRS and vMotion), (d) Review DRS settings—ensure resource pools are not over-committed.

Remediation plan documented: 'Step 1: Reduce web-server-01 vCPU from 8 to 6. Step 2: Verify %RDY drops back under your threshold (KB 304594: under 5% in normal operation). Step 3: Re-measure VM application latency to ensure no performance regression.'
Do NOT immediately apply these changes in a VCDX interview context—outline the plan and success criteria. Panelists are assessing your diagnostic reasoning, not your willingness to make changes.
Step 7

Document your CPU analysis in a structured format: Observation (metric values), Interpretation (what the metrics indicate), Action (specific change or next step), Rationale (why this action addresses the root cause).

Documented OIAR (Observation, Interpretation, Action, Rationale) analysis for the CPU task in your lab notes; OIAR records a finding, while the RCAR in the design reflection records requirements, constraints, assumptions and risks. Example: Observation: %RDY=20%, %CSTP=2%, vCPU count=8. Interpretation: Host CPU oversubscribed; SMP VM vCPU count misaligned. Action: Reduce vCPU to 6. Rationale: Reduce scheduling complexity.
The OIAR structure makes your reasoning traceable from evidence to action, which is what a VCDX panel probes for.

Validation Gate

Check: CPU analysis complete with documented metrics and remediation plan

Expected: Lab notes show interpreted esxtop metrics, identified contention pattern, and evidence-based remediation recommendation

Common Errors

Confusing %RDY with %WAIT. %RDY = time the world was ready but not scheduled; %WAIT = time in the blocked or busy-wait state, and it INCLUDES idle time (%IDLE), so an idle VM shows high %WAIT. To see I/O waits use %VMWAIT (blocked waiting for events) and %SWPWT for swap waits. High %RDY with low %VMWAIT = CPU contention; high %VMWAIT = the VM is waiting on storage/network/devices.
Fix: Before recommending vCPU changes, check %VMWAIT (not raw %WAIT, which includes idle). If %VMWAIT is high the VM is I/O-bound, not CPU-bound, and reducing vCPUs won't help.
Recommending vCPU reduction without considering application threading model. Some apps are multi-threaded; others are single-threaded but spawning many processes.
Fix: Ask: Is the VM's workload truly CPU-bound, or is it throttled by contention? Adjust vCPU proportionally to physical cores, not arbitrarily.
Ignoring %SYS (system overhead). High %SYS can indicate driver inefficiency, memory pressure causing page-table overhead, or excessive interrupt handling.
Fix: If %SYS is sustained well above your baseline, investigate memory pressure and driver versions before adjusting vCPU.

Task 2 Memory Performance Analysis and Pressure Detection

Task requires understanding of ESXi memory management layering: guest paging < balloon < compression < host swap. Panelists will ask: At what point does balloon activity become concerning? How do you distinguish memory pressure from memory efficiency? What is the performance cost of compression vs. ballooning vs. swapping?

Interpret memory reclamation metrics (balloon, swapping, compression) to identify memory pressure and recommend resource allocation adjustments.

Step 1

While in esxtop, press 'm' to switch to the memory view. Allow the view to render; it may take a few seconds.

Memory view displays columns including MCTLSZ (balloon-driven memory reclamation), SWCUR (current host swap usage), SWTGT (host swap target), ZIP/s and UNZIP/s (memory compressed / decompressed per second, MB/s), %ACTV (percentage of guest physical memory the guest is referencing), GRANT (guest physical memory mapped to the VM). The header line shows the memory overcommitment averages over 1, 5 and 15 minutes (1.00 = 100 percent overcommitment).
If memory columns don't fit the screen, press 'f' to customize fields. Ensure MCTLSZ, SWCUR, SWR/s, SWW/s, ZIP/s and %ACTV are visible.
Step 2

Examine the memory view for a VM allocated 64GB. Note MCTLSZ, %ACTV, SWCUR, SWTGT, SWR/s and SWW/s values. Document the current state.

Baseline: e.g., 'db-vm-01: Allocated=64GB, MCTLSZ=0GB, %ACTV=75 (about 48GB referenced), SWCUR=0GB, SWTGT=0GB, SWR/s=0, SWW/s=0, ZIP/s and UNZIP/s near 0'. This indicates healthy memory utilization: no balloon reclamation, no swap, negligible compression.
If you don't see compression/swap activity, intentionally increase VM workload (e.g., run 'memtester 50G' on the guest) to observe memory pressure responses.
Step 3

Introduce memory pressure scenario: Assume MCTLSZ=32GB (VM allocated 64GB, balloon reclaimed 32GB), SWCUR=1.2GB, and SWR/s and SWW/s above 0. Interpret: Balloon has reclaimed 32GB (50% of allocation), 1.2GB of this VM is in host swap (SWCUR), and the non-zero SWR/s and SWW/s show the host is swapping now. This indicates severe memory pressure.

Analysis: 'Host memory is exhausted. The ESXi hypervisor activated balloon driver to reclaim 32GB from this VM (reducing its effective memory to 32GB). Despite ballooning, 1.2GB of this VM is in host swap (SWCUR) and the non-zero SWR/s and SWW/s show that swapping is active now, indicating memory overallocation on this host. This is a critical issue: VM performance will degrade significantly due to swap I/O latency.'
Balloon reclamation > 20% of VM allocation is a red flag. Non-zero SWR/s or SWW/s (active swapping) is the critical indicator; SWCUR > 0 only shows that swapping happened at some point.
Step 4

Check ZIP and UNZIP metrics. High ZIP (compression) with high UNZIP (decompression) indicates the host is relying on memory compression as a stop-gap, which has CPU overhead. Compression costs CPU: compare the ZIP/s and UNZIP/s trend with the host's %SYS and CPU headroom rather than applying a fixed ratio (no documented CPU-per-ZIP/s rule exists).

Sustained high ZIP/s with rising UNZIP/s means the host is relying on compression as a stop-gap: record its CPU impact (%SYS and host CPU in esxtop) and treat it as a sign of memory overcommitment that needs workload rebalancing or more host memory.
Compression is a last resort before swapping. If compression is high, treat it as a symptom of memory overallocation, not a solution.
Step 5

Recommend memory remediation: (a) Add physical RAM to the host(s), (b) Reduce the number of VMs, (c) Right-size VM memory from active-memory (%ACTV) trends and use reservations only for workloads that must not balloon or swap, (d) Migrate workloads to less-pressured hosts via DRS, (e) Audit guest OS memory settings for misconfiguration (e.g., huge pages, buffer pools).

Remediation plan: 'Priority 1: Add 32GB RAM to host. Priority 2: Migrate 2 non-critical VMs to another cluster. Priority 3: Right-size db-vm-01 from its active-memory trend, and keep a reservation only if the database must not balloon or swap (a reservation guarantees memory to the VM; it does not trigger DRS migrations and it reduces consolidation).'
In a VCDX interview, provide a ranked plan with cost/benefit analysis. 'Add RAM' is expensive; demonstrate you've considered alternatives.
Step 6

Document memory pressure indicators and decision thresholds in your notes: (1) MCTLSZ > 10% of allocation = monitor closely. (2) MCTLSZ > 20% = action required. (3) SWR/s or SWW/s > 0 (active swapping) = emergency; SWCUR alone may be historical. (4) ZIP/s sustained above about 200 MB/s (ZIP/s is MB per second) = investigate; this threshold is a teaching guideline, not a documented value.

Reference table in lab notes with thresholds and recommended actions at each level. Example: 'If MCTLSZ=5GB on a 64GB VM and ZIP/s=150 MB/s, initiate DRS balancing but no hardware upgrade yet. If MCTLSZ=20GB, SWCUR=2GB and SWR/s or SWW/s is above 0, immediate escalation to management.'
These thresholds are guidelines, not absolutes. Adjust based on your organization's SLA and workload criticality.

Validation Gate

Check: Memory analysis complete with interpreted pressure indicators and ranked remediation plan

Expected: Lab notes document memory metrics, pressure assessment, and actionable remediation with cost/benefit trade-offs

Common Errors

Assuming MCTLSZ > 0 always means a problem. In highly consolidated environments, some ballooning is normal and expected.
Fix: Context matters. A 5GB balloon on a 64GB VM in a 256GB host is different from 32GB on a 64GB VM in a 256GB host. Compare MCTLSZ % to host free memory %.
Ignoring guest OS paging. If active memory (%ACTV) is low but VM is slow, the guest OS may be paging internally. esxtop can't see inside the guest.
Fix: Always cross-check: Open a guest console or SSH session and run 'free' (Linux) or 'wmic OS get TotalVisibleMemorySize,FreePhysicalMemory' (Windows) to see guest-level memory pressure.
Recommending memory upgrades without understanding the workload trend. Is memory pressure constant or spike-driven?
Fix: Use batch mode (see Task 4) to capture hourly trends. Spiky workloads may benefit from rightsizing instead of hardware upgrades.

Task 3 Storage and Disk I/O Analysis

Task covers data plane (disk latency) and control plane (vSAN metadata) diagnostics. Panelists will ask: How do you distinguish storage array latency from network latency in vSAN? What is the significance of QAVG vs. DAVG? When is storage latency a VM issue vs. a platform issue?

Diagnose storage latency using esxtop disk views and correlate metrics with vSAN or backend storage health.

Step 1

From esxtop, press 'u' to switch to disk device view (physical/logical disk). Wait for the view to render.

Disk view shows columns: DAVG (device average latency), KAVG (kernel average latency), GAVG (guest average latency), QAVG (queue average latency), CMDS/s (commands per second), READS/s, WRITES/s, BUSYQ, ACTV (active commands).
Latency in esxtop is displayed in milliseconds. If DAVG is not visible, press 'f' to add it.
Step 2

Examine your storage device list. Identify the primary data store used by your lab VMs. Note the baseline DAVG. A healthy SSD-backed datastore should show DAVG < 5ms. HDD-backed should show DAVG < 10ms.

Baseline recorded: e.g., 'vSAN-datastore: DAVG=3.2ms, READS/s=450, WRITES/s=120, QAVG=1.1, ACTV=2'. This indicates healthy storage with low latency and moderate queue depth.
Cross-validate esxtop disk metrics with esxcli storage core device stats get (ESX has no iostat) and, for a vSAN datastore, with vsantop on the host.
Step 3

Introduce disk I/O stress: Run a heavy I/O workload on a guest VM (e.g., 'fio --name=randrw --ioengine=libaio --iodepth=32 --rw=randrw --bs=8k --runtime=300'). Monitor esxtop disk view in real-time.

Stressed state captured: e.g., 'vSAN-datastore: DAVG=45ms, QAVG=12.3, ACTV=28, READS/s=1200, WRITES/s=400'. DAVG jumped from 3.2ms to 45ms (14x increase), QAVG is high, ACTV is near queue capacity.
45ms is representative of a backend storage bottleneck. Compare to baseline to identify the magnitude of degradation.
Step 4

Interpret the scenario: DAVG=45ms against the Step 3 baseline of 3.2ms = ~14x storage latency spike. Analyze root cause. In vSAN, check: (a) Is this a vSAN rebuild scenario (rebuilding a failed disk/host)? (b) Are vSAN resync operations active? (c) Is the storage network saturated (vSAN uses dedicated network)?

Diagnostic analysis: 'DAVG is the device (HBA-to-storage) response time; 45 ms is far above the KB 344099 guidance that DAVG, KAVG and GAVG should not exceed 10 ms sustained. Next step: check vSAN metrics (Skyline Health, Resyncing Objects, vSAN performance views for backend versus network latency). If vSAN is healthy, suspect a degraded backend array, network congestion, or storage-target queue limits.'
Per Broadcom KB 344099: DAVG is the device response time per command, KAVG is the time the command spends in the VMkernel, and GAVG is what the guest sees — DAVG + KAVG = GAVG. If KAVG is a large share of GAVG, I/O is queuing in the VMkernel (queue depth, SIOC or adapter throttling), not at the device. On vSAN, read backend and network latency from the vSAN performance service (and the VM-level GAVG) rather than inferring it from device DAVG, consistent with lab vcp-support-09.
Step 5

If your lab uses vSAN, SSH to a vSAN-enabled host and check vSAN with vsantop (per-host vSAN performance), esxcli vsan health cluster list, esxcli vsan debug resync summary get (resync/rebalance activity) and vdq -q (device eligibility); cluster-level latency comes from the vSAN Performance Service in vCenter ('vsanstats' and 'vsan.stats' did not surface as ESX commands; RVC is deprecated in VCF 9.0, p.233).

vSAN metrics reveal: e.g., 'Disk group 1: Component resync in progress (15% complete). Physical disk latency: 4ms avg. Network latency to peer hosts: 1ms avg.' This explains the DAVG spike: vSAN is rebalancing components, consuming I/O capacity.
If vSAN is not available in your lab, document this as a prerequisite limitation and focus on generic disk latency interpretation.
Step 6

Press 'v' (VM disk view) to see disk latency from the VM perspective (GAVG). Identify which VM is experiencing the worst latency. GAVG = DAVG + KAVG (KB 344099), so GAVG is always at least DAVG; the gap is KAVG — time spent in the VMkernel.

VM disk view: e.g., 'web-server-01: GAVG=48ms, DAVG=45ms, READS/s=800.' The 3 ms difference (48-45) is KAVG — the time the commands spent in the VMkernel. A small KAVG is normal; the problem here is the device latency.
If GAVG is well above DAVG (e.g., GAVG=100ms, DAVG=45ms), KAVG is high: look at VMkernel queuing — QAVG, device/adapter queue depth, an IOPS limit or SIOC throttling — rather than at the guest. GAVG does not include guest-OS time.
Step 7

Recommend storage remediation based on your analysis: (a) For vSAN rebuild scenarios, wait for rebuild to complete and re-measure latency. (b) For persistent high DAVG, increase storage capacity (add disks to vSAN disk groups, upgrade SAN array). (c) Check network for vSAN: ensure the vSAN network VLAN chosen in your design is not congested (there is no default vSAN VLAN number). (d) Consider QoS policies to throttle non-critical I/O during peak periods.

Remediation plan: 'Step 1: Wait for vSAN rebuild to complete (est. 2 hours). Step 2: Re-measure DAVG to confirm recovery. Step 3: If DAVG remains > 10ms, request 2 additional storage disks per host to increase vSAN IOPS capacity. Step 4: Implement VM-level I/O QoS (1000 IOPS cap on reporting VM, 5000 IOPS cap on production VM).'
Remediation for storage is often slow (hardware procurement, vSAN rebuilds). In a VCDX interview, emphasize patience and data-driven decision-making.

Validation Gate

Check: Storage I/O analysis complete with diagnosed latency spike and remediation recommendation

Expected: Lab notes document esxtop disk metrics, baseline vs. stressed state comparison, vSAN-specific diagnostics (if applicable), and tiered remediation plan

Common Errors

Confusing DAVG with GAVG. DAVG is device-level latency (what the storage returns); GAVG is what the VM observes at the vSCSI layer, and GAVG = DAVG + KAVG. High GAVG with low DAVG means KAVG is high — the VMkernel is queuing I/O (queue depth, throttling) — not a guest-OS or storage-array issue.
Fix: Always check all three. If GAVG is far above DAVG, read KAVG and QAVG: raise the adapter/device queue depth where supported, look for IOPS limits or SIOC throttling, and keep every one of DAVG/KAVG/GAVG under 10 ms sustained (KB 344099).
Ignoring QAVG. A high DAVG with low QAVG suggests the storage is busy but not backlogged—likely a transient spike. High QAVG with high DAVG indicates sustained contention and backlog.
Fix: QAVG trend is more important than DAVG snapshot. Monitor QAVG over time to distinguish spikes from sustained load.
Not considering vSAN-specific latency sources. On vSAN, components and replicas sit on other hosts, so the vSAN network affects what the VM sees. Read the vSAN performance service (backend and network views) and the VM's GAVG before blaming a device.
Fix: In vSAN environments, always check network latency first (vmkping -I <vSAN vmk> between vSAN hosts, with -s 8972 -d to prove the MTU). If network is healthy, then suspect disk or rebalance operations.

Task 4 Host-Level Analysis and Batch Mode Trending

Task emphasizes systematic trending and documentation. Panelists will ask: How do you define a healthy baseline? What time window is needed for statistically valid trending? How would you design a monitoring strategy based on batch data? Can you spot anomalies in trend data?

Analyze host-level resource saturation and establish performance baselines using esxtop batch mode for trend analysis and documentation.

Step 1

Return to esxtop interactive CPU view ('c'). Observe the host kernel metrics (typically listed first or in a specific row). Note host %USED and %SYS. Host %USED near 85%+ indicates resource saturation; a %SYS well above your baseline (sustained) indicates high system overhead (drivers, interrupts, page-table management) — no fixed threshold is published.

Host metrics captured: e.g., 'Host CPU: %USED=82%, %SYS=7.5%, %IDLE=10.5%'. The 82% utilization is near saturation; further VM consolidation risks contention. The 7.5% system overhead is within acceptable range.
Host %USED includes all VMs plus hypervisor overhead. Unlike VM %USED, it should not exceed 85% under normal operations; leave headroom for spikes.
Step 2

Interpret the scenario: Host %USED=85%, %SYS=8%. The host is near saturation, and system overhead is climbing. Possible causes: (a) Too many VMs for the hardware. (b) Driver inefficiency (NIC driver, storage controller driver). (c) Memory pressure causing high page-table walks. (d) Interrupt storms from overactive devices.

Analysis: 'Host approaching saturation. The 8% system overhead is elevated. Recommend: (1) Check driver versions (NIC, storage). (2) Review memory pressure (if memory is ballooned/swapping, page-table overhead increases). (3) Reduce VM count by 2-3 non-critical VMs or move them with DRS — deeper C-states save power and add wake-up latency; they do not relieve CPU load.'
Don't confuse saturation with utilization. 85% host %USED is not dangerous if load is predictable. 95% is dangerous if spikes occur.
Step 3

Exit esxtop interactive mode ('q'). Prepare to run esxtop in batch mode for trending. Batch mode captures metrics at regular intervals and outputs CSV data suitable for spreadsheet analysis.

esxtop exited cleanly. You're back at the ESXi shell prompt.
Batch mode output is CSV; you'll import it into a spreadsheet in the next step.
Step 4

Run esxtop in batch mode with the command: 'esxtop -b -d 5 -n 720 > /tmp/perf.csv'. This captures 720 samples at 5-second intervals (60 minutes of data). Let it run to completion.

esxtop batch mode runs silently. After ~60 minutes, /tmp/perf.csv will contain 720 rows of CSV data with timestamps and all esxtop metrics (CPU, memory, disk, network, virtual machine uptime).
Use '-n 720' for 1 hour. For daily trending, use '-n 17280' (24 hours). For larger captures, redirect to a host with ample disk space (e.g., /vmfs/volumes/datastore).
Step 5

Retrieve the CSV file to your workstation via SCP or FTP. Example: 'scp root@your-esxi-ip:/tmp/perf.csv ./perf-baseline.csv'. Open the file in a spreadsheet (Excel, Google Sheets, LibreOffice Calc).

CSV file downloaded to local workstation. When opened in a spreadsheet, rows represent time samples, columns represent metrics (timestamps, %USED, %RDY, %SYS, MCTLSZ, DAVG, etc.). Data is ready for analysis.
The first row may contain header metadata from esxtop; import carefully to avoid treating metadata as data.
Step 6

In the spreadsheet, create a chart of key metrics over time: (1) Plot host %USED, %RDY, %SYS as a multi-series line chart. (2) Identify peak utilization and note the timestamp. (3) Look for trends: Is utilization increasing over the hour, or stable? (4) Calculate moving average (15-minute window) to smooth spikes.

Trend chart generated showing: Host %USED varies between 50-85% over the hour, with peak at 10:35 AM. %RDY spikes to 18% around peak utilization, then returns to baseline. %SYS remains stable at 6-8% throughout. Moving average smooths out sub-minute spikes.
Trends reveal patterns. If %USED increases linearly over the hour, the workload is growing; expect saturation. If %USED spikes at regular intervals, there's a periodic job (backups, reports) that contends for resources.
Step 7

Create a performance baseline document in your lab notes (or as a formal write-up). Include: (1) Host specifications (CPU sockets, cores, RAM). (2) Baseline metrics during steady-state (low load): %USED, %RDY, %SYS, DAVG, MCTLSZ. (3) Baseline metrics during peak load: same metrics at 75-85% utilization. (4) Thresholds for alerting (when to escalate).

Baseline document: 'Host: 2x Intel Xeon 16-core CPUs, 256GB RAM. Steady-state: %USED=45%, %RDY=3%, DAVG=4ms, MCTLSZ=0GB. Peak load (85% utilization): %RDY should stay < 10%, DAVG < 15ms, MCTLSZ < 5GB for health. Alert threshold: %RDY > 15% or DAVG > 20ms triggers investigation.' These are example per-tier thresholds you set for your workloads; the lab's reference point is KB 304594 (%READY under 5% under normal operating conditions).
This baseline document is your reference for troubleshooting. Store it in your lab documentation for future comparisons.
Step 8

Document batch mode methodology in your notes: When to run, how long to run, what to measure, and how to interpret results. Include a sample command and sample output interpretation.

Methodology documented: 'Run esxtop -b -d 5 -n 17280 to capture 24-hour trends. Import CSV to spreadsheet. Calculate hourly averages and identify peak utilization window. If peak utilization > 85%, correlate with workload scheduling (backups, reports) and plan load balancing or capacity upgrade.'
This methodology is a VCDX-style artifact: it demonstrates that you approach performance analysis systematically, not reactively.

Validation Gate

Check: Batch mode trending complete with baseline document and trend analysis interpretation

Expected: Lab notes contain esxtop batch command, CSV trend analysis, performance baseline document with thresholds, and methodology for future trending

Common Errors

Running batch mode for insufficient time. 5 minutes of data is not enough to identify patterns. 60+ minutes is necessary for meaningful trending.
Fix: Always run at least 1 hour of batch mode; for weekly trends, capture 24 hours. Use '-d 10' or '-d 30' for longer captures (reduces file size).
Forgetting to account for workload timing. If batch mode captures a time window when no workloads are running, the baseline is artificially low.
Fix: Run batch mode during normal business hours when your lab VMs are operational. Document the time window and workload characteristics in your baseline report.
Misinterpreting CSV column order. esxtop batch mode output column order is not intuitive; without headers, metrics are hard to identify.
Fix: Open the CSV in a spreadsheet and manually add headers based on esxtop interactive view. Use the interactive esxtop field help ('f' in each panel) for column descriptions — the batch header row already names every counter.

Final Validation

Lab completed when all 4 tasks are executed, metrics are documented, and remediation recommendations are justified.

✓ CPU analysis with high-contention scenario analyzed and remediation recommended → Task 1 complete: %RDY, %CSTP metrics interpreted; vCPU adjustment documented with rationale

✓ Memory pressure scenario analyzed with reclamation metrics documented → Task 2 complete: MCTLSZ, SWCUR, ZIP metrics documented; memory pressure assessed; remediation ranked by priority

✓ Storage I/O latency diagnosed with baseline and stress comparisons → Task 3 complete: DAVG, KAVG, GAVG metrics documented; vSAN health checked (if applicable); storage remediation planned

✓ Host-level baseline established via batch mode trending → Task 4 complete: esxtop batch CSV captured, trending chart created, baseline document produced with alert thresholds

Cleanup / Restore

• Remove /tmp/perf.csv from ESXi host (or retain for 30 days as audit trail)

• Revert any VM I/O stress workloads (stop fio, stress-ng)

• Return host CPU/memory to baseline utilization (if VMs were intentionally overloaded)

• Restore baseline snapshots if any changes were made to VM configurations

Design Reflection (VCDX)

This lab emphasizes diagnostic methodology and evidence-based decision-making—core VCDX competencies. Panelists will probe: (1) How do you validate that your esxtop diagnosis matches actual VM performance issues? (2) When do you recommend quick fixes vs. architectural changes? (3) How do you balance performance tuning against operational risk? (4) Can you articulate the trade-offs between CPU overcommit and power efficiency? The RCAR framework below provides structure for panel discussion.

Requirements

  • ESXi host(s) with operational VMs and I/O workloads
  • SSH/CLI access to ESXi host
  • Spreadsheet software for CSV analysis and charting
  • Lab VMs running representative workloads (CPU, memory, disk I/O)
  • vSAN deployment (optional, for advanced storage metrics) or iSCSI/NFS datastore
  • Baseline performance data for the lab environment (to establish 'healthy' thresholds)

Constraints

  • esxtop must be run directly on ESXi host; remote monitoring tools (VCF Operations) provide different metrics and context
  • Batch mode CSV captures are host-specific; trending across multiple hosts requires aggregation logic outside esxtop
  • CPU/memory contention metrics are OS-kernel-dependent; different ESXi versions may report slightly different values
  • vSAN-specific metrics (rebuild status, rebalance operations) require vSAN Skyline Health or vSAN Management REST API; not all metrics available in esxtop
  • Interactive esxtop view requires SSH terminal session; cannot be automated or scheduled without batch mode

Assumptions

  • Lab VMs are configured with realistic resource allocations (not over-provisioned with 100 vCPUs on 16-core host)
  • Storage backend is functioning normally during the lab (no storage array failures masking remediation)
  • Network connectivity between ESXi hosts and vSAN network is stable (for vSAN trending)
  • Lab operator has basic Linux/Windows CLI knowledge to trigger I/O workloads and interpret OS-level metrics
  • Remediation recommendations assume you have approval to modify VM configurations (vCPU count, memory allocation, DRS settings)
  • Baseline 'healthy' values are organization-specific; lab uses the Broadcom KB 304594 guide (%READY under 5% in normal operation) as reference

Risks

  • Risk: Over-aggressive VM scaling during CPU contention analysis may cause application downtime if vCPU is reduced too much — Mitigation: In a VCDX interview, outline the plan and revert immediately if issues emerge. Do not apply changes without supervisor approval in production.
  • Risk: Batch mode trending captures can consume significant disk space if run for extended periods (e.g., 24-hour capture = 5-10 MB CSV) — Mitigation: Plan storage; use '-d 30' (30-second intervals) instead of '-d 5' for longer captures. Compress/delete old CSV files after analysis.
  • Risk: Memory pressure scenario (MCTLSZ > 20%, SWCUR > 0) may trigger emergency performance degradation or guest OS OOM (Out of Memory) kill — Mitigation: In a lab environment, monitor guest OS closely during memory stress. Have snapshot/revert strategy ready if guest becomes unresponsive.
  • Risk: Storage I/O stress (fio) may impact concurrent workloads if run on shared datastore; can cause noisy neighbor contention — Mitigation: Run I/O stress on a dedicated VM on dedicated datastore, or run during off-hours. Document timing in lab notes.

Self-Assessment Discussion Prompts

  1. Walk me through how you decided what %RDY per vCPU counts as 'concerning contention' (the VM row sums all vCPUs). Where does that number come from? Would you apply the same threshold to a DSS workload vs. an OLTP workload?
  2. You recommended reducing vCPU from 8 to 6. How would you validate that this change improves performance without harming application throughput? What metrics would you monitor post-change?
  3. In the memory pressure scenario (MCTLSZ=32GB, SWCUR=1.2GB), you recommended adding RAM as Priority 1. But hardware upgrades take weeks. What is your Priority 2 action to buy time while procurement happens?
  4. You observed DAVG spiking from 5ms to 45ms during I/O stress. Before blaming the storage array, what hypervisor/network factors would you investigate? How would you distinguish storage latency from network latency in a vSAN environment?
  5. Batch mode gave you 60 minutes of baseline data. Is that enough to define 'healthy' thresholds for the next quarter? What time window would you recommend for production baselining, and why?
  6. A VM shows %RDY=15%, DAVG=8ms, but the application team says performance is fine. How do you reconcile esxtop metrics with user-perceived performance? What layer are you missing?
  7. You're being asked to consolidate 20 more VMs onto this cluster. Based on your baseline, what's the maximum consolidated VMs this cluster can handle before %RDY exceeds the threshold you set for this tier (KB 304594 reference: under 5% under normal operating conditions)?

Extensions

Multi-Host Trending and Anomaly Detection

Extend the lab to capture esxtop batch data from 3+ ESXi hosts simultaneously. Aggregate CSV data into a unified spreadsheet, calculate cluster-level CPU and memory utilization, and identify outlier hosts (e.g., one host consistently near 85% while others idle at 45%). Create alert rules: if one host exceeds 80% while cluster average is 50%, trigger DRS rebalancing. This extension demonstrates cluster-level capacity planning and load-balancing logic.

Performance Regression Testing with VCF Operations

Compare manual esxtop analysis with VCF Operations metrics. Use VCF Operations to generate the same baseline charts as your esxtop CSV analysis, then overlay recommendations. This extension bridges ad-hoc troubleshooting (esxtop) with continuous monitoring (VCF Operations), demonstrating how to design a monitoring strategy based on esxtop findings.

Remediation Validation and A/B Testing

Apply the vCPU reduction (Task 1) or memory balancing (Task 2) changes in a controlled manner. Re-run esxtop batch mode for 60 minutes post-change and compare charts. Quantify the improvement: 'vCPU reduction from 8 to 6 reduced %RDY by 12 percentage points, reducing VM schedule latency by ~300 microseconds per operation.' This extension demonstrates rigor in validating that your diagnosis translates to measurable improvement.

⚠ Known Pitfalls

Treating esxtop snapshots as gospel truth. A single 30-second snapshot showing %RDY=20% may be a transient spike, not a sustained contention condition. Always trend over time (batch mode) before making capacity decisions.
Resolution: Use esxtop interactive for diagnosis, batch mode for validation. Require 60+ minutes of trending before declaring a problem 'confirmed.'
Ignoring the guest OS perspective. esxtop shows hypervisor-level metrics, but the guest OS may have independent memory paging, CPU throttling, or I/O queuing. A VM can appear healthy at the hypervisor level but slow to users due to guest-level issues.
Resolution: Always SSH into guest VMs and cross-check: Linux 'vmstat 1 5', Windows 'Get-WmiObject Win32_PerfFormattedData_PerfOS_Processor'. If guest metrics differ from esxtop, the bottleneck is inside the guest.
Applying one-size-fits-all thresholds. The acceptable %RDY threshold for a batch job (12-hour reporting window) is different from an interactive OLTP workload (< 3% %RDY). Thresholds must be context-aware.
Resolution: Define thresholds per workload tier, for example OLTP < 5% %RDY, batch < 15% %RDY, development < 25% %RDY (example values you set; KB 304594 gives under 5% under normal operating conditions). Document SLA requirements alongside metrics.
Assuming correlation = causation. High DAVG during vSAN rebuild is correlated with rebuild activity, but the root cause of slow storage may be elsewhere (network, disk failure). Always isolate variables when diagnosing.
Resolution: Use Traceflow (NSX) or network packet capture (tcpdump) to validate network path. Use the vSAN Resyncing Objects view (or esxcli vsan debug resync summary get) and Skyline Health to confirm a vSAN rebuild is actually the cause, not a coincidence.

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.