Academy/vSphere Foundation 9.0 Support (2V0-18.25)/Stretch lab (outside the 2V0-18.25 blueprint): Troubleshoot a Slow Kubernetes Application
This lab targets VCF 9.0

Stretch lab (outside the 2V0-18.25 blueprint): Troubleshoot a Slow Kubernetes Application

VCF 9.0Intermediatevcp-foundation⏱ 135 min

Stretch lab: outside the 2V0-18.25 (VVF Support) blueprint, which has no Kubernetes objective; kept for full-stack troubleshooting practice (D128).

Objectives

  • Diagnose a slow application across the stack: pod, worker node VM, ESX host, vSAN
  • Correlate Kubernetes events with vCenter and VCF Operations events for the same window
  • Recommend a fix at the layer that holds the root cause

Prerequisites

Holodeck VVF 9.0 instance with a Supervisor on VDS networking and one VKS cluster running a test application (see vvf-admin-10 and vvf-admin-12)

Lab Environment

Holodeck vSphere Foundation 9.0 instance (New-HoloDeckInstance -VVF; -vSANMode OSA by default, ESA where the physical layer supports it): vCenter managing one vSAN cluster of nested ESX hosts, plus VCF Operations. VVF has no SDDC Manager and no NSX.

Credentials

SystemUsernamePassword
vCenteradministrator@vsphere.localHolodeck documented default (Post Deployment table)
VCF OperationsadminHolodeck documented default (Post Deployment table)
Nested ESX hostsrootHolodeck documented default (Post Deployment table)

Tasks

Task 1 Full-stack diagnosis of a slow application

Kubernetes pod performance troubleshooting on VCF Supervisor requires understanding the full stack: pod → container runtime → worker node VM → ESXi host → vSAN/network. Performance issues can originate at any layer.

Diagnose pod performance problem: app is slow, determine if vSphere or K8s layer issue.

Step 1

Deploy the slow-app pod on the VKS cluster (or use an existing one)

Step 2

User reports: "App is slow, taking 10s per request"

Step 3

Check K8s layer: kubectl logs pod-name (see if errors)

Step 4

Check pod resources: kubectl describe pod, see CPU/memory requests

Step 5

Check node resources: kubectl top nodes (CPU/memory available?)

Step 6

Check PVC/storage: kubectl describe pvc shows binding, storage class and events (not latency); read latency from the vSAN performance charts for the backing datastore

Step 7

Check vSphere layer: esxtop over SSH on the worker VM's ESX host (%RDY, GAVG), or the VM's performance charts in vCenter

Step 8

Correlate: Is bottleneck vSAN latency, K8s resource contention, or app code?

Step 9

Make recommendation at the right layer: right-size requests/limits, add worker nodes or replicas, move the PVC to a policy/datastore with lower latency, or fix the app code (raising FTT adds replicas and write work, so it does not cut latency)

Validation Gate

Check: Diagnose a slow pod: check pod resource usage (kubectl top), worker node VM performance (esxtop on its host), vSAN datastore latency, and VDS/port-group connectivity. Identify root cause layer.

Expected: Root cause identified at specific layer: pod resource limits, worker node CPU/memory contention, vSAN I/O latency, or a VDS network issue (VVF has no NSX). Remediation applied at the correct layer.

Common Errors

Troubleshooting only inside the pod without checking the underlying infrastructure
Fix: K8s pod performance depends on the underlying VM (worker node) which depends on ESXi host resources. A pod with high CPU throttling may be caused by: pod resource limits too low, worker node VM CPU contention (check esxtop %RDY), or ESXi host over-commitment. Diagnose bottom-up: ESXi host → worker node VM → pod.
Not checking Kubernetes resource requests and limits
Fix: Pods with no CPU or memory requests or limits at all are BestEffort and are evicted first under pressure; a container with limits but no requests gets requests equal to its limits, so a pod whose containers all set CPU and memory limits this way is Guaranteed; a pod is Burstable when at least one request or limit is set but the Guaranteed criteria are not met (Kubernetes Pod QoS documentation). Pods without limits can consume unlimited resources, starving other pods. Check: kubectl describe pod <name> for resource requests/limits. Best practice: always set both requests (guaranteed minimum) and limits (maximum allowed).
Ignoring persistent volume performance when diagnosing pod latency
Fix: Pods using persistent volumes (PVCs) on vSAN may experience I/O latency from: vSAN resync activity, storage policy mismatch (RAID-1 vs RAID-5 for the workload type), or datastore congestion. Check vSAN performance metrics for the specific datastore backing the PVC.
Not correlating pod events with infrastructure events
Fix: A pod restart may correlate with: ESXi host maintenance mode (VM migrated, pod restarted), vSAN resync (I/O latency spike caused pod health check failure), or a VDS/uplink interruption (pod lost connectivity). Check kubectl events alongside vCenter events and VCF Operations timeline for the same time window.

Final Validation

Gate: every check below passes with the evidence recorded in the lab log.

✓ Kubernetes layer → Pod QoS class, requests/limits and node pressure recorded

✓ vSphere layer → Worker VM %RDY and datastore latency recorded from the host's esxtop or vCenter performance charts

✓ Root cause → One layer named with evidence and a fix at that layer

Cleanup / Restore

• Delete the test application

Design Reflection (VCDX)

Container workload troubleshooting on VCF demonstrates full-stack operational knowledge. VCDX panelists may present a Kubernetes performance scenario to test whether you can diagnose across VM, storage, and network layers — not just within Kubernetes.

Requirements

  • Troubleshoot pod performance across the full stack
  • Correlate Kubernetes events with infrastructure events
  • Understand persistent volume performance on vSAN

Constraints

  • Pod resource requests/limits must be configured for predictable performance
  • Worker node VM sizing constrains pod density
  • vSAN performance affects all PVC-backed pods

Assumptions

  • kubectl access is available for Kubernetes diagnostics
  • VCF Operations and vCenter provide correlated infrastructure metrics

Risks

  • Misdiagnosing pod-level symptom when root cause is infrastructure
  • Pod density exceeding worker node VM capacity causing noisy neighbor issues

⚠ Known Pitfalls

Diagnosing only inside Kubernetes when the root cause is ESXi host contention or vSAN latency — always check the full stack.
Running pods without resource requests and limits — unpredictable performance and eviction behavior during resource pressure.

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.