Stretch lab (outside the 2V0-18.25 blueprint): Troubleshoot a Slow Kubernetes Application
Objectives
- Diagnose a slow application across the stack: pod, worker node VM, ESX host, vSAN
- Correlate Kubernetes events with vCenter and VCF Operations events for the same window
- Recommend a fix at the layer that holds the root cause
Prerequisites
Holodeck VVF 9.0 instance with a Supervisor on VDS networking and one VKS cluster running a test application (see vvf-admin-10 and vvf-admin-12)
Lab Environment
Holodeck vSphere Foundation 9.0 instance (New-HoloDeckInstance -VVF; -vSANMode OSA by default, ESA where the physical layer supports it): vCenter managing one vSAN cluster of nested ESX hosts, plus VCF Operations. VVF has no SDDC Manager and no NSX.
Credentials
| System | Username | Password |
|---|---|---|
| vCenter | administrator@vsphere.local | Holodeck documented default (Post Deployment table) |
| VCF Operations | admin | Holodeck documented default (Post Deployment table) |
| Nested ESX hosts | root | Holodeck documented default (Post Deployment table) |
Tasks
Task 1 Full-stack diagnosis of a slow application
Diagnose pod performance problem: app is slow, determine if vSphere or K8s layer issue.
Deploy the slow-app pod on the VKS cluster (or use an existing one)
User reports: "App is slow, taking 10s per request"
Check K8s layer: kubectl logs pod-name (see if errors)
Check pod resources: kubectl describe pod, see CPU/memory requests
Check node resources: kubectl top nodes (CPU/memory available?)
Check PVC/storage: kubectl describe pvc shows binding, storage class and events (not latency); read latency from the vSAN performance charts for the backing datastore
Check vSphere layer: esxtop over SSH on the worker VM's ESX host (%RDY, GAVG), or the VM's performance charts in vCenter
Correlate: Is bottleneck vSAN latency, K8s resource contention, or app code?
Make recommendation at the right layer: right-size requests/limits, add worker nodes or replicas, move the PVC to a policy/datastore with lower latency, or fix the app code (raising FTT adds replicas and write work, so it does not cut latency)
Validation Gate
Check: Diagnose a slow pod: check pod resource usage (kubectl top), worker node VM performance (esxtop on its host), vSAN datastore latency, and VDS/port-group connectivity. Identify root cause layer.
Expected: Root cause identified at specific layer: pod resource limits, worker node CPU/memory contention, vSAN I/O latency, or a VDS network issue (VVF has no NSX). Remediation applied at the correct layer.
Common Errors
Final Validation
Gate: every check below passes with the evidence recorded in the lab log.
✓ Kubernetes layer → Pod QoS class, requests/limits and node pressure recorded
✓ vSphere layer → Worker VM %RDY and datastore latency recorded from the host's esxtop or vCenter performance charts
✓ Root cause → One layer named with evidence and a fix at that layer
Cleanup / Restore
• Delete the test application
Design Reflection (VCDX)
Container workload troubleshooting on VCF demonstrates full-stack operational knowledge. VCDX panelists may present a Kubernetes performance scenario to test whether you can diagnose across VM, storage, and network layers — not just within Kubernetes.
Requirements
- Troubleshoot pod performance across the full stack
- Correlate Kubernetes events with infrastructure events
- Understand persistent volume performance on vSAN
Constraints
- Pod resource requests/limits must be configured for predictable performance
- Worker node VM sizing constrains pod density
- vSAN performance affects all PVC-backed pods
Assumptions
- kubectl access is available for Kubernetes diagnostics
- VCF Operations and vCenter provide correlated infrastructure metrics
Risks
- Misdiagnosing pod-level symptom when root cause is infrastructure
- Pod density exceeding worker node VM capacity causing noisy neighbor issues
⚠ Known Pitfalls
References
- Kubernetes documentation — Pod Quality of Service classesTier 1 — Official
- VCF 9.0 documentation — vSphere Supervisor and VKSTier 1 — Official