Lab: Monitor vSAN Resync After Host Failure
Objectives
- Simulate a host failure and observe vSphere HA and vSAN behaviour (Absent components, Object Repair Timer)
- Monitor resync to completion and measure workload latency during it
- Decide when throttling (KB 326830) or rebalance settings are appropriate
Prerequisites
Holodeck VVF 9.0 instance with at least four nested hosts in the vSAN cluster (so a spare fault domain exists for the rebuild) and a running test workload
Lab Environment
Holodeck vSphere Foundation 9.0 instance (New-HoloDeckInstance -VVF; -vSANMode OSA by default, ESA where the physical layer supports it): vCenter managing one vSAN cluster of nested ESX hosts, plus VCF Operations. VVF has no SDDC Manager and no NSX.
Credentials
| System | Username | Password |
|---|---|---|
| vCenter | administrator@vsphere.local | Holodeck documented default (Post Deployment table) |
| VCF Operations | admin | Holodeck documented default (Post Deployment table) |
| Nested ESX hosts | root | Holodeck documented default (Post Deployment table) |
Tasks
Task 1 Host failure, repair timer and resync
Simulate host failure, observe vSAN resync in action, monitor progress, verify data accessibility.
vCenter → Cluster → Monitor → vSAN → Health (baseline green)
Identify one vSAN host to fail
Simulate failure: shutdown host or disconnect network
vSphere HA detects the failure: with default settings a host that receives no heartbeats for 12 seconds pings its isolation addresses, and other hosts treat it as failed after 15 seconds of isolation (VCF 9.0.1 docs p.7594)
Skyline Health turns yellow: objects show reduced availability and the failed host's components are Absent (vSAN treats a powered-off or unreachable host as a possibly temporary failure — VCF 9.0.1 docs p.1655, p.1831), not Degraded.
Note that nothing rebuilds yet: Absent components wait for the Object Repair Timer (default 60 minutes). Either wait it out, or — lab only — lower the timer under Cluster → Configure → vSAN → Services → Advanced Options, then watch Cluster → Monitor → vSAN → Resyncing Objects (objects, bytes left, ETA).
Monitor network bandwidth consumed during the resync (esxtop network view on the remaining hosts; vSAN performance views for resync throughput).
Verify VMs continue to run (data accessible despite failure)
Host recovers: power it on and wait for it to rejoin. If it returns within the repair timer, only its stale components resynchronise (vSAN resyncs the deltas); after the timer has expired vSAN may already be rebuilding those components on other hosts, and the returning host's copies are discarded.
vSAN health returns to green
Validation Gate
Check: After simulating a host failure: confirm the components are Absent and that rebuilding waits for the Object Repair Timer; then either wait or lower the timer (lab only), monitor Resyncing Objects to completion, confirm all objects return to compliant, and measure workload I/O impact during the resync
Expected: No rebuild before the Object Repair Timer (default 60 min) expires; afterwards Resyncing Objects shows decreasing objects and bytes left to resync. After completion: all objects compliant with full FTT. Workload I/O latency returns to the pre-failure baseline.
Common Errors
Final Validation
Gate: every check below passes with the evidence recorded in the lab log.
✓ Failure phase → Components Absent; HA restarts the affected VMs; no rebuild before the repair timer
✓ Resync → Resyncing Objects decreases to 0
✓ Compliance → All objects compliant with full FTT
✓ Workload → Latency returns to the pre-failure baseline
Cleanup / Restore
• Power the nested host back on
• Restore the Object Repair Timer to 60 minutes if it was lowered
Design Reflection (VCDX)
Host failure and vSAN resync is a likely VCDX defense scenario. Know the timeline: HA failure detection (12 s without heartbeats, then 15 s isolation window, p.7594) → vSphere HA VM restart (2-5 min) → components Absent, Object Repair Timer (60 min) → rebuild/resync (varies by data volume) — and the Absent versus Degraded distinction (Degraded rebuilds immediately). Be able to walk through each phase.
Requirements
- Monitor vSAN resync after host failure
- Understand resync timing and impact on workloads
- Investigate host failure root cause before returning host to cluster
Constraints
- Do not change rebalance settings or start disk operations during active resync
- Resync consumes bandwidth — workload I/O may be impacted
- CLOMD repair delay is configurable but 60 min is default for good reason
Assumptions
- Cluster has N+1 capacity for host failure
- vSAN network bandwidth is sufficient for resync within acceptable timeframe
Risks
- Data loss from second failure during resync
- Persistent hardware issue causing repeated failures if host returned without investigation