Academy/vSphere Foundation 9.0 Support (2V0-18.25)/Lab: Monitor vSAN Resync After Host Failure
This lab targets VCF 9.0

Lab: Monitor vSAN Resync After Host Failure

VCF 9.0Intermediatevcp-foundation⏱ 150 min

VVF 9.0 support lab (2V0-18.25). Built for a Holodeck VVF instance.

Objectives

  • Simulate a host failure and observe vSphere HA and vSAN behaviour (Absent components, Object Repair Timer)
  • Monitor resync to completion and measure workload latency during it
  • Decide when throttling (KB 326830) or rebalance settings are appropriate

Prerequisites

Holodeck VVF 9.0 instance with at least four nested hosts in the vSAN cluster (so a spare fault domain exists for the rebuild) and a running test workload

Lab Environment

Holodeck vSphere Foundation 9.0 instance (New-HoloDeckInstance -VVF; -vSANMode OSA by default, ESA where the physical layer supports it): vCenter managing one vSAN cluster of nested ESX hosts, plus VCF Operations. VVF has no SDDC Manager and no NSX.

Credentials

SystemUsernamePassword
vCenteradministrator@vsphere.localHolodeck documented default (Post Deployment table)
VCF OperationsadminHolodeck documented default (Post Deployment table)
Nested ESX hostsrootHolodeck documented default (Post Deployment table)

Tasks

Task 1 Host failure, repair timer and resync

Monitoring vSAN resync after host failure tests understanding of automatic data protection, rebuild timing, and impact on workload performance. This is a critical operational scenario that every VCF administrator must handle confidently.

Simulate host failure, observe vSAN resync in action, monitor progress, verify data accessibility.

Step 1

vCenter → Cluster → Monitor → vSAN → Health (baseline green)

Step 2

Identify one vSAN host to fail

Step 3

Simulate failure: shutdown host or disconnect network

Step 4

vSphere HA detects the failure: with default settings a host that receives no heartbeats for 12 seconds pings its isolation addresses, and other hosts treat it as failed after 15 seconds of isolation (VCF 9.0.1 docs p.7594)

Step 5

Skyline Health turns yellow: objects show reduced availability and the failed host's components are Absent (vSAN treats a powered-off or unreachable host as a possibly temporary failure — VCF 9.0.1 docs p.1655, p.1831), not Degraded.

Step 6

Note that nothing rebuilds yet: Absent components wait for the Object Repair Timer (default 60 minutes). Either wait it out, or — lab only — lower the timer under Cluster → Configure → vSAN → Services → Advanced Options, then watch Cluster → Monitor → vSAN → Resyncing Objects (objects, bytes left, ETA).

Step 7

Monitor network bandwidth consumed during the resync (esxtop network view on the remaining hosts; vSAN performance views for resync throughput).

Step 8

Verify VMs continue to run (data accessible despite failure)

Step 9

Host recovers: power it on and wait for it to rejoin. If it returns within the repair timer, only its stale components resynchronise (vSAN resyncs the deltas); after the timer has expired vSAN may already be rebuilding those components on other hosts, and the returning host's copies are discarded.

Step 10

vSAN health returns to green

Validation Gate

Check: After simulating a host failure: confirm the components are Absent and that rebuilding waits for the Object Repair Timer; then either wait or lower the timer (lab only), monitor Resyncing Objects to completion, confirm all objects return to compliant, and measure workload I/O impact during the resync

Expected: No rebuild before the Object Repair Timer (default 60 min) expires; afterwards Resyncing Objects shows decreasing objects and bytes left to resync. After completion: all objects compliant with full FTT. Workload I/O latency returns to the pre-failure baseline.

Common Errors

Changing rebalance settings during active resync
Fix: vSAN rebuilds components automatically after a failure, and in 9.0 it also rebalances automatically by default (Configure → vSAN → Services → Advanced Options → Automatic Rebalance, 30% threshold — VCF 9.0.1 docs p.1804). Leave rebalance settings alone during a resync; lower the threshold only for a planned rebalance in off-peak hours.
Not monitoring resync impact on workload I/O latency
Fix: vSAN resync consumes disk and network bandwidth. Resync competes with workload I/O. Monitor GAVG in esxtop and the vSAN performance charts. If a VM stops responding because of resync latency, throttle resync IOPS as described in KB 326830 (VCF 9.0.1 docs p.1803).
Panicking when resync ETA shows hours instead of minutes
Fix: Resync duration depends on data volume and available bandwidth. A large rebuild can take hours; that is expected and depends on data volume and network bandwidth. The ETA updates dynamically — initial estimates are often pessimistic. Monitor progress (components remaining, bytes remaining) rather than ETA.
Returning a failed host to the cluster before investigating root cause
Fix: After a host failure and successful vSAN resync, don't blindly reboot and rejoin the host. Investigate: why did it fail? Hardware error (check iLO/iDRAC logs), ESXi crash (check PSOD core dump), power issue (check UPS logs). Returning a host with a persistent hardware issue causes repeated failures and unnecessary resyncs.

Final Validation

Gate: every check below passes with the evidence recorded in the lab log.

✓ Failure phase → Components Absent; HA restarts the affected VMs; no rebuild before the repair timer

✓ Resync → Resyncing Objects decreases to 0

✓ Compliance → All objects compliant with full FTT

✓ Workload → Latency returns to the pre-failure baseline

Cleanup / Restore

• Power the nested host back on

• Restore the Object Repair Timer to 60 minutes if it was lowered

Design Reflection (VCDX)

Host failure and vSAN resync is a likely VCDX defense scenario. Know the timeline: HA failure detection (12 s without heartbeats, then 15 s isolation window, p.7594) → vSphere HA VM restart (2-5 min) → components Absent, Object Repair Timer (60 min) → rebuild/resync (varies by data volume) — and the Absent versus Degraded distinction (Degraded rebuilds immediately). Be able to walk through each phase.

Requirements

  • Monitor vSAN resync after host failure
  • Understand resync timing and impact on workloads
  • Investigate host failure root cause before returning host to cluster

Constraints

  • Do not change rebalance settings or start disk operations during active resync
  • Resync consumes bandwidth — workload I/O may be impacted
  • CLOMD repair delay is configurable but 60 min is default for good reason

Assumptions

  • Cluster has N+1 capacity for host failure
  • vSAN network bandwidth is sufficient for resync within acceptable timeframe

Risks

  • Data loss from second failure during resync
  • Persistent hardware issue causing repeated failures if host returned without investigation

⚠ Known Pitfalls

Changing rebalance settings or starting other disk operations during an active resync — it adds I/O and extends the resync.
Returning a failed host without root cause investigation — the same failure will recur.

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.