Academy/VCF 9.0 Support (2V0-15.25)/vSAN Disk Failure Recovery
This lab targets VCF 9.0

vSAN Disk Failure Recovery

VCF 9.0Intermediatevcp-foundationvcp-support⏱ 120 min

Comprehensive vSAN ESA disk failure diagnosis, recovery, and capacity impact analysis.

Objectives

  • Interpret vSAN component failure states (Absent, Degraded, Out of date) and the Object Repair Timer
  • Diagnose disk failures using esxcli vsan, vSAN health checks, and smart.log analysis
  • Execute disk removal and replacement procedures in vSAN ESA storage pools
  • Calculate resync impact on cluster capacity and VM performance
  • Configure vSAN proactive rebalancing and rebuild rate tuning
  • Document disk failure recovery using RCAR methodology

Prerequisites

VCF lab environment with 4+ host vSAN ESA cluster

Required skills:

  • SSH access to ESXi hosts
  • Basic vSAN architecture knowledge (ESA storage pools vs OSA disk groups)
  • esxcli command familiarity

Lab Environment

VCF workload domain with 4-host vSAN ESA cluster (ESA default / auto-policy: FTT=1 RAID-5 — Table 845, VCF 9.0.1 docs p.1713); a test policy FTT=1 RAID-1 is assigned to the lab VMs

  • Hosts needed: 4
  • VMs needed: 4
  • Storage: vSAN ESA with NVMe devices

Tasks

Task 1 vSAN ESA Disk Architecture & Health Assessment

Foundation task. Panelists probe: What changed from OSA to ESA? Why does ESA eliminate the cache/capacity tier distinction? How does single-tier NVMe affect failure domains?

Understand vSAN ESA storage pool architecture vs. legacy OSA disk groups, map disk states, and perform baseline health assessment before simulating failures.

Step 1

SSH to an ESXi host in the vSAN cluster. Run 'esxcli vsan storagepool list' (ESA — applicable only to vSAN ESA clusters, p.1821) to enumerate the storage pool devices. Document: device name, device type (NVMe/SSD), capacity, state and pool membership.

vSAN ESA shows NVMe devices in a single-tier storage pool — no separate cache and capacity tiers. Each host contributes one or more NVMe devices to the pool. Device status should be 'Healthy' for all devices. Minimum configuration: 1 NVMe TLC device per host.
ESA key difference from OSA: no disk groups, no cache tier. All NVMe devices contribute to a unified storage pool. This simplifies disk management but changes failure behavior — a single disk failure reduces both capacity and performance simultaneously.
Step 2

Run 'esxcli vsan health cluster list' to check overall cluster health. Examine each health check category: Network, Physical Disk, Data, Limits, Stretch Cluster. Document any warnings or errors. Pay special attention to 'Disk balance' and 'Component metadata health'.

Healthy cluster shows green across all categories. Key health checks: vSAN object health (all objects accessible), disk balance (data distributed evenly across hosts), network health (host connectivity, MTU consistency, latency — vSAN has used unicast only since 6.6), and hardware compatibility (all devices on HCL).
Run health checks BEFORE and AFTER any disk operation. Comparing pre/post health reports is essential for validating recovery completeness.
Step 3

Check vSAN capacity status. In vSphere Client navigate to Cluster > Monitor > vSAN > Capacity. Document: total raw capacity, used capacity, free capacity, dedup/compression savings ratio, and slack space reservation. Also run: esxcli vsan debug disk list.

Capacity view shows raw vs. usable space. FTT=1 RAID-1 uses 2x raw capacity. With Reserved Capacity enabled (VCF-VSAN-RCMD-CFG-003, VCF 9.0.1 docs p.555) vSAN holds an Operations Reserve (transient operations such as policy changes and rebalancing) plus a Host Rebuild Reserve sized to absorb one host failure — about 25% of a 4-node cluster, 8% at 12 nodes — instead of a flat 25-30% slack. Usable capacity = (raw − metadata ~2% − Reserved Capacity − filesystem ~2%) / FTT multiplier.
Capacity planning is critical for disk failure resilience. If the cluster is above 70% used, a single disk failure may not have enough free capacity (Reserved Capacity) for the rebuild — creating a data-at-risk scenario.
Step 4

Document the vSAN component failure states (VCF 9.0.1 docs p.1655, 'Component State: Degraded, Absent, and Out of date States'). They are alternative states, not steps in a sequence: Absent = vSAN detects a failure that may be temporary (host reboot, unplugged or unresponsive device); vSAN waits for the Object Repair Timer (default 60 minutes) before rebuilding. Degraded = vSAN detects a permanent failure (for example a device reporting errors) and starts rebuilding immediately. Out of date = the object's components do not match its current policy. A device that is explicitly removed from the storage pool is evacuated first and never passes through these states.

Failure-state table with timers: Absent → Object Repair Timer (default 60 minutes, cluster setting) then rebuild; Degraded → immediate rebuild; Out of date → policy reapply. During any rebuild the cluster must have sufficient free capacity (ESA host rebuild reserve).
The 60-minute Object Repair Timer is a key operational parameter. It is one cluster-wide advanced option (Configure > vSAN > Services > Advanced Options), not a per-device-type setting. Change it only with a documented reason (for example, raise it for a planned maintenance window to avoid unnecessary rebuilds) and restore the default afterwards.

Validation Gate

Check: Complete vSAN ESA architecture documentation and baseline health assessment

Expected: All storage devices enumerated, health checks passed, capacity breakdown documented, disk state machine understood.

Common Errors

Confusing vSAN ESA storage pools with OSA disk groups — ESA has no cache tier
Not accounting for Reserved Capacity (Operations + Host Rebuild Reserve, ~25% on 4 nodes) when calculating usable capacity
Ignoring the 60-minute absent timer — leads to unexpected rebuild operations during maintenance

Task 2 Disk Failure Simulation & Diagnosis

Hands-on troubleshooting task. Panelists ask: How do you distinguish a transient disk error from permanent failure? What is the impact on VMs with FTT=1 when one disk fails? When does data become at risk?

Simulate disk failure scenarios in the lab, diagnose using multiple tools, and understand the cascade effects on vSAN objects and VM availability.

Step 1

Before simulating failure, document the current state of vSAN objects. Run: esxcli vsan debug object health summary get. Note the count of healthy, degraded, and absent objects. Also run: esxcli vsan debug object list | head -50 to see individual object placement.

All objects should be healthy with correct number of components per FTT policy. FTT=1 RAID-1 objects have 2 data components + 1 witness component distributed across different hosts. Document the object count as baseline.
Object health is the true measure of vSAN resilience — even if disks are healthy, misplaced or degraded objects indicate underlying issues.
Step 2

Simulate a disk failure. In the Holodeck environment, identify a vSAN data device on one host. Remove the virtual disk from the ESXi VM configuration (simulates physical disk failure). Immediately check: esxcli vsan storagepool list (ESA; the removed device is gone from the pool or shows as unhealthy), esxcli vsan debug object health summary get (objects with components on that device show reduced availability with Absent components).

The components on the removed device are marked Absent (vSAN treats an unplugged device as possibly temporary). Affected objects show reduced availability with no rebuild yet; Skyline Health warns under the Data category. The Object Repair Timer (60 minutes) starts; rebuild begins only when it expires.
In a real production scenario, disk failure is detected by SMART monitoring, ESXi kernel error messages, and vSAN health checks. The simulation skips detection but the recovery process is identical.
Step 3

Analyze the impact on VMs. Check which VMs have components on the failed device: use the vSphere Client (cluster > Monitor > vSAN > Virtual Objects, filter on reduced availability) or on the host 'esxcli vsan debug object list' and look for components in the ABSENT state. For each affected VM, document: VM name, object type (vmdk, vmx, vswp), current compliance status, and whether the VM is still accessible.

VMs with FTT=1 RAID-1 remain accessible after single disk failure — the mirror copy on another host serves reads. However, they are now unprotected (FTT=0 effective) until rebuild completes. If a second disk fails before rebuild, data loss occurs.
Critical VCDX point: FTT=1 protects against ONE failure. After first failure, data is at risk until rebuild completes. This is why vSAN monitors and alerting are essential — immediate awareness of degraded state enables proactive response.
Step 4

Monitor the repair timer countdown. Check repair progress: esxcli vsan debug resync summary get. Note: before the 60-minute timer expires, no repair activity occurs (vSAN waits for the disk to potentially return). After timer expiry or manual trigger, rebuild begins. Monitor rebuild progress: component count, data movement rate (MB/s), estimated time remaining.

Resync summary shows: objects to sync, bytes to sync, completion percentage, and estimated time. Rebuild rate depends on: cluster load, number of hosts, network bandwidth, and configured rebuild rate limit. Typical rebuild for 500 GB per disk: 2-8 hours.
Rough planning heuristic only: rebuild time ≈ data on the failed device ÷ aggregate resync throughput of the participating hosts. vSAN manages resync adaptively; there is no published per-host default rate. If resync hurts VM latency, throttle the IOPS used for resynchronisation per Broadcom KB 326830 (VCF 9.0.1 docs p.1803) and remove the throttle afterwards.
Step 5

Calculate the capacity impact of disk failure. Document: (a) raw capacity lost (failed disk size), (b) usable capacity lost (raw / FTT multiplier), (c) remaining cluster free space, (d) whether remaining capacity is sufficient for rebuild (need free space >= data on failed disk). If cluster is above 80% used, calculate whether rebuild can complete.

Single device failure in a 4-host cluster with one 4 TB device per host: raw capacity drops from 16 TB to 12 TB. Usable capacity at FTT=1 RAID-1 with ~27% reserved (Reserved Capacity on 4 nodes plus ~4% metadata/filesystem): 16 × 0.69 / 2 ≈ 5.5 TB before, 12 × 0.69 / 2 ≈ 4.1 TB after. If current usage is 3 TB, rebuild has sufficient space. If usage is 4 TB, cluster enters data-at-risk state.
This capacity analysis is exactly what VCDX panelists expect. Show the math: raw capacity - metadata - slack - FTT overhead = usable. Demonstrate you can calculate failure impact quickly.

Validation Gate

Check: Successfully simulate disk failure, diagnose impact, and calculate capacity implications

Expected: Disk failure simulated, affected objects identified, VM impact assessed, capacity calculations documented, repair timer behavior understood.

Common Errors

Not documenting pre-failure baseline — makes it impossible to compare post-recovery state
Confusing Absent (failure may be temporary — unplugged device or host reboot; rebuild waits for the 60-minute Object Repair Timer) with Degraded (permanent failure detected — rebuild starts immediately)
Forgetting that FTT=1 objects are unprotected during rebuild — a second failure causes data loss
Not calculating whether cluster has sufficient free space for rebuild before triggering it

Task 3 Disk Replacement & Rebuild Operations

Operational execution task. Panelists expect precise command sequences and understanding of maintenance mode implications: ensure data availability vs. full data migration.

Execute the complete disk replacement workflow in vSAN ESA — from removing the failed disk to adding the replacement to verifying full cluster recovery.

Step 1

Remove the failed disk from the vSAN storage pool. Use esxcli (ESA): 'esxcli vsan storagepool remove --disk <device_id>' (Table 856, p.1839). Verify removal: 'esxcli vsan storagepool list' should no longer show the device. Check that objects previously on this disk are being rebuilt to remaining disks.

Disk removal is immediate. vSAN begins rebuilding components that were on the removed disk to other healthy disks in the cluster. Rebuild progress visible in esxcli vsan debug resync summary get.
In ESA, disk removal from the storage pool is simpler than OSA disk group removal — no cache tier concerns. But verify that the cluster has sufficient capacity before removing.
Step 2

While waiting for rebuild, understand vSAN maintenance mode options. Three modes: (1) Ensure accessibility — fastest, moves minimum data to maintain FTT=0 access; (2) Full data migration — slowest, moves ALL data off the host (required for host decommission); (3) No data migration — fastest, used for quick reboots when host returns quickly. Document when each mode is appropriate.

Ensure accessibility: use for disk replacement (host stays, just one disk gone). Full data migration: use for host removal from cluster (permanent). No data migration: use for firmware update reboot (<60 minutes). Pre-check available: cluster validates whether the requested mode can complete.
VCDX design point: maintenance mode choice directly impacts maintenance window duration. Full data migration for a 4 TB host takes 4-8 hours. Ensure accessibility takes minutes. Choose wisely based on the operation.
Step 3

Simulate adding a replacement disk. In Holodeck, add a new virtual disk to the ESXi VM. Claim it for the ESA storage pool: 'esxcli vsan storagepool add --disk <new_device_id>' (or use the vSphere Client: cluster > Configure > vSAN > Disk Management > Claim Unused Disks). Verify: 'esxcli vsan storagepool list' shows the new device in Healthy state.

New disk joins the storage pool immediately. vSAN begins proactive rebalancing to distribute data across all disks including the new one. Rebalancing runs at lower priority than rebuild to minimize VM impact.
After adding a new disk, rebalancing may take hours depending on data volume. Monitor with: esxcli vsan debug resync summary get. Do not add another disk until rebalancing completes.
Step 4

Monitor the complete recovery. Track: (a) resync progress (esxcli vsan debug resync summary get), (b) object health returning to fully compliant (esxcli vsan debug object health summary get), (c) cluster capacity returning to expected level (vSphere Client capacity view), (d) VM I/O performance during rebuild (esxtop disk view). Document the total recovery timeline.

Full recovery sequence: disk removal (immediate) -> rebuild to remaining disks (2-8 hours) -> replacement disk added (immediate) -> rebalancing to new disk (2-8 hours) -> all objects compliant (final state). Total recovery window: 4-16 hours depending on data volume.
Resync tuning: vSAN manages resynchronisation adaptively by default. If VMs suffer latency from a resync, throttle the IOPS used for resynchronisation following Broadcom KB 326830 (VCF 9.0.1 docs p.1803), and remove the throttle once the resync completes. 'esxcli vsan policy setdefault' sets default storage-policy values and does not control resync.
Step 5

Perform post-recovery validation. Run complete health check: esxcli vsan health cluster list. Verify: all objects healthy, disk balance within 5% variance, no stale components, capacity matches expected values. Compare with pre-failure baseline documented in Task 1.

Post-recovery cluster should match pre-failure baseline: same object count (healthy), same capacity (with new disk), all health checks green. Disk balance may take additional time to equalize across all hosts.
Document the complete recovery timeline as a reference for future incidents and capacity planning. This data is invaluable for SLA calculations and VCDX design justification.

Validation Gate

Check: Complete disk replacement workflow with verified cluster recovery

Expected: Failed disk removed, replacement added, all objects rebuilt to compliance, cluster health restored to baseline.

Common Errors

Removing disk without verifying cluster has capacity for rebuild — triggers data-at-risk state
Using 'Full data migration' maintenance mode for disk replacement — unnecessary, wastes hours
Adding replacement disk before rebuild completes — causes competing I/O and extends total recovery time
Not running post-recovery health check — may miss partially rebuilt objects or balance issues

Task 4 Proactive Monitoring Design & VCDX Defense

Synthesis task connecting operational procedures to architectural design. Panelists expect monitoring strategy, SLA impact analysis, and justification for FTT policy choices.

Design a comprehensive vSAN disk health monitoring strategy and prepare to defend disk failure recovery design decisions in VCDX context.

Step 1

Design a vSAN disk health monitoring framework. Define alerts for: (a) SMART pre-failure warnings (disk showing errors before failure), (b) disk state transition alerts (Healthy to Degraded/Absent), (c) capacity threshold alerts (70% warning, 80% critical, 85% emergency), (d) rebuild progress alerts (rebuild started, 50% complete, completed), (e) performance impact alerts (VM latency exceeding baseline during rebuild). Map each alert to VCF Operations or vCenter alarm configuration.

Monitoring framework with 5 alert categories, each mapped to specific metrics, thresholds, and notification targets. SMART monitoring catches 60-80% of disk failures before they occur. Capacity alerts prevent rebuild failures due to insufficient space.
Proactive monitoring is the key differentiator between reactive and mature operations. VCDX panelists specifically ask: How do you prevent disk failure incidents? SMART monitoring and capacity alerts are the answers.
Step 2

Calculate SLA impact for different failure scenarios. Build a table: Scenario | Detection Time | Repair Time | Total Impact | SLA Effect. Scenarios: (a) single disk failure with FTT=1 RAID-1 (data accessible, rebuild 4h), (b) single disk failure with FTT=1 RAID-5 (data accessible, rebuild 6h due to parity calculation), (c) dual disk failure with FTT=1 (data loss if on same object), (d) host failure (all disks on host lost, rebuild from remaining 3 hosts). Correlate with SLA targets: 99.9% = 8.76h/year, 99.99% = 52.6 min/year.

SLA impact table showing that FTT=1 handles single-failure scenarios within SLA. Dual failure before rebuild completes is the risk scenario. Host failure with FTT=1 is equivalent to disk failure (1 of 2 copies lost). FTT=2 provides dual-failure protection but costs 3x capacity (RAID-1) or 1.5x (RAID-6).
VCDX defense: always connect storage policy (FTT level) to SLA requirement. FTT=1 for 99.9%, FTT=2 for 99.99%+ with financial workloads. Show the cost-capacity tradeoff.
Step 3

Document the complete disk failure recovery runbook for VCF operations team. Include: (a) incident detection and classification, (b) impact assessment checklist, (c) step-by-step recovery procedure with esxcli commands, (d) communication template for stakeholders, (e) post-incident review template. This runbook should be usable by L2 support without escalation.

Runbook covers: detection (alert fires) to resolution (all objects compliant) with specific commands, expected outputs, and decision points. Communication template includes: what happened, what is the impact, what is being done, when will it be resolved.
A well-documented runbook reduces MTTR from hours to minutes. VCDX panelists value operational maturity — show that your design includes operational procedures, not just architecture diagrams.
Step 4

Prepare four VCDX defense responses: (1) 'Why FTT=1 instead of FTT=2 for this workload domain?' — cost analysis shows FTT=2 RAID-1 requires 3x capacity (vs 2x for FTT=1), adding $X per TB. Risk analysis shows single-failure probability is 10x higher than dual-failure, making FTT=1 appropriate for non-critical workloads. (2) 'What happens if a disk fails during a VCF upgrade?' — disk failure during ESXi upgrade triggers vSAN rebuild on remaining hosts; upgrade pauses if cluster health drops below threshold; resume after rebuild stabilizes. (3) 'How do you handle disk failure in a 4-host minimum cluster?' — 4-host cluster can tolerate 1 host/disk failure with FTT=1; during rebuild, cluster operates at reduced capacity; if the cluster drops below 3 healthy hosts (fault domains), policy-compliant objects at FTT=1 cannot be created — provisioning fails unless force provisioning is set, which a design should not rely on. (4) 'Why ESA over OSA for this design?' — ESA eliminates cache/capacity tier complexity, supports larger NVMe devices (up to 32 TB), delivers higher IOPS per device, and simplifies disk replacement procedures.

Four defense responses with quantitative backing. Each demonstrates storage design understanding connected to business requirements.
Always lead with the business requirement, then the technical justification, then the risk mitigation. This RCAR flow is natural for VCDX defense.

Validation Gate

Check: Complete monitoring framework, SLA analysis, operational runbook, and VCDX defense preparation

Expected: Monitoring framework with 5 alert categories documented. SLA impact table created. Recovery runbook ready for L2 operations. Four VCDX defense responses prepared.

Common Errors

Not connecting FTT policy to SLA requirements — storage policy must trace to business availability targets
Overlooking SMART monitoring — proactive detection prevents 60-80% of disk failure incidents
Writing runbooks that require L3/architect escalation for routine operations — defeats the purpose of runbook automation
Ignoring rebuild performance impact on VMs — rebuild generates significant I/O that competes with production workloads

Final Validation

Complete vSAN disk failure recovery with proactive monitoring and VCDX defense readiness

✓ vSAN ESA architecture understood → Storage pool architecture, disk states, and ESA vs OSA differences documented

✓ Disk failure diagnosed → Failure simulated, objects identified, capacity impact calculated

✓ Recovery completed → Disk replaced, objects rebuilt, cluster restored to baseline health

✓ Monitoring framework designed → SMART alerts, capacity thresholds, and rebuild monitoring configured

✓ VCDX defense prepared → Four defense responses with quantitative analysis documented

Cleanup / Restore

• Verify all vSAN objects are healthy: esxcli vsan debug object health summary get

• Confirm cluster capacity matches expected values

• Check disk balance across all hosts

• Revert to snapshot if needed for next lab

Design Reflection (VCDX)

vSAN disk failure recovery demonstrates operational maturity in VCF storage management. The design connects storage policy (FTT level) to SLA requirements, implements proactive SMART monitoring, and provides graduated recovery procedures. ESA storage pool architecture simplifies disk management compared to OSA disk groups.

Requirements

  • R-001: vSAN cluster must tolerate single disk failure without VM downtime
  • R-002: Disk failure recovery must complete within 8 hours (SLA target)
  • R-003: Proactive monitoring must detect 80%+ of disk failures before data loss

Constraints

  • Minimum 3 hosts per vSAN HCI cluster (VCF 9.0 docs, vSAN Deployment Options: "vSAN HCI cluster or a single site vSAN cluster consists of a minimum of three ESX hosts"); 4 used here so a host can be in maintenance and the ESA RAID-5 auto-policy still applies
  • vSAN ESA requires NVMe devices (no SAS/SATA support)
  • Rebuild rate must be balanced against production VM performance

Assumptions

  • Spare NVMe disks available within 4-hour delivery window
  • Operations team trained on vSAN disk replacement procedures
  • VCF Operations or equivalent monitoring deployed and configured

Risks

  • Dual disk failure before rebuild completes causes data loss with FTT=1 — mitigate with FTT=2 for critical workloads
  • Cluster at >80% capacity may not have sufficient space for rebuild — proactive capacity monitoring required
  • Firmware defects on NVMe devices can cause correlated multi-device failure — keep devices on vSAN-certified (Broadcom Compatibility Guide) firmware managed by vLCM, stage firmware updates host by host, and use FTT=2 for critical workloads (deliberately mixed firmware breaks the certified configuration)

Self-Assessment Discussion Prompts

  1. At what cluster utilization percentage would you block new VM provisioning to protect rebuild capacity?
  2. How does vSAN ESA change your disk failure response compared to OSA?
  3. What is the cost-benefit breakpoint for FTT=1 RAID-5 vs FTT=1 RAID-1 in a capacity-sensitive environment?
  4. How would you design disk failure resilience for a stretched vSAN cluster?

Extensions

vSAN RAID-5/6 Erasure Coding Recovery

Configure a storage policy with FTT=1 RAID-5 (erasure coding). Simulate disk failure and compare rebuild time and I/O impact with RAID-1 mirroring. Document the rebuild overhead difference: RAID-5 requires parity recalculation, making rebuild ~50% slower than RAID-1.

Multi-Disk Failure Scenario

Prerequisite: FTT=2 needs 2n+1 = 5 hosts (fault domains) contributing capacity for RAID-1, or at least 6 for RAID-6, so add hosts to the 4-host lab cluster before running this extension. Simulate simultaneous failure of 2 disks on different hosts in that FTT=2 cluster. Verify data remains accessible. Document the degraded capacity and rebuild timeline for dual-failure recovery.

Automated Disk Replacement with vSAN Proactive Operations

Configure VCF Operations to detect SMART warnings, automatically create a change ticket, and pre-stage the disk replacement workflow. Design the end-to-end automation from detection to replacement verification.

⚠ Known Pitfalls

Removing a failed device without verifying the cluster has enough free capacity (Host Rebuild Reserve) for the rebuild — if cluster is above 80% used, rebuild may fail and trigger data-at-risk state
Using 'No data migration' host maintenance mode before a permanent host or device removal — with FTT≥1 the objects become non-compliant (one copy fewer), not lost, but a second failure during the window would be; use 'No data migration' only for short reboots and 'Full data migration' before permanent removal
Not monitoring rebuild progress — rebuilds can stall due to network congestion or capacity constraints, requiring manual intervention
Ignoring SMART pre-failure warnings — proactive replacement during maintenance window is 10x less risky than reactive replacement after failure

References

  • vSAN 9.0 Administration Guide — Disk Management and Maintenance chapter
  • vSAN 9.0 Troubleshooting Guide — Object Health and Repair
  • VMware KB 2107713 — vSAN disk failure troubleshooting and replacement procedures
  • vSAN ESA Architecture and Operations Guide
  • VMware VCF 9.0 Storage Management — vSAN Lifecycle Operations
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.