Multi-Site vSAN Stretched Cluster Design

Document witness appliance sizing tiers (VCF 9.0 / vSAN 9.0):

The witness is deployed as an OVA appliance with predefined tiers:

Component count estimation:

• Each VM object creates 2-4 components (depending on FTT)
• Formula: VMs × average_VMDKs × (FTT+1) × 2 (for witness metadata + data components)

- Example: 200 VMs × 2 VMDKs × 2 (FTT=1) × 2 = 1,600 components → Tiny sufficient
- With growth: 200 × 1.2³ × 2 × 2 × 2 = 2,765 → Medium recommended

Design witness placement:

Placement requirements:

1. Must be in a THIRD fault domain (not Site A or Site B)

• Purpose: In a site partition, witness casts the deciding vote for data availability
• If witness is co-located with Site A: Site A always wins the vote, Site B VMs become unavailable in any partition

2. Placement options:

a) Corporate office (Site C) — Recommended if ≤200ms RTT to both data sites
b) Cloud (AWS/Azure/GCP) — Supported via Witness OVA on cloud VM; adds cloud cost
c) Colocation facility — Best for independence; higher cost
d) Third data center — Ideal but requires existing infrastructure

3. Witness does NOT store VM data — only metadata (component placement, configuration)

• Bandwidth requirement: <5 Mbps typically (metadata only)
• Latency: ≤200ms RTT to each data site

4. Witness HA:

• Deploy witness on infrastructure NOT managed by the stretched cluster it protects
• Run on separate vCenter or standalone ESXi at witness site
• If witness fails: Cluster continues operating normally until a site fails + witness is down simultaneously

Configure site affinity policies:

vSAN stretched cluster site affinity:

1. Preferred site: Site A (primary)

• In a partition without witness, VMs on the preferred site continue running
• VMs on the non-preferred site are powered off to prevent split-brain

2. VM-site affinity rules:

• Site A affinity group: VMs that should primarily run on Site A hosts
• Site B affinity group: VMs that should primarily run on Site B hosts
• No affinity: VMs that can run on either site (DRS decides)

3. Data locality:

• vSAN places replica components on the same site as the VM (site-aware)
• Read-local policy: VMs read from local replica (no cross-site reads for normal I/O)
• Writes: Synchronous to both sites (this is the RTT-sensitive path)

4. Design for 600-VM environment:

• Site A: 350 VMs (Tier-1 primary + Tier-2 subset)
• Site B: 250 VMs (Tier-2 remainder + Tier-3)
• Cross-site vMotion: Used only for planned maintenance, not routine DRS

Design stretched cluster capacity for single-site failure:

Capacity rule: Each site must have enough capacity to run ALL VMs when the other site fails.

Site A capacity:

• 8 hosts × 64 cores × 512 GB RAM = 512 cores, 4 TB RAM
• Under normal operation: 350 VMs (Tier-1 + Tier-2 subset)
• During Site B failure: Must accommodate 250 additional VMs from Site B
• Total: 600 VMs requiring ~640 vCPU, ~4.5 TB RAM (with Tier-2/3 overcommit)

- Fit check: 512 cores with applicable overcommit → sufficient

Site B capacity:

• 8 hosts × 64 cores × 512 GB RAM = 512 cores, 4 TB RAM
• Must accommodate 350 additional VMs from Site A during failure

Storage:

• FTT=1 site mirroring: Each site stores a full copy of all data
• Usable capacity per site must fit total dataset
• With failure: No protection degradation (data still mirrored, just missing one site copy)
• Rebuild: When failed site recovers, full resync occurs (plan for network impact)

Total host count: 16 data hosts + 1 witness = 17 hosts for stretched cluster

Build a failure scenario matrix:

Deep-dive Scenario 4 (Worst case: Site A + Witness fail simultaneously):

This is the hardest VCDX defense question. Walk through step by step:

1. Site A fails: Hosts power off, local data copies unavailable
2. Witness fails simultaneously: No quorum vote possible
3. Site B hosts detect:

• Lost connectivity to Site A hosts (HA isolation detection)
• Lost connectivity to witness (cannot determine partition vs failure)
• vSAN cannot confirm whether Site A is truly down or just partitioned

4. Result:

• VMs that were running on Site B hosts: Continue running (local compute still works)
• Data components: Only Site B copies available; vSAN marks Site A components as absent

   - WITHOUT quorum: vSAN cannot promote Site B components to primary → data READ-ONLY or INACCESSIBLE for objects where Site B doesn't hold the primary copy
5. Recovery:
   - Option A: Restore witness first → quorum restored → vSAN promotes Site B components → full read/write

• Option B: If witness cannot be restored, use vSAN force recovery (CMMDS partition repair) — RISK: potential data inconsistency

Document: This scenario requires witness HA at the witness site (redundant power, network). It is the architectural justification for witness placement in a highly-available facility.

Design recovery procedures for each scenario:

Recovery from Scenario 1 (Site A failure):

1. Immediate: HA restarts Site A VMs on Site B (automatic)
2. Capacity check: Verify Site B can handle combined workload (may need to power off non-critical Tier-3 VMs)
3. Site A recovery: When Site A comes back online, vSAN detects stale components

4. Resync: vSAN resyncs all stale components from Site B → Site A

• Duration: Depends on data volume and inter-site bandwidth
• Example: 50 TB of data, 80% changed, 10 Gbps link = ~9 hours

5. Post-resync: DRS rebalances VMs back to original site affinity
6. Verify: vSAN health shows all components in compliance

Recovery from Scenario 6 (Network partition, witness on A side):

1. Site B VMs powered off by vSAN (no quorum)
2. Do NOT manually restart Site B VMs — this risks split-brain with conflicting writes
3. Fix network partition
4. vSAN automatically reconciles and restarts Site B VMs
5. Verify: No object version conflicts in vSAN health

Create a stretched cluster failure runbook:

1. Detection:

   - Monitor: vSAN Health → Stretched Cluster → Site connectivity
   - Alert: Any site connectivity loss → Critical alert → page on-call

• Dashboard: Aria Operations stretched cluster widget showing site status

2. Decision tree:

• Is it a site failure or network partition?

     → Check witness connectivity to both sites
     → Check out-of-band connectivity (IPMI/iLO) to failed site hosts
   - Is data accessible?
     → Check vSAN health → Object Health
     → Count accessible vs inaccessible objects
   - Can we fail back safely?
     → Resync complete? (vSAN health → Resync dashboard)
     → All components healthy?

3. Escalation:

• Level 1: On-call verifies scenario, checks vSAN health
• Level 2: VMware Support engagement (GSS SR, Severity 1 for data loss risk)
• Level 3: vSAN force recovery (only with VMware Support guidance)

4. Post-incident:

• Root cause analysis within 48 hours
• Update risk register if scenario was not previously anticipated
• Test witness failover to verify future resilience

Create design decision D-008 — Multi-Site Architecture Pattern:

Decision: vSAN Stretched Cluster (synchronous active-active)
Alternatives considered:
A) Active-Passive with SRM:

• RPO: Minutes to hours (asynchronous replication)
• RTO: 30-60 minutes (SRM recovery plan execution)
• Pro: Simpler, no inter-site latency sensitivity
• Con: Data loss up to RPO; manual/semi-automatic failover
• Rejected: Customer requires RPO=0 for Tier-1 applications

B) Active-Active with HCX:

• HCX provides VM mobility but not storage replication
• Would need separate storage replication (e.g., array-based)
• Rejected: Adds complexity, violates single-vendor constraint

C) Pilot Light DR:

• Minimal infrastructure at DR site, powered up on demand
• RTO: 2-4 hours (power up infrastructure + restore from backup)
• Pro: Lowest cost
• Con: High RTO, manual process
• Rejected: Does not meet 4-hour RTO for 600 VMs

D) VMware Site Recovery (vSphere Replication):

• RPO: 5 minutes to 24 hours (configurable)
• RTO: 30-60 minutes
• Pro: Simpler than stretched cluster, no latency requirements
• Acceptable alternative if RPO=0 is relaxed to RPO=5min

Justification: Stretched cluster is the only VCF-native option providing RPO=0 with automatic failover.

Document total cost comparison:

Cost premium for RPO=0:

• Additional hosts: 4 × host cost
• Additional licensing: 256 × per-core cost
• Network: Dark fiber lease vs MPLS
• Total estimated premium: 30-40% over active-passive

Document: Is RPO=0 worth the cost premium? Map back to business requirement.

Prepare VCDX defense for stretched cluster challenges:

Challenge 1: 'Stretched clusters add complexity. Why not just use SRM?'
Response: Customer requirement R-001 specifies RPO=0 for Tier-1 applications. SRM minimum RPO is 5 minutes. The business case (financial trading / healthcare) cannot tolerate any data loss. However, for Tier-2/3, I would recommend SRM or vSphere Replication as a cost-effective alternative.

Challenge 2: 'What if the inter-site link latency degrades beyond 5ms?'
Response: vSAN will continue operating but with degraded write performance (every write waits for cross-site acknowledgment). If sustained >10ms, I would recommend converting to asynchronous replication (SRM) and accepting RPO>0. The design includes monitoring alerts at 3ms (warning) and 5ms (critical) thresholds.

Challenge 3: 'Your witness site is a corporate office — what if it loses power?'
Response: The witness only provides quorum votes. If the witness fails while both data sites are healthy, the cluster continues normally (Scenario 3 in the failure matrix). The risk window is witness failure + simultaneous site failure. Mitigation: UPS at witness site with 4-hour runtime, plus documented procedure for deploying replacement witness from OVA.

Challenge 4: 'How do you handle a planned site-wide maintenance (e.g., power shutdown at Site A)?'
Response: Planned failover procedure: (1) DRS evacuates all VMs to Site B, (2) Enter maintenance mode on all Site A hosts with 'Ensure data accessibility', (3) Power down Site A, (4) During maintenance: Site B runs at full capacity with witness providing quorum, (5) Reverse process to restore. Total planned failover time: ~2 hours for 350 VMs.

Create the final stretched cluster design summary:

1. Architecture:

• 2 data sites (8+8 hosts) + 1 witness site (Medium OVA)
• vSAN ESA with FTT=1 site mirroring (data copy at each site)
• L3 routed inter-site connectivity (no L2 stretch)
• NSX overlay for workload networking

2. Network:

• Inter-site: 2×100 GbE dark fiber (primary) + 1×10 GbE MPLS (backup)
• Measured RTT: 0.8ms (well within 5ms requirement)
• MTU: 9000 end-to-end, 1700 for Geneve overlay

3. Witness:

• Tier: Medium (supports up to 21,833 components)
• Placement: Corporate office (Site C), 50km from both sites
• Connectivity: 1 Gbps MPLS, 2.5ms RTT to Site A, 3.0ms to Site B

4. Failure handling:

• 7 scenarios documented with expected behavior
• Preferred site: Site A
• Recovery runbook with decision tree and escalation path

5. Monitoring:

• Aria Operations dashboards for inter-site latency, resync progress
• Alerts: RTT >3ms warning, >5ms critical
• Weekly witness health verification