Multi-Site Design Decision Framework

Design Tier-1 failover workflow (Stretched Cluster — automatic):

1. Trigger: Site A failure detected by vSAN and HA

• vSAN detects host unreachability (5 heartbeat misses × 1s = 5s)
• HA declares hosts as failed (30s default isolation timeout)
• Total detection time: ~35 seconds

2. Automatic actions (no human intervention):

• vSAN: Witness votes for Site B (preferred site was A, but A is down)
• HA: Restarts Tier-1 VMs on Site B hosts (priority order)
• DRS: Distributes restarted VMs across available Site B hosts
• DFW: Rules follow VMs (pushed to new host kernel modules)

3. Restart priority order:

a. Database VMs (highest) — data consistency depends on clean startup
b. Application VMs (high) — reconnect to databases after DB is up
c. Web VMs (medium) — front-end reconnects to app tier

4. Post-failover validation (manual, by on-call team):

• [ ] All 80 Tier-1 VMs running on Site B

   - [ ] Database consistency check (application-specific: MySQL → check InnoDB recovery log)

• [ ] Application health checks passing (HTTP 200 from health endpoints)
• [ ] DNS: If using GSLB, verify Site B endpoints are active
• [ ] Monitoring: Aria Operations shows all VMs on Site B, no lingering alerts
• [ ] Communication: Notify stakeholders of site failure and successful failover

Design Tier-2 failover workflow (SRM — semi-automated):

1. Trigger: Site A declared unavailable by operations team

• NOT automatic — requires human decision to invoke SRM recovery plan
• Reason: Avoiding false failover due to transient network issue

2. Pre-failover checklist:

• [ ] Verify Site A is truly unavailable (out-of-band verification via IPMI/iLO)
• [ ] Verify last successful replication timestamp (RPO check)
• [ ] Notify stakeholders: planned failover in progress
• [ ] Confirm DR site capacity can handle Tier-2 workload

3. SRM Recovery Plan execution:

a. SRM powers off placeholder VMs at DR site (if any)
b. SRM reconfigures replicated VMs:

• Update IP addresses (if DR uses different subnet) or rely on NSX L2 extension
• Mount replicated VMDKs
• Adjust VM settings (if different hardware profile at DR)

c. Power on VMs in priority groups:

• Group 1: Infrastructure services (AD, DNS, monitoring)
• Group 2: Database VMs (delay: 120s for service stabilization)
• Group 3: Application VMs (delay: 60s)
• Group 4: Web/front-end VMs (delay: 30s)

d. Run recovery plan custom scripts:

• Post-power-on: Verify database connectivity
• Post-recovery: Update load balancer VIP to DR site

4. Expected timeline:

• Decision to invoke: 5-15 minutes (human assessment)
• SRM plan execution: 15-30 minutes (depends on VM count and startup delays)
• Total RTO: 20-45 minutes for 300 Tier-2 VMs

Design failback workflow (return to primary site):

Failback is more complex than failover because data has been modified at the DR site.

Tier-1 (Stretched Cluster) failback:

1. Repair/restore Site A hosts

2. Add hosts back to cluster (SDDC Manager → Commission Host)
3. vSAN detects stale components on Site A
4. vSAN resync: Copies changed data from Site B → Site A

• Duration: Proportional to data change rate × time since failure
• Example: 10 TB changed over 24 hours, 10 Gbps link = ~2.5 hours

5. After resync complete: DRS migrates VMs back to Site A (per site affinity rules)
6. Verification: All components in compliance, site affinity satisfied

Tier-2 (SRM) failback:

1. Ensure primary site restored and healthy

2. SRM reprotect: Reverse replication direction (DR → Primary)
   - SRM creates placeholder VMs at primary site
   - Replication begins DR → Primary (initial sync may take hours)
3. Wait for replication to be in sync
4. Execute SRM planned failback (not a forced failover — clean cutover)
   a. Shut down VMs at DR site
   b. Final replication sync (delta only)
   c. Power on VMs at primary site with original network config
   d. Verify application health
5. Re-protect: Restore replication Primary → DR

6. Test: Run SRM test recovery to verify DR readiness restored

Tier-3 (Pilot Light) failback:

1. Restore VMs from backup at primary site
2. Verify data from last backup is acceptable
3. Power on VMs at primary site
4. Update monitoring to track primary site VMs

Design DR testing strategy:

1. SRM recovery plan testing:

• Frequency: Quarterly
• Method: SRM test recovery (non-disruptive — creates test VMs at DR site in isolated network)
• Duration: 2-4 hours
• Validation: Application health checks, database connectivity, user acceptance
• Output: Test report documenting RTO achieved, failures, and remediation

2. Stretched cluster failover testing:

• Frequency: Semi-annual
• Method: Planned maintenance failover (move all VMs to Site B via DRS)
• Duration: 4-6 hours (including failback)
• Validation: Zero data loss, VM accessibility throughout, performance baseline comparison
• Cannot do: Simulate actual site failure without planned outage (use Holodeck instead)

3. Backup restore testing:

• Frequency: Monthly
• Method: Restore random sample of VMs from backup to isolated network
• Validation: VM boots, application starts, data is recent and consistent
• Output: Restore success/failure report, backup tool health

4. DR test calendar:

Configure vSphere Replication for Tier-2 workloads:

vSphere Replication architecture:

• vSphere Replication Appliance (VRA): Deployed at both sites
• Replication: VM-level, asynchronous, host-based (no array dependency)
• RPO: 5 minutes minimum, configurable per VM
• Protocol: TCP/31031 (encrypted by default in 9.0)

- Replication data flow: ESXi host → VRA (source) → VRA (target) → target datastore

Configuration for Tier-2 (300 VMs):

• RPO: 15 minutes (balance between data protection and bandwidth)
• Replication schedule: Continuous (not scheduled — delta-based)
• Bandwidth calculation:
  • Average daily change rate: 5% of VM storage per day
  • Tier-2 storage: 300 VMs × 150 GB = 45 TB
  • Daily change: 45 TB × 5% = 2.25 TB/day
  • Hourly: 2.25 TB / 24 = 93.75 GB/hour
  • Bandwidth needed: 93.75 GB × 8 / 3600 = ~208 Mbps sustained
  • With overhead (protocol + encryption): ~250 Mbps

  - Available WAN: 1 Gbps → sufficient with headroom

• Seed copy: For initial replication of 45 TB
  • Over 1 Gbps WAN: 45 TB × 8 / 1000 Mbps = ~100 hours (4+ days)
  • Alternative: Ship physical disk with initial copy, then switch to network replication

Design SRM recovery plan configuration:

SRM recovery plan structure:

1. Protection groups:

• PG-Tier2-Databases: All Tier-2 database VMs (30 VMs)
• PG-Tier2-AppServers: All Tier-2 application VMs (180 VMs)
• PG-Tier2-WebServers: All Tier-2 web servers (90 VMs)

2. Recovery plan: RP-Tier2-Full-Recovery

• Priority Group 1: PG-Tier2-Databases
  • Pre-power-on steps: Check target datastore capacity
  • Power-on delay: 0 seconds
  • Post-power-on steps: Wait for SQL/MySQL service to respond (timeout: 300s)
• Priority Group 2: PG-Tier2-AppServers
  • Pre-power-on steps: None
  • Power-on delay: 120 seconds (wait for databases)
  • Post-power-on steps: HTTP health check on port 8080 (timeout: 120s)
• Priority Group 3: PG-Tier2-WebServers
  • Pre-power-on steps: None
  • Power-on delay: 60 seconds (wait for app servers)
  • Post-power-on steps: HTTP health check on port 443 (timeout: 60s)

3. Network mapping:

• Option A: Same IP addresses at DR (if using NSX L2 extension or stretched VLANs)
• Option B: IP customization (if DR uses different subnets)
  • SRM handles: IP re-addressing, default gateway, DNS server update
  • Not handled: Application-level connection strings (requires DNS update or config management)
• Design decision: Use NSX L2 extension for critical VMs (no IP change needed)

Compare replication technologies for edge cases:

Decision rule:

• If vSAN and RPO=0: Stretched cluster
• If vSAN and RPO>0: vSphere Replication (native, no array dependency)
• If external storage: Array replication OR vSphere Replication
• If cloud DR target: HCX DR (best cloud mobility)

- If cross-platform (VMware → non-VMware): HCX (heterogeneous support)

Design RPO verification and monitoring:

1. vSphere Replication monitoring:

   - Dashboard: Aria Operations → vSphere Replication management pack
   - Key metrics:
     - RPO violation count (VMs exceeding configured RPO)
     - Replication lag (time since last successful sync)
     - Transfer rate (MB/s per VM)
   - Alerts:
     - RPO breach >2× configured value → Warning
     - RPO breach >4× configured value → Critical
     - Replication error (connection lost) → Critical

2. Stretched cluster monitoring:

   - vSAN Health → Stretched Cluster section

• Inter-site latency trending
• Resync progress (during recovery)
• Component health across sites

3. Compliance reporting:

• Monthly RPO compliance report:
  • % of time each VM met its RPO target
  • Number of RPO violations per VM
  • Root cause of violations (bandwidth saturation, host issues)
• Target: 99.9% RPO compliance for Tier-2 VMs

4. Bandwidth monitoring:

• Track actual replication bandwidth vs provisioned
• If actual approaches 80% of available: trigger capacity alert
• Trending: Project when bandwidth will be insufficient (growth-adjusted)

Create design decision D-011 — Multi-Site DR Strategy:

Decision: Tiered DR with stretched cluster (Tier-1), SRM (Tier-2), backup (Tier-3)

Alternatives:
A) Stretched cluster for ALL VMs:

• RPO=0 for everything
• Cost: Extreme (600+ VMs × 2 sites, 30+ hosts per site)
• Rejected: Cost prohibitive; Tier-2/3 do not require RPO=0

B) SRM for ALL VMs:

• RPO=5-15 min for everything
• Cost: Moderate (DR hosts need not be 1:1)
• Rejected: Tier-1 requires RPO=0; SRM cannot provide this

C) HCX DR for ALL VMs:

• Similar to SRM capability set
• Pro: Simplified management, cloud DR option
• Con: HCX DR is newer, less mature DR testing framework than SRM
• Rejected: SRM has more robust recovery plan testing and compliance reporting

D) Backup-only for ALL VMs:

• RPO=24 hours, RTO=8-24 hours
• Cost: Minimal
• Rejected: Unacceptable for Tier-1 and Tier-2 SLAs

Justification: Tiered approach optimizes cost by matching DR investment to business criticality.
Cost savings: ~45% reduction vs all-stretched, with Tier-1 still at RPO=0.

Prepare VCDX defense responses:

Challenge 1: 'Your Tier-2 has RPO=15 min. What if the customer says they need RPO=5 min?'

Response: vSphere Replication supports RPO=5 min. Change is configuration-only. However, bandwidth impact increases 3×: 250 Mbps → ~750 Mbps. Verify WAN capacity before committing. If WAN insufficient, either upgrade link or negotiate RPO=10 min as compromise.

Challenge 2: 'What is the total cost of the multi-site design?'
Response: Primary: 21 hosts + DR: 12 hosts + Witness: 1 appliance = 33 hosts total + 1 Gbps WAN + 100 Gbps inter-site dark fiber. Licensing: 33 × 64 cores × per-core cost. Compare to single-site: 21 hosts — DR adds ~57% to host count but protects $X million in application availability. Business case: If Tier-1 outage costs $Y/hour, stretched cluster pays for itself after Z hours of prevented downtime.

Challenge 3: 'How do you handle a DR test for Tier-1 stretched cluster?'
Response: Cannot simulate site failure without actual outage. Alternative: Planned failover — DRS evacuates all Tier-1 VMs to Site B, verify application health, then fail back. Additionally, test partition scenarios on Holodeck to validate failure matrix without production risk.

Challenge 4: 'What if the customer adds a regulatory requirement for a third geographic region?'
Response: vSAN stretched cluster supports 2 data sites only. For 3-region DR: Deploy primary at Region A, stretched cluster to Region B (same country/jurisdiction), asynchronous SRM replication to Region C (different jurisdiction). This gives RPO=0 within jurisdiction and RPO=15min to remote region.

Create the multi-site architecture summary:

1. Sites:

• Primary (Site A): 21 hosts — Management (4) + Production (13) + Dev/Test (4)
• DR (Site B): 12 hosts — Stretched mirror for Tier-1 (8, shared with Site A count) + DR for Tier-2 (4)
• Witness (Site C): 1 Medium OVA appliance

2. DR pattern mapping:

3. Network:

   - Inter-site (A↔B): 100 Gbps dark fiber (stretched cluster + replication)
   - WAN (A→B for SRM): 1 Gbps MPLS (replication traffic)
   - Witness (C↔A/B): 1 Gbps MPLS

4. Testing schedule:

• Monthly: Backup restore spot check (10 VMs)
• Quarterly: SRM test recovery (all Tier-2)
• Semi-annual: Stretched cluster planned failover

5. Monitoring:

• Aria Operations dashboards for replication health, RPO compliance, inter-site latency
• Alert thresholds documented and configured

Final design validation checklist:

• [ ] Every VM has an assigned DR tier with defined RPO/RTO
• [ ] DR infrastructure sized for the workload it must protect
• [ ] Bandwidth calculated and sufficient for replication load
• [ ] Failover procedure documented for each tier with startup order
• [ ] Failback procedure documented (including SRM reprotect)
• [ ] DR testing calendar established with test types and frequency
• [ ] RPO monitoring configured with alert thresholds
• [ ] Cost comparison documented (tiered vs uniform DR)
• [ ] Design decision register entry with alternatives and justification
• [ ] VCDX defense responses prepared for 4+ challenge questions
• [ ] Architecture diagram shows all sites, connectivity, and data flow

- [ ] Runbook: On-call procedure for site failure detection → failover invocation → verification