Disaster Recovery Architecture Design

Design the Tier-1 stretched cluster topology. Define the 4-node vSAN cluster split across Site-A (2 nodes) and Site-B (2 nodes) with a 1-node witness host at Site-C (or logically isolated on-prem). Document the placement: hosts A1, A2 at Site-A datastore 1; hosts B1, B2 at Site-B datastore 2; witness at Site-C (witness appliance ~40 GB footprint; stores metadata/quorum only, no VM data). Specify vSAN cluster configuration: stripes=1, replicas=2 (for 4 hosts, this ensures each site has full copy), failures to tolerate (FTT)=1. Include network topology: dedicated 10Gbps replication link between Site-A and Site-B for vSAN traffic, dedicated NSX overlay network for VM communication, separate management VLAN at each site with redundant NSX Edges for cross-site routing. Create a Visio or ASCII diagram showing host placement, replication links, and network zones.

Design the Tier-2 SRM + vSphere Replication topology. Create two separate vSphere clusters: primary at Site-A (4 nodes, local vSAN storage) and secondary at Site-B (2-4 nodes, local vSAN storage, initially powered down or in standby). Document vSphere Replication configuration: Recovery Point Objective = 1 hour (set replication interval to 60 minutes or smaller for granularity), Recovery Verification Test (RVT) scheduled daily, replication network isolated to dedicated VLAN to avoid impacting production traffic. Design the SRM recovery plan: specify which VMs from Site-A will replicate to Site-B, network mapping configuration (IP address remapping from Site-A subnet 10.0.0.0/24 to Site-B subnet 10.1.0.0/24), and NSX Tier-1 router migration (power off Source Router at Site-A, activate Destination Router at Site-B, update BGP weight on destination Edge to attract traffic). Include failback plan: resync Site-A from Site-B, reverse network mappings, reactivate Site-A primary systems. Create SRM recovery plan diagram and failover playbook.

Design the Tier-3 backup-based DR topology. Specify VADP backup policy: full backup on weekends, incremental backups daily, scheduled at 2 AM to avoid business hours. Define backup retention: 7-day incremental chain for point-in-time recovery, 4-week full backup retention (for 4hr RTO requirement, restore from most recent backup). Document backup storage: 6 TB local or cloud storage for 2.5TB Reporting System with compression ratio 2:1, dual-site backup copies (Site-A primary, Site-B secondary for disaster recovery of backup infrastructure itself). Define recovery procedures: restore RTO 4 hours assumes 1.5 hours for backup retrieval + 1 hour for storage re-initialization + 1.5 hours for application startup and data consistency checks. Include backup verification testing: monthly full restore to isolated cluster, document recovery time and data integrity validation. Create backup topology diagram and RTO calculation worksheet.

Consolidate NSX cross-site networking requirements for all tiers. Document NSX Tier-1 Logical Router configuration: distributed routing at Site-A and Site-B with BGP peering over Edge Cluster uplinks (BGP weight 100 at primary, 200 at secondary). Specify BGP failover convergence: typical 30-180 seconds depending on router configuration. Design failover behavior for each tier: Tier-1 stretched cluster failover is near-instantaneous (vSAN quorum-based), BGP weight change can be simultaneous; Tier-2 SRM failover requires manual BGP weight adjustment as part of recovery plan (documented step); Tier-3 backup restore means manual BGP re-pointing to Site-B backup infrastructure. Document DNS/DHCP handling: specify whether Site-B has read-only or writable DNS replicas, whether DHCP relay agents exist at both sites. Create an NSX Cross-Site Networking checklist.

Simulate a host failure in the Tier-1 stretched cluster. Gracefully power off one node at Site-A (e.g., host A2) or ungracefully disconnect it from the network (simulate network partition by disabling vSAN heartbeat). Monitor vSAN cluster status: expected behavior is immediate failover (within 60 seconds) to remaining majority (A1 + B1 + B2 + witness = quorum). Verify that all Trading Platform VMs continue running without interruption or data loss; use vSAN observer or vsphere CLI (vsan.health -a) to confirm cluster state. Document the Recovery Time from host failure to full cluster availability: should be <30 seconds. Document any dropped packets or storage IO stalls using vSphere performance monitoring (vSAN IO latency, VM packet loss).

Test SRM failover for Tier-2 Risk Analytics workload. Create a network partition between Site-A and Site-B (simulate replication link failure) and monitor vSphere Replication lag. Execute the SRM recovery plan for Risk Analytics VMs: (1) open SRM, navigate to Protection Groups, select Risk Analytics group, verify replication status (should show 'Ready'), (2) perform a Recovery Plan Test (non-disruptive) first: SRM creates temporary copies of VMs at Site-B without affecting Site-A, verify connectivity by pinging Site-B Recovery VMs, check that network mapping (10.0.0.0/24 → 10.1.0.0/24) is working, confirm database consistency (connect to Risk Analytics DB on Site-B test copy, verify record count matches Site-A), (3) clean up test copy, (4) execute actual failover: click 'Failover', confirm all VMs are migrated to Site-B, update BGP weight to redirect traffic to Site-B Edge (documented in playbook), verify access to Risk Analytics application (should be available within 20-30 minutes from failover start). Document the actual failover time and any manual interventions required.

Validate RPO compliance through replication lag monitoring. For Tier-1 and Tier-2, enable continuous replication lag monitoring. For Tier-1 (stretched vSAN), check the vSAN Replication Witness Latency metric (should be <5ms for healthy sync), and check the Replication Network Throughput to ensure sustained replication of workload changes (500GB workload with 5-10% daily change = 25-50GB/day change, should fit within replication window of 24 hours). For Tier-2 (vSphere Replication), open vSphere Replication dashboards, check the Replication Lag metric for each VM (target <60min for 1hr RPO). Simulate a peak load period: run a synthetic workload (e.g., IOmeter) on Tier-2 VMs that increases write IO by 50% for 1 hour, measure replication lag during peak (lag should not exceed 90min, otherwise RPO is violated). Document the test results and identify any conditions where RPO is at risk.

Document recovery runbook for each tier based on validated test results. For Tier-1, create a 'Stretched Cluster Failure Response' runbook: (1) detect cluster failure (monitor quorum), (2) if majority at Site-A, continue operations (no failover needed), (3) if majority at Site-B, check Site-A status (network partition vs. hardware failure), (4) if Site-A hosts are permanently lost, declare disaster, contact storage vendor to confirm vSAN data consistency, verify all critical VMs are running at Site-B, document incident. For Tier-2, create 'SRM Risk Analytics Failover Runbook': prerequisites (verify replication lag <60min, test failover succeeds), steps (contact incident commander, execute test failover, notify Site-B team to prepare, execute planned failover, update DNS/BGP, verify application access, begin recovery operations). For Tier-3, create 'Backup Restore Runbook': (1) identify backup point (most recent daily backup), (2) request backup restore from backup admin, (3) allow 4-6 hours for restore (verify actual time from validation test), (4) validate data integrity, (5) notify business stakeholders. Include contact info, escalation path, and success criteria for each runbook.

Create Decision Document DD-001 for Tier-1 Stretched Cluster DR. Structure the document with sections: (1) Business Context: Trading Platform must have zero data loss (RPO=0) and <5min recovery (RTO<5min); transactions are critical and any loss is unacceptable. (2) Requirements (R): must support synchronous replication (RPO=0), must tolerate 1 host failure without loss of quorum (requires witness), must meet 1ms latency SLA between sites, must support 500GB workload with 5-10% daily change rate (25-50GB/day replication traffic). (3) Constraints (C): bandwidth limitation (10Gbps link available), latency requirement (<1ms for sync replication, achievable only with dedicated link and fiber optic connectivity), licensing (requires vSAN Enterprise), staffing (requires 2 dedicated SREs trained on vSAN stretch cluster troubleshooting). (4) Assumptions (A): Site-to-site latency will remain <1ms (assume fiber link stability, no network overload), witness host will remain available 99.99% uptime (host placement at neutral Site-C or HA-protected), workload change rate will not exceed 10% daily (if trading volume increases >10% change/day, architecture must be revisited). (5) Risks (R) with IMPACT and MITIGATION: Witness host failure (IMPACT: split-brain, VMs may diverge on two sites; MITIGATION: Host HA on witness, daily failover tests, automated alerting on witness heartbeat loss); Replication link failure (IMPACT: if latency spike >5ms, replication throughput drops, queuing delays increase, RPO drifts; MITIGATION: dedicated 10Gbps link, bandwidth reservation QoS, lag monitoring alert at 5min, failover to Site-B if lag approaches 5min); Workload change rate exceeds capacity (IMPACT: replication lag accumulates, RPO violations; MITIGATION: cap trading volume in DB max row insert rate, monitor change rate continuously, scale up to 40Gbps link if needed). Include alternatives analysis: (1) Multi-site active-active with conflict resolution (rejected due to licensing cost for multiple clusters, complexity of distributed transaction handling), (2) SRM + vSphere Replication for Tier-1 (rejected: async replication cannot meet RPO=0). Conclude with recommendation: 'Implement stretched vSAN cluster for Tier-1 based on RPO=0 requirement, pending confirmation that <1ms latency is achievable.'

Create Decision Document DD-002 for Tier-2 SRM + vSphere Replication DR. Structure document: (1) Business Context: Risk Analytics workload has 30min RTO and 1hr RPO; data loss of up to 1 hour is acceptable, but >30min recovery would impact daily risk reporting. (2) Requirements: support asynchronous replication (RPO up to 1 hour acceptable), support failover to standby cluster in <30min, include network remapping for IP address translation between sites, must include NSX policy failover for firewall/NAT rules, must support Tier-2 application startup on Site-B (dependency ordering, app readiness checks). (3) Constraints: SRM licensing limits number of VMs (check VCF edition), replication network link limited to 1Gbps (shared with other traffic, not dedicated like Tier-1), standby cluster at Site-B has fewer nodes (2-4) to reduce cost, recovery plan complexity grows with number of VMs. (4) Assumptions: vSphere Replication lag will remain <60min under normal operations (workload change rate <5GB/hour), NSX policies are identical on both sites (pre-configured, not replicated), application startup is idempotent (can be safely restarted at Site-B after failover), database consistency is maintained during async replication (app-level transactions are ACID, no application-specific replication needed). (5) Risks: Replication lag exceeds 60min during business hours (IMPACT: RPO violation, Site-B data stale; MITIGATION: monitor lag, alert at 45min, consider upgrade to dedicated replication link if peak hours consistently breach threshold), NSX policy drift between Site-A and Site-B (IMPACT: failover succeeds but firewall blocks traffic; MITIGATION: version control NSX policies, automated policy sync or annual audit to ensure Sites are identical), application dependency ordering failures (IMPACT: database starts before file services, transactions fail; MITIGATION: define startup dependency sequence in recovery plan, test monthly), human error during SRM failover (IMPACT: wrong recovery plan executed or step skipped; MITIGATION: SRM runbook with checklist, two-person approval for planned failover). Alternatives: (1) Stretch cluster for Tier-2 (rejected: 1ms latency not critical for 30min RTO, cost of sync replication infrastructure not justified), (2) Backup-only for Tier-2 (rejected: 1-4hr restore time cannot meet 30min RTO). Conclude: 'Implement SRM + vSphere Replication for Tier-2 based on RTO/RPO balance and cost efficiency.'

Create Decision Document DD-003 for Tier-3 Backup-Based DR. Structure document: (1) Business Context: Reporting System is non-critical, used for historical analysis and not time-sensitive; 4hr RTO and 24hr RPO are acceptable for business continuity. (2) Requirements: backup system must support daily incremental snapshots, weekly full backups, retention of 7-day incremental chain and 4-week full backups (covering up to 28 days of recovery points), backup must be stored off-site or on secondary storage at Site-B, restore process must be documented and tested quarterly to ensure 4hr RTO is achievable. (3) Constraints: backup storage is shared infrastructure (limited to 6TB for 2.5TB data with 2:1 compression), backup link is 1Gbps (shared with replication and other traffic, limiting restore speed), backup restore depends on manual intervention (no automated recovery plan, requires human to initiate restore and validate), staff resources for backup management are limited (1 FTE). (4) Assumptions: data does not need to be point-in-time consistent (application can accept 24hr RPO, no transactions are lost, just day-old snapshots), restore from backup does not require parallel restore paths (serial restore acceptable within 4hr window), storage administrators can reliably execute restore procedures without error (assumption on staff competency). (5) Risks: Backup storage failure (IMPACT: cannot restore Reporting System after disaster, hours/days of data unavailable; MITIGATION: dual-site backup copies, monthly restore validation test, alert if backup copy fails), Restore exceeds 4hr RTO (IMPACT: Reporting System unavailable for extended period; MITIGATION: test actual restore time with full 2.5TB workload monthly, if test shows >4hr, reduce retention window or upgrade link bandwidth), RPO violation from missed backups (IMPACT: if backup job fails and is not noticed, next available backup is >24hr old; MITIGATION: backup job monitoring, alert if backup fails, daily verification that backup completed successfully). Alternatives: (1) Tier-2 SRM for Reporting System (rejected: cost of continuous replication not justified for non-critical workload, 30min RTO is overkill), (2) Snapshot-based replication (rejected: snapshot cost and storage overhead higher than backup approach for this use case). Conclude: 'Implement VADP backup-based DR for Tier-3 based on RTO/RPO requirements and cost efficiency.'

Create an Executive Summary document consolidating DD-001, DD-002, DD-003 into a single 1-page overview for stakeholder approval. Summary should include: (1) Overall DR architecture for all 3 tiers in a single table (Tier | RTO/RPO | Technology | Key Features | Cost Estimate), (2) Risk Register: top 5 risks across all tiers with IMPACT, MITIGATION, and Residual Risk (after mitigation), (3) Critical Success Factors: assumptions that must hold for DR to work (e.g., <1ms latency for Tier-1, NSX policies synchronized for Tier-2, backup link available for Tier-3), (4) Implementation Schedule: Tier-1 (months 1-3, stretch cluster build-out), Tier-2 (months 2-4, SRM configuration and testing), Tier-3 (months 1-2, backup policy configuration), (5) Approval signatures: CTO/VP Eng, Storage Lead, Networking Lead, Backup Admin, App Owner (Risk Analytics and Reporting), (6) Success Metrics: all 3 tiers tested and validated, zero RPO violations during monthly monitoring, failover runbooks approved and team trained, recovery RTO measured within 10% of design estimate. Include a visio diagram showing all three tiers and data flow paths for quick reference.