End-to-End Architecture Design Scenario

Design cluster topology and sizing:

Domain architecture:

1. Management Domain (4 hosts):

• vCenter, SDDC Manager, NSX Manager (3-node), Aria Suite
• Sizing: 4 × 64 cores, 4 × 512 GB RAM
• Storage: 15 TB vSAN ESA, FTT=1 RAID-5

2. Workload Domain 1 — Clinical (production):

• Cluster A: EHR + PACS (stretched cluster, 8+8 hosts across 2 sites)
  • 70 VMs (40 EHR + 30 PACS)
  • Storage challenge: 170 TB usable (20 TB EHR + 150 TB PACS)
  • Host spec: 64 cores, 768 GB RAM, 8× 7.68 TB NVMe (for PACS storage density)
  • Raw per host: 61.44 TB; 16 hosts total = 983 TB raw

     - After FTT=1 site mirror + 30% slack + overhead: ~330 TB usable → sufficient

• Cluster B: Clinical Applications (4 hosts)
  • 120 VMs, 15 TB storage
  • Standard host spec: 64 cores, 512 GB RAM, 4× 3.84 TB NVMe

3. Workload Domain 2 — Business (4 hosts):

• 100 business VMs, 12 TB storage
• Standard host spec

4. Workload Domain 3 — Research (separate domain, 4 hosts):

• 60 research VMs, 30 TB storage
• Isolated: Separate vCenter, separate NSX transport zone
• Higher compute density: GPU-equipped hosts for genomics analysis

5. Dev/Test cluster (within WD-2, 4 hosts):

• 100 VMs, 8 TB storage
• Lower-spec hosts acceptable

Total: 4 + 16 + 4 + 4 + 4 + 4 = 36 hosts (primary site + DR combined)

Design storage architecture:

Challenge: PACS drives 64% of total storage (150 TB) with unique characteristics:

• Large sequential writes (DICOM image storage)
• Random reads (radiologist viewing)
• Very low dedup ratio (~1.1:1 for medical images)
• Latency requirement: <5ms (radiologist workflow SLA)

Storage design:

1. EHR + PACS cluster (stretched, 16 hosts):

• Hosts: 8× 7.68 TB NVMe per host = 61.44 TB raw per host
• Per-site raw: 8 × 61.44 = 491.52 TB
• FTT=1 site mirror: Data mirrored between sites (effective 2× overhead)
• After mirror: 491.52 / 2 = 245.76 TB per-site usable before slack

   - After 30% slack: 245.76 × 0.70 = 172 TB usable → sufficient for 170 TB

• Headroom: ~1% — tight. Plan expansion at 70% utilization alert.

2. Storage policy design:

Note: PACS uses compression-only (not dedup) because medical images have near-zero dedup ratio.

Design NSX network with healthcare segmentation:

Segment design (healthcare-specific zones):

Critical DFW rules:

- BLOCK: Research ↔ Clinical (any direction, any port) — IRB requirement
- BLOCK: Dev/Test → Clinical (any direction) — prevent test data contamination  
- ALLOW: Clinical-Apps → EHR (TCP/1433 for Epic Caché DB)
- ALLOW: PACS → Clinical (DICOM port TCP/104, HTTPS/443)

• LOG ALL: Any traffic touching PHI-Clinical zone (HIPAA audit requirement)

Design encryption and HIPAA compliance controls:

1. Encryption architecture:

• Data at rest: vSAN encryption (AES-256) — ALL workload domains
• Data in transit: vSAN in-transit encryption between hosts
• VM-level: vTPM on all clinical VMs (required for HIPAA)
• Key management: External KMS (Thales CipherTrust) for production, NKP for dev

   - Key hierarchy: KEK (KMS) → DEK (per-host) → standard two-tier model

2. HIPAA Security Rule mapping:

3. Audit log architecture:

• Sources: vCenter, NSX DFW, SDDC Manager, ESXi, vSAN
• Collector: Aria Operations for Logs (primary), external SIEM (secondary)
• Retention: 90 days hot (Aria Ops for Logs), 6 years cold (archive to NFS)

   - Archive storage: ~2 TB/year of compressed logs → 12 TB for 6-year retention

• Integrity: Log checksums to prevent tampering (HIPAA requirement)

Design HA for EHR/PACS (stretched cluster):

1. Stretched cluster configuration:

• Site A (Primary Hospital DC): 8 hosts
• Site B (DR DC): 8 hosts
• Witness: Medium OVA at administrative office (Site C)
• Inter-site: 100 Gbps dark fiber, 0.6ms RTT (15km)
• Preferred site: Site A

2. EHR-specific HA:

• Epic Caché database VMs: Highest restart priority
• Epic Hyperspace (client) VMs: High priority
• Epic Interconnect (integration engine): High priority
• Anti-affinity: Caché primary and shadow never on same host

   - VM restart order: Caché DB → Interconnect (120s delay) → Hyperspace (60s delay)

3. PACS-specific HA:

• PACS archive server: Highest priority (must be available for image retrieval)
• PACS web viewer: Medium priority (radiologists need within 5 minutes)
• DICOM gateway: High priority (incoming studies from imaging equipment)
• Site affinity: PACS VMs prefer Site A (where imaging equipment connects)

4. Availability calculation:

• Target: 99.997% = ≤15 min downtime/year
• HA detection: ~35 seconds (heartbeat miss + declaration)
• VM restart: ~2-3 minutes (boot + Epic service start)
• Total RTO: ~3.5 minutes per event
• Allowed events: 15 min / 3.5 min = ~4 events per year
• vSAN stretched cluster + HA + anti-affinity achieves this target

Design DR for clinical and business applications (SRM):

1. SRM configuration for clinical apps (120 VMs):

• RPO: 1 hour (vSphere Replication)
• RTO: 30 minutes (SRM recovery plan)
• DR hosts: 4 hosts at Site B (separate from stretched cluster)

   - Over-subscription: 120 VMs at DR on 4 hosts → acceptable with degraded performance
     - Normal: 4 vCPU avg, 3:1 overcommit → 4 hosts handle 768 vCPU
     - DR mode: 120 VMs × 4 vCPU = 480 vCPU → within capacity

2. Recovery plan for clinical apps:

• Priority Group 1: Laboratory Information System (10 VMs)
• Priority Group 2: Pharmacy system (8 VMs)
• Priority Group 3: Radiology viewers (15 VMs)
• Priority Group 4: Remaining clinical apps (87 VMs)
• Startup delays: 120s between groups
• Post-recovery: Verify HL7 interface connectivity to EHR

3. Business application DR:

• RPO: 4 hours
• RTO: 2 hours
• DR hosts: Shared with clinical DR hosts (lower priority in recovery plan)
• Recovery plan: Executed AFTER clinical apps are confirmed running

4. Research DR:

• RPO: 24 hours (daily backup)
• RTO: 4 hours
• No dedicated DR hosts — restore to available capacity at Site B
• Lowest priority in any multi-tier recovery scenario

Write the failover runbook:

Runbook: EHR/PACS Stretched Cluster — Site A Failure

Step 1: Detection (0-5 minutes)

• [ ] vSAN Health alert received: Site A hosts unreachable

- [ ] Verify: Ping Site A management IPs from Site B → No response
- [ ] Verify: Check IPMI/iLO at Site A → No response (confirms site failure, not network partition)

• [ ] Escalation: Page on-call infrastructure lead AND clinical informatics officer

Step 2: Automatic Failover (5-10 minutes)

• [ ] Confirm: HA has restarted EHR VMs on Site B (check vCenter events)
• [ ] Confirm: Caché database shadow promoted to primary (Epic-specific verification)
• [ ] Confirm: PACS archive server accessible from radiology workstations

- [ ] Monitor: vSAN health → verify all objects accessible at Site B

Step 3: Clinical Verification (10-20 minutes)

- [ ] Clinical test: Open Epic Hyperspace client → verify patient record retrieval
- [ ] Clinical test: Open PACS viewer → verify recent study retrieval
- [ ] Clinical test: Send test HL7 message through Interconnect → verify integration

• [ ] Notify: Clinical informatics that EHR is operational on DR site
• [ ] Notify: Radiology department that PACS is operational

Step 4: Secondary Recovery (20-60 minutes)

• [ ] Assess: Are clinical app VMs (SRM) also affected?
  • If yes: Execute SRM recovery plan for clinical apps
  • If no: Clinical apps still running on Site A infrastructure (separate failure domain)
• [ ] Monitor: Aria Operations dashboards for performance degradation
• [ ] Document: Timeline of events for incident report

Step 5: Communication (ongoing)

• [ ] Update: Hospital incident command (if activated)
• [ ] Update: VMware Support SR (Severity 1)
• [ ] Update: Executive stakeholders every 30 minutes until stable

Step 6: Post-Incident (24-72 hours)

• [ ] Root cause analysis
• [ ] Site A recovery and vSAN resync
• [ ] DRS failback to restore site affinity
• [ ] Update risk register with lessons learned

Design DR testing for healthcare:

1. Testing constraints:

• Clinical systems cannot have extended downtime for testing
• SRM test recovery uses isolated network (no production impact)
• Stretched cluster planned failover during lowest-census period (Sunday 2-6 AM)
• Joint Commission and HIPAA require documented DR testing

2. Annual DR test calendar:

3. Success criteria for each test:

• EHR functional: Patient search, order entry, result review within 5 minutes
• PACS functional: Study retrieval, image display within 10 seconds
• Clinical apps: HL7 message processing within 60 seconds
• RTO measured: Stopwatch from failover initiation to clinical verification
• Documentation: Test report filed within 5 business days

4. HIPAA DR documentation:

• HIPAA § 164.308(a)(7)(ii)(D): Testing and revision procedures
• Requirement: Regular testing of contingency plan
• Evidence: DR test reports with timestamps, success/failure, corrective actions
• Retention: 6 years (align with other HIPAA records)

Compile healthcare-specific design decisions:

Create the risk register with healthcare-specific risks:

Top risks (score ≥12): HR-002 (data breach), HR-003 (ransomware), HR-005 (PACS capacity), HR-008 (upgrade failure)

Prepare VCDX defense for healthcare challenges:

Challenge 1: 'Your PACS cluster is at 98% capacity after 3 years of growth. What do you do?'
Response: The design includes a 70% capacity alert triggering procurement with 12-week lead time. At 98%, emergency options: (1) Enable HCI Mesh from underutilized business cluster for temporary overflow, (2) Archive older studies (>5 years) to external NFS with DICOM viewer redirect, (3) Add hosts to PACS cluster via SDDC Manager. Long-term: Evaluate tiered storage with cold tier for archived images.

Challenge 2: 'How do you handle a ransomware attack that encrypts both data sites?'
Response: The design includes immutable backups with an air-gapped recovery environment. Recovery procedure: (1) Isolate affected segments via DFW emergency quarantine rule, (2) Stand up clean vCenter from air-gapped backup, (3) Restore VMs from immutable backup to isolated recovery network, (4) Forensic analysis of infection vector, (5) Validate clean state before reconnecting to production. RTO: 4-8 hours for EHR, 24 hours for full environment. Immutable backups use WORM storage that cannot be encrypted by ransomware.

Challenge 3: 'A researcher needs access to clinical data for a study. How does your design handle this?'
Response: The design enforces complete isolation between research and clinical domains (separate workload domain, separate transport zone, DFW absolute deny). For approved studies: (1) Clinical data team de-identifies patient data per HIPAA Safe Harbor or Expert Determination method, (2) De-identified dataset is exported to a staging area, (3) Staging area is scanned for residual PHI, (4) Approved dataset is transferred to research domain via audited one-way data pipeline, (5) Full audit trail logged for IRB compliance.

Challenge 4: 'Why not use a single workload domain with DFW rules instead of 4 separate domains?'
Response: HIPAA requires demonstrable isolation — DFW rules can be misconfigured, but separate workload domains provide domain-level isolation that survives DFW errors. Separate domains also enable independent upgrade schedules (clinical cannot wait for research domain upgrades), separate vSAN datastores (research workloads cannot impact clinical storage performance), and cleaner audit scope (HIPAA audit covers clinical domain only, not research).

Create the final design summary and quality checklist:

1. Architecture summary:

• 4 workload domains + 1 management domain = 5 domains
• 36 hosts total (including DR)
• 2 sites + 1 witness
• 235 TB usable storage (150 TB PACS dominant)
• VCF 9.0 with vSAN ESA, NSX, Aria Suite

2. Compliance summary:

• HIPAA Security Rule: 6 § mapped to VCF controls
• HITRUST CSF: Architecture supports certification timeline
• Audit: 6-year log retention, quarterly DR testing, annual access review

3. Cost summary:

• 36 hosts × 64 cores × 2 sockets = 4,608 licensed cores
• VCF per-core licensing + external KMS + Aria Suite
• Target: $4.5M capex (verify against host + storage + networking BOM)
• Opex: $800K/year (licensing subscription + support + operations staff)

4. Final quality checklist:

• [ ] Every HIPAA § has a corresponding VCF control
• [ ] Research isolation verified at domain + network + DFW + storage levels
• [ ] PACS storage sized for 3-year growth with alerts
• [ ] Failover runbook includes clinical-specific verification steps
• [ ] Air-gapped depot documented for SDDC Manager
• [ ] 6-year log archival sized and designed
• [ ] All 10 design decisions trace to requirements
• [ ] All 8 risks have specific mitigations
• [ ] VCDX defense responses prepared for 4+ healthcare challenges
• [ ] Design internally consistent — no contradictions between sections