vSAN Storage Design with Erasure Coding

Design storage policies for each workload tier:

Policy: Tier-1-Critical

• Failures to tolerate (FTT): 2
• Failure tolerance method: RAID-6 (erasure coding)
• Object space reservation (thick provisioning): 25% (for database workloads)
• IOPS limit: None (unlimited)
• Force provisioning: No
• Deduplication and compression: Enabled (ESA — compression-only for database if dedup ratio < 1.5:1)
• Encryption: Enabled (vSAN native encryption, AES-256)
• Checksum: Enabled (object-level CRC)

Policy: Tier-2-Business

• FTT: 1
• Failure tolerance method: RAID-5
• Object space reservation: 0% (thin provisioned)
• IOPS limit: 2,000 per VM (prevent noisy neighbor)
• Force provisioning: No
• Deduplication and compression: Enabled
• Encryption: Enabled

Policy: Tier-3-DevTest

• FTT: 1
• Failure tolerance method: RAID-5
• Object space reservation: 0%
• IOPS limit: 1,000 per VM
• Force provisioning: Yes (allow provisioning even if policy cannot be fully satisfied — acceptable for dev/test)
• Deduplication and compression: Enabled
• Encryption: Optional (depends on data classification)

Design IOPS and throughput controls:

vSAN QoS (IOPS limits):

• Purpose: Prevent noisy-neighbor effect where one VM starves others
• Tier-1: No limit (critical workloads get priority)
• Tier-2: 2,000 IOPS per VMDK (sufficient for most business apps)
• Tier-3: 1,000 IOPS per VMDK (dev/test can tolerate throttling)

Storage I/O Control (SIOC) — not available on vSAN (uses native QoS instead)

Document expected IOPS per host:

• ESA with NVMe: ~200K-400K IOPS per host (depending on device and workload)
• 8-host cluster: ~1.6M-3.2M IOPS total (theoretical)
• Overhead from RAID-5: ~15-20% reduction in write IOPS
• Target: maintain <5ms latency at P99 for Tier-1 workloads

Plan deduplication and compression:

ESA dedup/compression behavior:

• Operates inline (during write) — no post-process cycle needed
• ESA dedup is block-level (4K granularity)
• Compression: LZ4 (fast, moderate ratio) applied after dedup
• Cannot selectively enable/disable per VM — it is a cluster-wide setting in ESA

(Note: In ESA, dedup+compression is either ON or OFF for the entire cluster)

Expected ratios by workload type:

Apply to capacity calculation:

• Conservative estimate: 2.0:1 combined ratio
• 62 TB usable (FTT=1 RAID-5) × 2.0 = 124 TB effective capacity
• This changes the fit analysis: now sufficient for 83 TB requirement with headroom

Design vSAN datastore layout:

VCF architecture: Each workload domain gets its own vSAN datastore automatically

• Management domain: vSAN datastore for management VMs (dedicated)
• Workload Domain 1 (Tier-1): vSAN datastore with Tier-1-Critical policy as default
• Workload Domain 2 (Tier-2/3): vSAN datastore with Tier-2-Business policy as default

Additional datastores:

• vSAN HCI Mesh (if needed): Share unused capacity across workload domains without vMotion
  • Use case: Workload Domain 2 has excess capacity; Workload Domain 1 can mount it remotely
  • Constraint: Remote datastore performance is lower (network hop); suitable for Tier-3 only

• NFS/VMFS datastores (if external storage exists):
  • Use for backup targets, ISO libraries, templates
  • Never mix vSAN + external storage for the same workload tier (inconsistent performance)

Document: Default storage policy per datastore, exceptions, and the policy change workflow.

Design key management hierarchy:

vSAN encryption uses a two-tier key model:

1. Key Encryption Key (KEK): Stored in external KMS or vCenter Native Key Provider

• Encrypts/wraps the DEK
• Rotated annually (or per compliance policy)
• If KEK is lost: data is permanently inaccessible

2. Data Encryption Key (DEK): Generated per host, stored locally (encrypted by KEK)

• Used for actual data encryption (AES-256-XTS)
• Re-wrapped when KEK rotates (data is NOT re-encrypted)
• Shallow re-key: New KEK wraps existing DEK (fast, no I/O)
• Deep re-key: New DEK generated, data re-encrypted (slow, I/O intensive)

Key Provider options:
A) vCenter Native Key Provider (NKP):

• Built into vCenter 7.0u2+ and 9.0
• No external KMS needed
• Keys backed up via vCenter backup
• Limitation: Single point of failure if vCenter backup is lost

B) External KMS (KMIP 1.1 compliant):

• Examples: HyTrust KeyControl, Thales CipherTrust, Entrust
• HA: Deploy KMS cluster (2+ nodes)
• Stronger compliance posture (separation of key management from infrastructure)

Design decision: External KMS for production (SOC 2 requirement), NKP for dev/test (cost savings).

Configure encryption layers:

Layer 1 — vSAN Data-at-Rest Encryption:

• Enabled at cluster level
• Encrypts all data on vSAN datastore (VM files, swap, snapshots, metadata)
• Performance impact: 5-10% overhead on write IOPS (AES-NI hardware acceleration)
• Verify: Hosts must have AES-NI instruction support (all modern Xeon/EPYC do)

Layer 2 — VM Encryption (vSphere-level, optional):

• Per-VM encryption using storage policy
• Encrypts VMDK at the vSphere layer before reaching vSAN
• Double encryption warning: Enabling both vSAN encryption AND VM encryption creates double encryption overhead with no security benefit
• Recommendation: Use vSAN encryption for all VMs; add VM encryption only for cross-vCenter vMotion scenarios (encrypts data in transit)

Layer 3 — vSAN in-transit encryption:

• Encrypts vSAN data replication traffic between hosts
• Options: Cluster-only (within cluster) or Preferred/Required
• Performance: Adds ~5% latency to vSAN replication
• Recommended: Enable for compliance environments

Design key rotation and lifecycle:

1. Key rotation schedule:

• KEK rotation: Annually (shallow re-key, minimal impact)
• DEK rotation: Every 3 years or after suspected compromise (deep re-key, plan maintenance window)
• Post-incident rotation: Immediate shallow re-key after any security event

2. Key backup and recovery:

• NKP: Keys included in vCenter file-level backup; backup to separate secure location
• External KMS: KMS cluster handles replication; backup KMS database independently
• Test recovery: Quarterly test of key recovery procedure (restore vCenter from backup, verify data accessibility)

3. Failure scenarios:

• KMS unavailable during host boot: Host cannot mount vSAN datastore; VMs do not start

Mitigation: KMS cluster with 2+ nodes across failure domains

• vCenter down: Existing hosts continue running (keys cached in memory); new hosts cannot join

Mitigation: Ensure vCenter HA is configured; documented break-glass procedure for emergency vCenter recovery

• KEK lost permanently: Data is UNRECOVERABLE

Mitigation: Multiple KMS backup copies in geographically separate locations

Create encryption design decision entry:

Decision D-006 — Encryption Architecture:
Decision: vSAN data-at-rest encryption with external KMS for Tier-1/2, NKP for Tier-3
Alternatives:
A) VM-level encryption per policy — Rejected: more complex policy management, double encryption risk, vMotion encryption is separate anyway
B) No encryption, rely on physical security — Rejected: SOC 2 requires encryption at rest, drive disposal risk
C) Full-disk encryption (SED drives) — Rejected: vendor lock-in, limited key management flexibility, no integration with vSphere policy framework

Compliance mapping:

• SOC 2 CC6.1: Encryption of data at rest ✓
• SOC 2 CC6.7: Key management separation ✓ (external KMS)
• NIST 800-171: FIPS 140-2 validated algorithm (AES-256) ✓

Document the key management network design:

• KMS communication: KMIP port 5696 (TCP)
• Network: Dedicated management VLAN, firewalled

- vCenter → KMS: Mutual TLS authentication (client certificate)

Design vSAN Health monitoring integration:

1. vSAN Health Service checks to monitor:

• Cluster health: Overall status, network partition detection
• Disk health: Disk balance, disk latency, congestion
• Data health: Object health, policy compliance, stale components
• Network health: vSAN VMkernel connectivity, multicast/unicast status
• Physical disk health: SMART data, wear indicators for NVMe

2. Aria Operations integration:

• Enable vSAN management pack
• Custom dashboards: Capacity trending, IOPS distribution, resync progress

   - Alerts: NVMe wear level >80% → Warning; >90% → Critical
   - Capacity alert thresholds:
     - 70% used → Warning (plan procurement)
     - 80% used → Critical (restrict new VM provisioning)
     - 85% used → Emergency (begin workload migration or expansion)

3. Skyline Health (Broadcom):

• Proactive support: Auto-detects known issues against KB articles
• Requires outbound HTTPS — for air-gapped: use Skyline Collector appliance

Design maintenance mode behavior:

Three evacuation modes:

1. Ensure data accessibility (fastest):

• Moves data components to maintain accessibility but does not rebuild full protection

   - Data is temporarily at reduced FTT (e.g., FTT=2 → FTT=1 during maintenance)

• Use for: Short maintenance (<4 hours), patching, firmware updates

2. Full data migration (slowest, safest):

• Rebuilds all components on remaining hosts to maintain full FTT
• Requires sufficient free capacity on other hosts
• Use for: Host decommission, extended maintenance (>24 hours)
• Capacity requirement: Host's capacity × FTT multiplier must fit on remaining hosts

3. No data migration (fastest, risk):

• Does not move any data; VMs are migrated but data components remain on offline host
• If another host fails during maintenance, data loss possible
• Use for: Quick reboot only (< 15 minutes)

Design decision: Default to 'Ensure accessibility' for patching; require change approval for 'No data migration'

Plan disk replacement and expansion procedures:

1. Disk failure replacement:

• vSAN ESA auto-detects disk failure and begins component rebuild on remaining disks
• Rebuild time depends on data volume and cluster load (~100 GB/hour estimate)
• During rebuild: Performance degraded (~20-30% IOPS reduction cluster-wide)
• After physical replacement: vSAN reclaims new disk automatically (ESA — no disk group concept)

2. Capacity expansion options:

a) Add disks to existing hosts:

• Hot-add NVMe supported on most server platforms
• vSAN discovers new disk, adds to storage pool
• Rebalance starts automatically (proactive rebalance in ESA)

b) Add hosts to cluster:

      - VCF workflow: SDDC Manager → Add Host to Cluster

• New host contributes storage pool capacity + compute
• Minimum host add: 1 host at a time
• Post-add: Enable vSAN proactive rebalance for even distribution

c) HCI Mesh (share capacity across domains):

• Remote mount of vSAN datastore from another cluster
• Latency: +1-2ms over direct local access
• Use case: Short-term capacity relief while procurement completes

Create operational design documentation:

1. Capacity planning cadence:

• Monthly: Review vSAN capacity usage trending in Aria Operations
• Quarterly: Procurement forecast based on growth rate
• Annual: Full capacity audit with lifecycle refresh planning

2. Maintenance window design:

• Host maintenance: Serial, one host at a time, minimum 4 hours between hosts
• Reason: vSAN resync must complete before next host enters maintenance
• Monitor: vSAN resync dashboard — proceed only when resync objects = 0
• Sequence: Follow VCF upgrade order (KB 390634) — SDDC Manager first, then management domain, then workload domains

3. Design decision entry:

Decision D-007 — Maintenance Mode Default:
Decision: 'Ensure data accessibility' as default evacuation mode
Alternative: 'Full data migration' — Rejected as default: too slow for routine patching (8+ hours per host in large clusters)
Alternative: 'No data migration' — Rejected as default: unacceptable data-loss risk during host reboot failure
Risk: If second host fails during maintenance with 'Ensure accessibility', data at reduced FTT
Mitigation: Health pre-check before maintenance; abort if any components already in degraded state