Academy/VCAP — VCF Administrator (3V0-11.26)
VCF

VCAP — VCF Administrator (3V0-11.26)

VCF 9.0vcapadminvcdx

VCAP-level VCF administration covering fleet lifecycle management, Private Cloud/Fleet/Instance hierarchy, identity & certificate management, configuration drift & compliance, advanced troubleshooting with SoS & log bundles, vSAN ESA operations, and VCF 9.0 day-2 operational excellence.

Version Evolution

VCAP-Admin exam covers advanced VCF day-2 operations. VCF 9.0 shifted management from SDDC Manager UI to VCF Operations and vSphere Client. vSAN ESA replaces OSA as the default storage architecture. Identity management gains OIDC federation support. The exam tests scenario-based troubleshooting and operational decision-making at scale.

Learning Outcomes

  • Master VCF 9.0 fleet lifecycle management: bring-up, expand, shrink, upgrade, and decommission operations across Private Cloud, Fleet, Instance, and Domain hierarchy
  • Configure and manage identity federation (vCenter SSO, AD/LDAP, OIDC), certificate lifecycle (VMCA, custom CA, certificate rotation), and password policies at scale
  • Detect and analyse configuration drift of vCenter instances and vSphere clusters with VCF Operations Configuration Drifts (configuration templates assigned by policy, Detect Drift, Drift Status Report), and remediate in vCenter or by reconciling the template
  • Execute advanced troubleshooting using SoS utility, log bundles, REST API diagnostics, and component health checks
  • Manage vSAN ESA operations: disk lifecycle, maintenance mode, data evacuation, and fault domain management
  • Implement VCF security hardening: STIG compliance, ESXi lockdown mode, TPM attestation, and audit logging

VCF 9.0 Fleet Lifecycle Management#

VCF 9.0 introduces a hierarchical management model: Private Cloud → Fleet → Instance → Domain. Understanding this hierarchy and the lifecycle operations at each level is critical for VCAP-level administration.

VCF 9.0 Taxonomy & Hierarchy:


Private Cloud (highest level):

  • Represents the entire VMware Cloud Foundation deployment
  • Contains one or more Fleets
  • Managed by VCF Operations (central management plane)
  • Policy and compliance scope: organization-wide baselines

Fleet:

  • A logical grouping of VCF Instances
  • Used for grouping by geography, business unit, or operational boundary
  • Fleet-level policies cascade to all Instances within
  • Example: "EMEA Fleet", "APAC Fleet", "Production Fleet"

Instance:

  • A single VCF deployment with its own management components
  • Contains: 1 management domain + 0-N workload domains
  • Each Instance has its own: vCenter, NSX Manager, SDDC Manager (deprecated UI)
  • In VCF 9.0: Instance management shifts from SDDC Manager to VCF Operations and vSphere Client

Domain:

  • Management Domain: infrastructure services (vCenter, NSX, VCF Operations, etc.)
  • Workload Domain: tenant/application workloads
  • Each domain has: vSphere cluster(s), vSAN storage, NSX networking

Lifecycle Operations by Level:


Instance-Level Operations:

Bring-Up: Initial VCF deployment (VCF Installer → management domain of the first VCF Instance; the VCF Installer replaces the Cloud Builder appliance in 9.0 — VCF 9.0.1 release notes p.226)

Commission/Decommission Hosts: Add/remove ESXi hosts from the Instance pool

Create Workload Domain: Deploy new domain with cluster, vSAN, NSX

Expand Domain: Add cluster or hosts to existing domain

Shrink Domain: Remove hosts from domain (with vSAN data evacuation)

Delete Workload Domain: Remove domain entirely (destructive)

Fleet-Level Operations:

  • Add/Remove Instance: Manage Instance membership in Fleet
  • Fleet Policy Push: Apply configuration baselines to all Instances
  • Fleet Health Monitoring: Aggregate health view across Instances

Upgrade Lifecycle:

VCF Lifecycle Management (LCM) orchestrates upgrades across the entire stack:

  1. Pre-check: compatibility validation, health check, disk space
  2. Download: fetch update bundles from depot (online) or import (offline)
  3. Stage: prepare update on target components
  4. Upgrade sequence (VCF 9.0.1 pp.1261, 1366): management planes first (VCF Operations and fleet management, Aria Suite Lifecycle where present), then the management-domain core in order SDDC Manager and VCF services → NSX Manager (and Global Manager) → vCenter → ESX hosts; vSAN ships with ESX/vCenter and is not a separate step

Upgrade Modes:

  • Rolling: One host at a time, VMs migrate via vMotion (zero workload downtime)
  • Parallel: Multiple hosts simultaneously (faster but requires more spare capacity)
  • Maintenance Window: All hosts at once (requires VM downtime)

Host Lifecycle in VCF:


  1. Commission: Add host to SDDC Manager inventory
  • Host must meet VCF HCL requirements
  • Network connectivity validated (management, vMotion, vSAN, TEP VLANs)
  • Credentials stored in VCF credential store
  1. Add to Domain: Assign commissioned host to a workload domain cluster
  • ESXi configuration applied automatically (NTP, DNS, syslog, lockdown mode)
  • vSAN disk groups claimed
  • NSX transport node configuration applied
  • Host enters cluster and participates in HA/DRS
  1. Maintenance: Enter maintenance mode for patching/hardware
  • vSAN data evacuation mode selection:

a. Full data evacuation (safest, longest, ensures all data replicated to other hosts)

b. Ensure accessibility (faster, maintains access but may reduce resilience)

c. No data evacuation (fastest, risk of data loss if another host fails during maintenance)

  1. Remove from Domain: Reverse of add — vSAN evacuation, VM migration, NSX cleanup
  1. Decommission: Remove from SDDC Manager inventory (host returns to standalone ESXi)

Key Takeaways

  • VCF 9.0 hierarchy: Private Cloud (org-wide) → Fleet (logical grouping) → Instance (single VCF deployment) → Domain (management + workload). Each level has distinct lifecycle operations.
  • Host lifecycle: Commission → Add to Domain → Operational → Maintenance → Remove → Decommission. vSAN data evacuation mode selection is critical during maintenance operations.
  • LCM orchestrates upgrades across the full stack in dependency order: management planes (VCF Operations / fleet management) first, then SDDC Manager → NSX Manager → vCenter → ESX hosts (VCF 9.0 order). Rolling upgrades enable zero workload downtime.
  • SDDC Manager UI is deprecated in VCF 9.0 — functions migrated to VCF Operations and vSphere Client, but SDDC Manager API remains available during transition.

Identity, Certificate & Password Management#

VCF identity management spans SSO federation, certificate lifecycle, and password rotation across all components. At VCAP level, you must understand the interactions between vCenter SSO, Active Directory, OIDC providers, and the certificate chain of trust across the VCF stack.

Identity Architecture:


vCenter SSO (Platform Services Controller):

  • Central identity provider for all vSphere/VCF components
  • Embedded PSC in vCenter 7.0+ (no external PSC deployment)
  • Default domain: vsphere.local
  • Supports: local users, AD/LDAP, OIDC identity providers

AD/LDAP Integration:

vCenter → Administration → Single Sign-On → Identity Sources → Add

Types:

Active Directory over LDAP(S): bind with a service account; use LDAPS (Integrated Windows Authentication is removed in vCenter 9.0 — migrate to AD over LDAPS or identity federation with MFA, VCF 9.0.1 p.226)
Open LDAP: Non-Microsoft LDAP directories

Best Practices:

  • Use AD/LDAP for human administrators (centralized password policy)
  • Keep a local vsphere.local admin account for break-glass access
  • Map AD groups to vCenter roles (not individual users)
  • Service accounts for API integrations should be local SSO accounts

OIDC Federation (VCF 9.0):

VCF 9.0 supports OIDC identity providers (Okta, Azure AD, etc.)

  • Configuration: vCenter → Identity Provider → Add OIDC Provider
  • Benefits: MFA support, centralized identity governance, SSO across VCF + cloud services
  • Limitations: OIDC does not replace SSO — it federates INTO SSO as an identity source

Certificate Architecture:


VCF uses a hierarchical certificate model:

VMCA (VMware Certificate Authority):

  • Embedded CA in each vCenter (PSC component)
  • Issues certificates to: ESXi hosts, vCenter services, NSX components, Solution Users
  • Default: VMCA is the root CA (self-signed root)
  • Production (VCF): replace the default VMCA-signed certificates with trusted CA-signed certificates and manage them, including auto-renewal, in VCF Operations certificate management (VCF-SEC-RCMD-CERT-001); VMCA as subordinate CA under enterprise PKI is a vSphere option

Certificate Hierarchy:

Enterprise Root CA (optional)

    └── VMCA (subordinate or self-signed root)
          ├── ESXi host certificates
          ├── vCenter Machine SSL certificate
          ├── vCenter Solution User certificates
          ├── NSX Manager certificate
          └── VCF component certificates

Certificate Operations:

Replace VMCA root certificate:

  1. Generate CSR from VMCA
  2. Sign CSR with enterprise CA
  3. Import signed certificate + CA chain into VMCA
  4. VMCA re-issues all leaf certificates automatically

Replace individual certificates (custom mode):

  1. Generate CSR for specific service
  2. Sign with enterprise CA
  3. Import signed certificate via Certificate Manager
  4. Repeat for each service (labor-intensive, not recommended)

Certificate rotation in VCF:

  • VCF tracks certificate expiry across all components
  • Certificate lifecycle is run from the VCF Operations console (Fleet Management > Certificates): configure a CA (Microsoft CA or OpenSSL), generate CSRs, replace with CA-signed certificates, renew manually, or activate auto-renewal, which renews 60 days before expiry and is set separately for VCF Management and VCF Instance components; no documented link ties certificate rotation to upgrade cycles
  • VCF Operations provides certificate expiry notifications across the VCF components; the alert threshold is a design choice (for reference: NSX raises a medium alarm 30 days and a high alarm 7 days before expiry by default, configurable — VCF 9.0.1 p.2572; the SoS --certificate-health check reports YELLOW at 15-30 days and RED below 15 days — VCF 5.2 docs p.609)

Password Management:


VCF manages passwords for all deployed components:

Password Lifecycle:

  • VCF credential store holds passwords for: vCenter, NSX, ESXi, SDDC Manager, VCF Operations
  • Password rotation: VCF supports auto-rotation with configurable schedule
  • Rotation scope: per-component or fleet-wide
  • Emergency rotation: forced rotation after security incident

Password Policies:

  • Minimum length, complexity, history, lockout threshold
  • ESXi: /etc/pam.d/passwd (local) or vCenter-managed
  • vCenter: SSO password policy (Administration → Configuration → Local Accounts)
  • NSX: CLI and API password policies (independent from vCenter SSO)

Best Practices:

  1. Use AD/LDAP for human access — single password policy, centralized audit
  2. Rotate service account passwords quarterly via VCF auto-rotation
  3. Maintain a break-glass procedure for the vsphere.local administrator account
  4. Monitor VCF Operations for password expiry alerts across all components
  5. Document password dependencies — some passwords (ESXi root) affect VCF operations if changed outside VCF

Key Takeaways

  • Identity stack: vCenter SSO (central IdP) ← AD/LDAP (human admins) + OIDC (MFA/federation in VCF 9.0) + local accounts (break-glass). Map AD groups to vCenter roles, never individual users.
  • Certificate hierarchy: Enterprise Root CA → VMCA (subordinate) → leaf certs (ESXi, vCenter, NSX). In VCF, replace the default management-domain certificates with trusted enterprise CA-signed certificates from VCF Operations (Fleet Management > Certificates); VMCA subordinate mode is a vSphere option, and the certificate-manager CLI sits outside that VCF Operations workflow.
  • Password management: VCF credential store holds all component passwords. Auto-rotation is configurable. Never change ESXi root password outside VCF — it breaks lifecycle operations.
  • Certificate rotation: VCF Operations provides expiry notifications across all components. Replacement and renewal run from VCF Operations > Fleet Management > Certificates, with auto-renewal 60 days before expiry. Alert thresholds are a design choice; reference points are the SoS bands (YELLOW 15-30 days, RED under 15) and the NSX default alarms (30 and 7 days).

Configuration Drift Detection & Compliance#

Configuration drift occurs when infrastructure components deviate from their desired state. In VCF 9.0 the drift feature of VCF Operations is Configuration Drifts (exam objective: monitor VCF configuration using VCF Operations Config Drift). It works from configuration templates and applies to vCenter instances and vSphere clusters deployed in VCF 9.0 or later — not to ESXi hosts, NSX or vSAN objects, which are governed elsewhere (vSphere Configuration Profiles for cluster-level host settings; NSX and vSAN by their own tooling).

Configuration Drifts model:


Configuration template: a set of configurations representing a desired state, extracted from a source vCenter (Fleet Management > Configuration Drifts > Configuration Templates > Create Config Template). The schema defines which settings are tracked. Templates can be kept under source control (GitLab/GitHub) with merge-request review; each template is stored as <name>-desiredState.json, -metaData.json and -manifest.json.

Assignment: a template is attached to targets through a VCF Operations policy (Infrastructure Operations > Configurations > Policy Definition > policy > Configuration Templates > Activated). Each vCenter needs exactly one active policy carrying the template.

Privileges: Configuration Drifts Manage (create templates, assign to policy, run drift check) and View.

Enabled objects: vCenter 9.0+ instances with a template attached, plus vSphere clusters with Configuration Profiles enabled — shown in the Configuration Enabled Objects Summary bar of the Overview page.

Drift detection process:

  1. Detect Drift on one or more vCenter instances (on demand, or scheduled as an Automation Central job)
  2. Comparison per template: Current vCenter Value versus Config Template Value
  3. Status on the Overview charts (vCenter Instances Drift Status, Cluster Drift Status); 'No drift found for this Config Template since last configuration drift check' when clean
  4. Report: Download Drift Status Report — date, resource, template and drift status, 7 days by default, up to 30 via Global Settings

Remediation:

Remediation is not performed from the VCF Operations console. Revert the value in vCenter to the template value, or — when the new value is intended — edit the template so the desired state follows.
Exceptions: an object that must legitimately differ gets its own template via a separate policy, or the setting is left out of the shared template; there is no per-object exclusion tag.

STIG Compliance (Security Technical Implementation Guide):


DoD STIG profiles for vSphere/VCF are available as pre-built compliance baselines:

Categories:

  • CAT I: High severity — exploitable vulnerabilities (must fix immediately)
  • CAT II: Medium severity — potential vulnerabilities (fix within 30 days)
  • CAT III: Low severity — best practice deviations (fix within 90 days)

Common STIG checks for ESXi:

  • SSH disabled (CAT II)
  • Lockdown mode enabled (CAT II)
  • DCUI timeout configured (CAT III)
  • Persistent logging to remote syslog (CAT II)
  • TLS 1.2 minimum for all management interfaces (CAT I)
  • SNMP v3 only (v1/v2c disabled) (CAT II)
  • vMotion traffic encrypted (CAT II)

CIS Benchmark compliance follows similar structure with Level 1 (essential) and Level 2 (defense-in-depth) profiles.

Key Takeaways

  • Configuration Drifts (VCF 9.0): configuration templates define the desired state of vCenter instances and vSphere clusters; assigned through a VCF Operations policy; applies to 9.0+ objects only.
  • Detect Drift compares Current vCenter Value with Config Template Value per template; results on the Overview charts and in the Drift Status Report (7-30 days); detection can be scheduled in Automation Central.
  • Remediation is manual in vCenter (or the template is edited to adopt the new value); legitimate differences are handled with separate templates/policies, not exclusion tags.
  • STIG/CIS compliance remains a host-level control set (CAT I/II/III): SSH disabled, lockdown mode, TLS profile (COMPATIBLE/NIST_2024 on ESX 9.0), encrypted vMotion, persistent syslog — checked with esxcli or vSphere Configuration Profiles.

Advanced Troubleshooting: SoS, Log Bundles & API Diagnostics#

VCAP-level troubleshooting requires mastery of VCF's built-in diagnostic tools: the SoS (Supportability and Serviceability) utility for health checks, log bundle generation for support cases, REST API diagnostics for programmatic analysis, and component-specific troubleshooting procedures.

SoS Utility (Supportability and Serviceability; VCF 9.0.1 pp.1297-1298):


Location: SDDC Manager VM → /opt/vmware/sddc-support/sos
Purpose: Comprehensive health check and log collection for all VCF components

Key SoS Commands (run as root on the SDDC Manager appliance; VCF 9.0.1 pp.1298-1300):

Health Check (non-disruptive diagnostic):

sudo /opt/vmware/sddc-support/sos --health-check [--domain-name <name>|ALL] [--short] [--force]

--health-check runs every check; individual checks: --general-health, --connectivity-health, --compute-health, --storage-health, --ntp-health, --certificate-health, --password-health (5.2 table; lists VCF-managed passwords with last-change and expiry dates)

Output: GREEN (normal) / YELLOW (warning) / RED (critical) per check, written under /var/log/vmware/vcf/sddc-support unless --log-dir is given; --json-output-dir writes the results as JSON; --short lists only failures and warnings (VCF 5.2 docs pp.608-609)

Log Bundle Collection:

sudo /opt/vmware/sddc-support/sos --collect-all-logs [--domain-name <name>] [--log-dir <dir>] [--zip]

Per-component selectors instead of everything: --vc-logs, --nsx-logs, --esx-logs, --sddc-manager-logs, --operations-logs, --operations-fleet-logs, --automation-logs, --api-logs, --vm-screenshots, --vcf-installer-logs

Other options: --no-health-check (skip the health check run as part of collection), --no-clean-old-logs, --log-folder <name>, --history (last twenty SoS operations)

There is no date-range option: bundle size is controlled by the component flags and --log-dir

Output: compressed bundle for the Broadcom support case

Log Architecture & Key Log Locations:


SDDC Manager:

  • /var/log/vmware/vcf/sddc-manager-ui-app/: UI application logs
  • /var/log/vmware/vcf/commonsvcs/: Common services (LCM, inventory)
  • /var/log/vmware/vcf/domainmanager/: Domain lifecycle operations
  • /var/log/vmware/vcf/operationsmanager/: Operations and health
  • vCenter:
  • /var/log/vmware/vpxd/vpxd.log: Core vCenter service
  • /var/log/vmware/vsan-health/: vSAN health service
  • /var/log/vmware/sso/: SSO authentication
  • /var/log/vmware/content-library/: Content library operations

ESXi:

  • /var/log/vmkernel.log: Kernel messages (PSOD, hardware, storage)
  • /var/log/hostd.log: Host daemon (VM operations, API)
  • /var/log/vobd.log: VMware Observation Broker (events)
  • /var/log/esxupdate.log: Update/patch operations
  • /var/log/fdm.log: HA fault domain manager

NSX Manager:

  • /var/log/proton/nsxapi.log: API requests
  • /var/log/corfu/corfu.log: Distributed database (CCP)
  • /var/log/policy/policy.log: Policy engine operations

REST API Diagnostics:


VCF SDDC Manager API:

Base: https://<sddc-mgr>/v1/
Authentication: Bearer token (POST /v1/tokens with credentials)

Key diagnostic endpoints:

  • GET /v1/hosts: All commissioned hosts with status
  • GET /v1/domains: Domain inventory and health
  • GET /v1/tasks: Lifecycle operation history and status
  • GET /v1/credentials: Password inventory and expiry dates
  • GET /v1/bundles: Available update bundles

Troubleshooting workflow:

  1. Note the failed task ID (SDDC Manager UI task panel)
  2. GET /v1/tasks/{id} → task detail (VCF 9.0.1 docs p.2845)
  3. Read the failed subtask's error (also shown in the task panel, p.1454)
  4. Correlate subtask errors with component logs for root cause

Common Troubleshooting Scenarios:


  1. Host commission fails:
  • Check: SoS connectivity check → network path to host
  • Check: DNS resolution (forward and reverse) for host FQDN
  • Check: Host firmware/driver versions against VCF HCL
  • Logs: /var/log/vmware/vcf/domainmanager/domainmanager.log
  1. Workload domain creation stalls:
  • Check: the task's subtasks (task panel or GET /v1/tasks/{id}) → identify stuck subtask
  • Common causes: vSAN disk claim failure, NSX TEP connectivity, vCenter deployment timeout
  • Logs: domainmanager.log + component-specific logs based on stuck subtask
  1. LCM upgrade fails midway:

Check: SDDC Manager UI → In-Progress Updates → View Status; a failed component shows Failed with the reason and remediation steps (VCF 9.0.1 docs p.1456)

Action: Do NOT retry without understanding the failure

Logs: /var/log/vmware/vcf/lcm/ → lcm-debug log (after the upgrade to 9.0 all LCM logging goes to lcm-debug; p.1456)

Recovery: resolve the root cause, then retry the update / failed task in the SDDC Manager UI (VCF 9.0.1 docs p.1456, p.1388); for an API-driven retry take the call from the SDDC Manager API reference for your release (not documented in the VCF 9.0.1 docs)

  1. Certificate expiry warnings:

Check: SoS health-check → certificate section

Action: renew or replace the certificate in VCF Operations > Fleet Management > Certificates (or activate auto-renewal, 60 days before expiry); the VCF documentation directs certificate management for all VCF components to the VCF Operations console, not the vCenter certificate-manager CLI

Prevention: alert well ahead of expiry — this design chooses a 60-day threshold (a design value, not a product default) so rotation is scheduled before the SoS YELLOW band (15-30 days)

Key Takeaways

  • SoS utility: --health-check (GREEN/YELLOW/RED per check; --domain-name <name>|ALL, --short), component log collection with --vc-logs/--nsx-logs/--esx-logs/--sddc-manager-logs or --collect-all-logs (--log-dir, --zip; no date filter), --connectivity-health and --password-health as individual checks. Run the health check first in any troubleshooting scenario.
  • Key log paths: SDDC Manager (/var/log/vmware/vcf/), vCenter (/var/log/vmware/vpxd/), ESXi (/var/log/vmkernel.log, hostd.log), NSX (/var/log/proton/, /var/log/corfu/).
  • REST API diagnostics: failed task ID (task panel) → GET /v1/tasks/{id} (VCF 9.0.1 docs p.2845) → read the failed subtask's error → correlate it with component logs. This is the primary troubleshooting workflow for failed lifecycle operations.
  • Recovery pattern: identify failed subtask → check component logs → resolve root cause → retry the failed task or update (SDDC Manager UI, p.1456; API call per the SDDC Manager API reference). Never blindly retry without understanding the failure.

vSAN ESA Operations & Day-2 Management#

vSAN Express Storage Architecture (ESA) is the default storage architecture in VCF 9.0, replacing the original storage architecture (OSA). VCAP Admins must understand ESA's operational model, disk lifecycle, maintenance procedures, and fault domain management.

vSAN ESA vs OSA — Operational Differences:


Architecture Comparison:

OSA (Original Storage Architecture):

  • Two-tier: Cache (SSD) + Capacity (SSD or HDD)
  • Disk Groups: 1 cache + 1-7 capacity disks per group
  • RAID-1 mirroring for data protection (FTT=1 uses 2x storage)
  • Maximum 5 disk groups per host

ESA (Express Storage Architecture):

  • Single-tier: All NVMe drives in a flat pool (no cache/capacity distinction)
  • No disk groups: All disks contribute to a single storage pool per host
  • RAID-5/6 erasure coding for data protection (OSA RAID-5 FTT=1 = 1.33x; ESA adaptive RAID-5 FTT=1 = 1.25x at 6+ hosts or 1.5x below; RAID-6 FTT=2 = 1.5x)
  • Minimum 128 GB RAM per host
  • Significant storage efficiency improvement (RAID-5 vs RAID-1)

ESA Host Requirements (VCF 9.0):

  • CPU: 2 sockets minimum, 32+ cores recommended
  • Memory: 128 GB minimum (ESA metadata caching is memory-intensive)
  • Storage: 2+ NVMe drives per host (no SATA/SAS SSD)
  • Network: 25 GbE minimum for vSAN traffic (10 GbE insufficient for ESA performance)

vSAN ESA Day-2 Operations:


Disk Lifecycle:

Add Disk:

  1. Insert new NVMe drive into host
  2. ESXi detects drive automatically
  3. vSAN claims disk into the storage pool (no disk group management needed)
  4. Data automatically rebalances across all disks (background operation)

Remove/Replace Disk:

  1. Mark disk for decommission in vSAN UI
  2. vSAN evacuates data from the disk (rebuilds on remaining disks)
  3. Wait for evacuation to complete (monitor vSAN health)
  4. Physically remove disk
  5. Insert replacement if needed → auto-claimed

Disk Failure Handling:

  • vSAN detects failure immediately via NVMe health monitoring
  • Failed disk marked as 'Degraded' or 'Absent'
  • vSAN begins automatic rebuild on remaining disks
  • Rebuild time depends on data volume and cluster load
  • FTT policy determines if data is still accessible during rebuild

Host Maintenance Mode in vSAN ESA:


Three evacuation modes (same as OSA, but ESA behavior differs):

  1. Full Data Migration:
  • All data on the host's disks is rebuilt on other hosts
  • Safest option: cluster can tolerate additional failures during maintenance
  • Longest duration: proportional to data on the host (can take hours for TB-scale)
  • Use for: extended maintenance (hardware replacement, firmware update)
  1. Ensure Accessibility:
  • Only data components that would become inaccessible are migrated
  • Faster than full migration
  • Cluster resilience is reduced during maintenance (cannot tolerate additional failure)
  • Use for: short maintenance (reboot, quick patch)
  1. No Data Migration:
  • No data is moved (immediate entry to maintenance mode)
  • Risk: if another host fails, data may be inaccessible
  • Use for: emergency maintenance only or when cluster has sufficient redundancy (FTT=2)

Fault Domain Management:


Fault domains define failure boundaries for data placement:

  • Without fault domains: vSAN distributes replicas across hosts (host = failure unit)
  • With fault domains: vSAN distributes replicas across domains (rack/enclosure = failure unit)

Configuration:

vSphere Client → Cluster → Configure → vSAN → Fault Domains → Add Domain
Assign hosts to domains based on physical topology (rack, power circuit, network switch)

Rules:

  • Minimum 3 fault domains for FTT=1 (data on 2 domains, witness on 3rd)
  • Data replicas/stripes are placed in DIFFERENT fault domains
  • If a domain has fewer hosts, it limits the fault domain's capacity contribution

Stretched Cluster Configuration:

  • Two data sites + 1 witness site
  • Latency requirements: < 5ms RTT between data sites (no geographic distance restriction)
  • Site affinity: VMs prefer local storage reads but replicate writes to remote site

Storage Policy Management:


vSAN storage policies define data protection and performance characteristics:

Key policy parameters:

  • FTT (Failures to Tolerate): 0-3 (ESA supports higher efficiency via RAID-5/6)
  • RAID: RAID-1 (mirroring), RAID-5/6 (erasure coding — ESA preferred)
  • Object Space Reservation: Thin (0%) or Thick (100%)
  • IOPS Limit: Per-object IOPS cap for noisy neighbor prevention

ESA-specific policies:

Compression: Always on (ESA compression is near-zero CPU overhead)

Deduplication: not available on a vSAN ESA cluster in the vSAN chapter of the 9.0.1 documentation (p.1775; ESA/OSA feature table p.1651) — compression is the ESA space-efficiency feature, enabled through the storage policy. The 9.0.1 text also lists a VCF Operations property ESAGlobalDedupRatio (p.118), which points to an opt-in global deduplication in vSAN 9.0; treat it as opt-in and verify in the vSAN 9.0 release notes before citing it

Encryption: At-rest encryption enabled per cluster; in ESA all disks in the cluster use one Data Encryption Key (DEK), protected by keys from the key provider (Native Key Provider or external KMS) (VCF 9.0.1 p.1779)

Key Takeaways

  • ESA vs OSA: Single-tier NVMe pool (no disk groups), RAID-5/6 erasure coding (ESA adaptive RAID-5: 1.25x at 6+ hosts, 1.5x below; RAID-6 4+2 = 1.5x - versus 2x for RAID-1 mirroring), 128 GB RAM minimum for the smallest ReadyNode, 10 GbE minimum with 25 GbE recommended (VCF 9.0.1 p.1664). ESA is the VCF 9.0 default.
  • Disk lifecycle: ESA disks are auto-claimed into a flat pool. Add = auto-claim + rebalance. Remove = evacuate data + physical removal. Failure = auto-detect + auto-rebuild.
  • Maintenance mode: Full Migration (safest, longest), Ensure Accessibility (faster, reduced resilience), No Migration (emergency only). Choice depends on maintenance duration and cluster FTT level.
  • Fault domains: map to physical topology (rack, power, switch). Minimum 3 domains for FTT=1. Stretched cluster: 2 data sites + witness, < 5ms RTT requirement.

VCF Security Hardening & Audit#

VCF security hardening covers ESXi lockdown mode, TPM attestation, STIG compliance, audit logging, and secure lifecycle operations. VCAP Admins must implement defense-in-depth security while maintaining operational access for management tools.

ESXi Lockdown Mode:


Lockdown mode restricts direct access to ESXi hosts, forcing all management through vCenter:

Modes:

Normal Lockdown: DCUI access allowed, direct SSH/API blocked

  • Only vCenter can manage the host
  • DCUI available for emergency access (root login at console)
  • Exception users: configured in vCenter for specific access needs

Strict Lockdown: DCUI access also blocked

  • Only vCenter management path available
  • No local console access (even DCUI)
  • WARNING: If vCenter is unavailable, the host is unmanageable
  • Use with caution — requires reliable vCenter HA

Exception Users:

  • Added via vCenter → Host → Configure → Security Profile → Lockdown Mode
  • Purpose: Allow specific service accounts (backup agents, monitoring) direct access
  • Best practice: Minimize exception users; route through vCenter API where possible

VCF Impact:

VCF requires certain service accounts to have direct ESXi access

SDDC Manager/VCF Operations use exception user accounts for lifecycle operations
Do NOT enable strict lockdown on management domain hosts without testing VCF operations

TPM Attestation:


Trusted Platform Module (TPM) 2.0 enables host attestation:

  • Verifies ESXi boot integrity (Secure Boot + TPM)
  • vCenter displays attestation status per host
  • VCF can enforce TPM requirement during host commission

Key Manager Integration:

vSAN encryption and VM encryption require key management:

Native Key Provider: vCenter built-in, no external key server; hosts receive a Key Derivation Key that persists across reboots by default. A TPM 2.0 is optional (enhanced security; the provider can be limited to TPM 2.0 hosts). Back up the provider

External KMS: KMIP-compliant server (Thales, Hytrust, etc.)

VCF integrates KMS at the Instance level — all domains share the KMS

Audit Logging Architecture:


Comprehensive audit trail across VCF components:

ESX: audit records — esxcli system auditrecords local enable stores them (default directory /scratch/auditLog); esxcli system auditrecords remote enable sends them to the Syslog.global.logHost targets
vCenter: Events database — all configuration changes, logins, permissions

  • Export via syslog: vCenter Appliance Management Interface (https://<vcenter>:5480) → Syslog → Forwarding Configuration (up to three remote hosts; TLS, TCP, RELP or UDP)

NSX: Audit logs via API or syslog — all policy changes, login events
SDDC Manager: /var/log/vmware/vcf/sddc-manager-audit.log — lifecycle operations

Centralized Logging Best Practice:

  1. Configure all components to forward syslog to centralized collector

(VCF Operations for Logs / Aria Operations for Logs / Splunk)

  1. Set syslog protocol to TCP+TLS (not UDP — UDP drops logs under load)
  2. Retain audit logs per the customer's compliance policy (for example PCI DSS v4.0 requirement 10.5.1: at least 12 months, with the most recent three months immediately available for analysis); HIPAA's 6-year rule (45 CFR 164.316(b)(2)(i)) covers Security Rule documentation, not a log period
  3. Configure alerts on critical audit events:
  • Root login to ESXi
  • Permission changes on vCenter
  • DFW policy changes in NSX
  • VCF credential rotation events

Security Hardening Checklist (VCF 9.0):


  1. ESXi Hosts:
  • [ ] Lockdown mode enabled (Normal minimum)
  • [ ] SSH disabled (enable only for troubleshooting, with timeout)
  • [ ] NTP configured (time sync critical for certificate validation and log correlation)
  • [ ] Syslog forwarding to centralized collector
  • [ ] TLS 1.2 minimum for all management interfaces
  • [ ] vMotion encryption enabled
  • [ ] vSAN encryption enabled (at-rest + in-transit)
  • [ ] Firewall rules restricted to VCF-required ports only
  1. vCenter:
  • [ ] AD/LDAP integration for human admin accounts
  • [ ] MFA enabled via OIDC federation (VCF 9.0)
  • [ ] Password policy: 15+ characters, complexity, 90-day rotation
  • [ ] Least-privilege roles (no blanket Administrator access)
  • [ ] Session timeout: 30 minutes (configurable)
  • [ ] Audit logging to remote syslog
  1. NSX:
  • [ ] Management plane certificates: CA-signed (not self-signed)
  • [ ] API rate limiting enabled
  • [ ] Audit logging for all policy changes
  • [ ] DFW default deny for east-west traffic
  1. VCF:
  • [ ] Credential auto-rotation enabled (90-day cycle)
  • [ ] LCM depot secured (HTTPS only, authenticated access)
  • [ ] Break-glass procedure documented and tested quarterly

Key Takeaways

  • Lockdown mode: Normal (DCUI allowed, SSH/API blocked) vs Strict (all local access blocked). VCF requires exception users for lifecycle operations — never enable Strict without testing.
  • TPM 2.0 attestation verifies host boot integrity. Native Key Provider needs no external key server and no TPM (a TPM 2.0 is optional and adds security); back it up. External KMS (KMIP) required for enterprise key management with vSAN/VM encryption.
  • Audit logging: ESX (audit records, default /scratch/auditLog, remote via Syslog.global.logHost), vCenter (events DB + syslog forwarding in the VAMI), NSX (API/syslog), SDDC Manager (audit.log). Centralize via TCP+TLS syslog to a SIEM with compliance-aligned retention.
  • Hardening checklist: lockdown mode, SSH disabled, TLS 1.2, encrypted vMotion/vSAN, AD/LDAP + MFA, least-privilege roles, credential auto-rotation, break-glass procedure documented.

Exam Mapping: 3V0-11.26 — VCAP — VMware Cloud Foundation Administrator

  • Section 1: Fleet Lifecycle Management — bring-up, expand, shrink, upgrade, decommission
  • Section 2: Identity, Certificate & Password Management — SSO, AD/LDAP, OIDC, VMCA, credential rotation
  • Section 3: Configuration Drift & Compliance — baselines, drift detection, STIG/CIS, remediation
  • Section 4: Advanced Troubleshooting — SoS, log bundles, API diagnostics, component health
  • Section 5: vSAN ESA Operations — disk lifecycle, maintenance mode, fault domains, storage policies
  • Section 6: Security Hardening — lockdown mode, TPM, audit logging, encryption

Labs in This Section

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.