Password & Secrets Management via VCF 9 Password Manager
Objectives
- Understand VCF 9 credential architecture: where passwords are stored (NSX Manager, SDDC Manager, vCenter SSO), rotation policies, and audit trails
- Configure and execute automated password rotation for common service accounts (vCenter SSO admin, ESXi root, NSX admin)
- Implement manual rotation for break-glass accounts and validate recovery scenarios
- Integrate VCF Password Manager with external vault systems (HashiCorp Vault, Azure Key Vault) for enterprise compliance
- Interpret password rotation failure scenarios and audit compliance gaps using SDDC Manager logs
Prerequisites
VCF 9.0.x management domain deployed and operational. SDDC Manager accessible. All four management domain ESXi hosts commissioned and showing Active status. vCenter and NSX Manager fully initialized. Lab account must have SDDC Manager Administrator role.
Prior labs: holodeck-02
Required skills:
- SDDC Manager navigation (Inventory, Security, Operations sections)
- Understanding of VCF credential hierarchy (root accounts, service accounts, SSO passwords)
- Basic understanding of PKI and certificate rotation (optional but helpful)
- Command-line tools: PowerShell/PowerCLI for vCenter, NSX API access via curl or Postman
- Familiarity with audit logging concepts and compliance requirements
Lab Environment
VCF 9.0.x management domain with 4x ESXi hosts, SDDC Manager, vCenter Server, NSX Manager. All components deployed and healthy from a prior lab (holodeck-02 or equivalent). No workload domains required for this lab.
IP Addressing
| Network | Purpose | VLAN |
|---|---|---|
Management (SDDC Manager, vCenter, NSX) | 1644 |
Credentials
| System | Username | Password |
|---|---|---|
| SDDC Manager | administrator@vsphere.local | Set during VCF bring-up |
| vCenter Server | administrator@vsphere.local | Synced with SDDC Manager SSO password |
| NSX Manager | admin | Set during VCF bring-up; managed by SDDC Manager |
| ESXi Hosts (root) | root | Set during host commissioning; rotatable via SDDC Manager |
Tasks
Task 1 Audit current credential inventory and rotation policies
manageabilityMap the complete credential landscape in VCF — which passwords are managed by SDDC Manager, which are external, and which cannot be auto-rotated. This is a VCDX-critical foundational step: before you can architect a secure credential model, you must understand the current state.
Open SDDC Manager UI: https://10.0.0.4. Navigate to Administration > Security > Credentials.
Expand the ESXi Hosts section. Verify that all 4 hosts show a root credential entry. Click on one host entry to view: credential username, last rotation date, next rotation date (if automated), and rotation policy (manual vs automated).
Expand the vCenter section. Verify the vCenter SSO admin credential (administrator@vsphere.local). Note: this password is also the SDDC Manager administrator password (they are synced). Verify the rotation policy and compare with ESXi policy — they may differ.
Expand the NSX Manager section. Verify the NSX admin credential (admin@nsx-manager.local or equivalent). Check if the rotation is managed by SDDC Manager (auto) or if it requires manual intervention.
Scroll down to view Service Account Credentials (e.g., service accounts for vRealize, vSAN, vMotion, etc.). Document which service accounts are credential-managed vs hardcoded.
Export the credential inventory to a CSV file for compliance auditing. In SDDC Manager, click the export icon (if available) or manually document: Component, Account Username, Rotation Policy, Last Rotated Date, Next Rotation Date. Save to your lab notes.
Validation Gate
Check: SDDC Manager Credentials page: all 6+ credentials visible and documented (4x ESXi root, vCenter SSO, NSX admin, service accounts). Export or manual inventory complete with rotation dates.
Expected: Complete credential map with no missing entries. Rotation ages documented.
Common Errors
Task 2 Configure automated password rotation for service accounts
securityEstablish a baseline rotation schedule that balances security (frequent rotation) with operational stability (not too disruptive). This mirrors production change management: you'll learn how SDDC Manager coordinates rotation across dependent services.
In SDDC Manager, navigate to Administration > Security > Password Policies. Review the default rotation schedule.
Select the ESXi root credential policy. Modify the rotation interval from 90 days to 30 days (simulating a more aggressive production policy). Click 'Save Policy'.
Verify the vCenter SSO password policy is set to 'Manual only' (not auto-rotated). Confirm this is intentional: the vCenter SSO password is complex to rotate because it affects multiple services (SDDC Manager, NSX Manager integration, vRealize, etc.). Click 'Edit Policy' and note the reason field: 'Multi-service dependency — requires coordinated rotation'.
Set the NSX admin password rotation to 45 days (from the default 60 days). Save the policy.
Navigate to Administration > Security > Scheduled Tasks. Verify a 'Password Rotation' task exists and is scheduled to run at a safe time (typically off-hours, e.g., 2:00 AM). Check that it's configured to send notifications on success/failure.
For each credential you modified (ESXi root, NSX admin), click 'Preview Next Rotation' to view which accounts will be affected and in what order. This is the rotation sequence — SDDC Manager rotates them serially to avoid cascading failures.
Validation Gate
Check: Password Policies: ESXi root = 30 days, vCenter SSO = Manual, NSX admin = 45 days. Scheduled rotation task visible and configured with notifications.
Expected: All policies configured and documented. Rotation order confirmed (serial, no parallels).
Common Errors
Task 3 Execute a manual password rotation and validate the rotation sequence
availabilityUnderstand the mechanics of rotation: the order (why hosts before NSX), the validation checks (ensuring service availability post-rotation), and the recovery path if rotation fails mid-sequence. This is hands-on expertise VCDX panelists will probe.
In SDDC Manager, navigate to Administration > Security > Credentials. Select one ESXi host (e.g., esxi-01). Click 'Rotate Now' to manually trigger an immediate rotation (not waiting for the scheduled task).
Click 'Confirm' to start the rotation. A progress indicator appears. Watch the status update: 'Generating new password... Applying to esxi-01... Verifying SSH connectivity... Rotation complete.'
Verify connectivity to the rotated ESXi host. In SDDC Manager, navigate to Inventory > Hosts. Click on esxi-01. Verify Status = 'Active' with no new warnings or errors. If the host shows any connectivity issues, rotation may have failed mid-way.
Now manually rotate the vCenter SSO admin password (the only manual credential). This requires more care because vCenter, SDDC Manager, and NSX Manager all depend on this password being synchronized. Click Credentials > vCenter > administrator@vsphere.local > 'Rotate Now'.
Before proceeding with vCenter SSO rotation, verify all three services are healthy: SDDC Manager dashboard (already open), vCenter: navigate to https://10.0.0.6 and confirm login works, NSX Manager: navigate to https://10.0.0.10 and confirm login works. Document timestamps.
Proceed with vCenter SSO rotation. Click 'Confirm'. Monitor the progress. SDDC Manager will: (1) Generate a new 32-character password, (2) Update vCenter SSO, (3) Update SDDC Manager local admin, (4) Perform a test login to each service to confirm the new password works.
Validation Gate
Check: SDDC Manager Credentials: ESXi-01 shows new rotation timestamp. vCenter SSO shows new rotation timestamp. Inventory > Hosts: ESXi-01 status Active. Inventory > Workload Domains: vCenter connectivity shows Active.
Expected: Two rotations (ESXi and vCenter) completed successfully with full connectivity validation.
Common Errors
Task 4 Simulate a rotation failure and execute break-glass account recovery
recoverabilityPrepare for the failure scenario you'll face in production. When automated rotation fails (account locks, service unavailable, network timeout), you need a documented recovery path. This task builds operational resilience.
Intentionally lock an ESXi root account to simulate a failed rotation. SSH into ESXi-02 (use SDDC Manager credentials from Task 3). Run: 'passwd -l root' to lock the account.
Now attempt to rotate the locked ESXi-02 root account via SDDC Manager. Navigate to Administration > Security > Credentials > ESXi-02 > 'Rotate Now'. Watch the operation fail.
Check SDDC Manager's failure notification. Navigate to Administration > Recent Tasks. Find the failed rotation task. Click it to view the detailed error log. Document: timestamp, error message, affected system, and the recommended recovery action.
Initiate break-glass recovery. In SDDC Manager, navigate to Administration > Security > Break Glass. Select ESXi-02. Click 'Reset Credentials' (if available) or document the alternative: you will need to physically access the ESXi console or use a recovery password.
In vCenter, navigate to Hosts and Clusters. Right-click ESXi-02 > Enter Maintenance Mode (if VMs are running, vCenter will vMotion them to other hosts first). Once in maintenance mode, the host is safely isolated.
Now reset the ESXi root password manually. In the Holodeck environment (nested), this requires SSH access via the HoloRouter or using DCUI. For this lab, document the recovery step: 'In production, call data center staff to reset root password at the console (DCUI Alt-F2, then set password).' Alternatively, if you have DCUI/console access to the nested ESXi VM (via vCenter console), open it: vCenter > Hosts and Clusters > ESXi-02 > Click Console. Log in as root with the old password if it's not locked, or use DCUI (Ctrl-Alt-F2) to reset.
Validation Gate
Check: ESXi-02 is in Maintenance Mode. Failed rotation task is documented with error details. Recovery plan is articulated (manual password reset via console or out-of-band access).
Expected: Break-glass recovery procedure is documented and tested. Understanding of rotation failure modes is demonstrated.
Common Errors
Task 5 Integrate external vault (HashiCorp Vault) and audit credential access
securityModern enterprise compliance (SOC 2, ISO 27001) requires centralized secret management with audit trails. VCDX-level design means understanding how VCF credentials can be offloaded to external vaults while maintaining operational capability. This task demonstrates that integration model.
In SDDC Manager, navigate to Administration > System Configuration > External Vault Integration. Review whether Vault integration is available in your VCF 9.0 version. (Note: This feature may be limited in some lab editions. If not available, document the expected behavior and skip to Step 3.)
If Vault integration is available, review the configuration steps (don't fully complete unless you have a live Vault instance available). Document what would be required: (a) Vault server IP/FQDN, (b) authentication method (AppRole, JWT, or TLS certificate), (c) KV v2 mount path for VCF secrets (e.g., /secret/vcf/), (d) network connectivity from SDDC Manager to Vault. Test connectivity: SDDC Manager > System Configuration > External Vault > Test Connection.
Even if Vault integration is not deployed in your lab, document the architecture: where credentials would be stored (Vault backend), how SDDC Manager would retrieve them (API call at rotation time), and the audit trail (Vault audit log shows every secret access). Create a design diagram in your lab notes: SDDC Manager <--> Vault <--> Audit Log.
Review SDDC Manager's local credential audit log. Navigate to Administration > Audit Logs. Filter for 'Credential' operations. You should see entries for: password rotations, credential reads (e.g., when SDDC Manager uses a password to authenticate to NSX), and rotation failures.
Export the credential audit log. In SDDC Manager, Audit Logs > Filter for past 7 days > Export CSV. Document the compliance findings: (a) How many times were credentials accessed? (b) Which accounts were rotated? (c) Were there any unauthorized access attempts? (d) Are there any gaps (e.g., accounts that were never rotated)?
Write a brief compliance summary (1 paragraph) for a hypothetical security audit: 'Based on the credential audit trail from April 10-17, 2026: All managed accounts (4x ESXi root, vCenter SSO, NSX admin) were rotated successfully. No unauthorized access attempts detected. Service account credentials are scheduled for rotation weekly. Recommendation: Investigate any manual credential access outside of SDDC Manager automation.' Document any gaps you notice.
Validation Gate
Check: External Vault integration reviewed (or documented as unavailable). Credential audit log exported for past 7 days. Compliance summary written with findings.
Expected: Understanding of enterprise credential architecture (local SDDC Manager + external Vault option) and audit trail capabilities demonstrated.
Common Errors
Final Validation
You have demonstrated comprehensive credential lifecycle management in VCF 9.0: audited all managed accounts, configured rotation policies, executed manual rotations, recovered from failure scenarios, and reviewed enterprise vault integration. You understand the security, availability, and compliance implications of credential management at the VCDX architect level.
✓ Credential Inventory → All 6+ accounts (4x ESXi root, vCenter SSO, NSX admin, service accounts) documented with rotation history
✓ Rotation Policies → ESXi = 30 days, vCenter SSO = Manual, NSX = 45 days. Policies saved and scheduled task configured.
✓ Manual Rotation → Two successful rotations (ESXi-01 and vCenter SSO) with validation checks passed. New passwords functional.
✓ Break-Glass Recovery → Failure scenario simulated. Recovery procedure (console/DCUI reset) documented and practiced.
✓ Audit and Compliance → Credential audit log exported. 7-day compliance summary written with findings and recommendations.
Cleanup / Restore
Snapshot: vcp-admin-07-complete
• Exit ESXi-02 from Maintenance Mode: vCenter > Hosts and Clusters > ESXi-02 > Exit Maintenance Mode
• Document all rotated passwords in your secure password manager (or lab notebook for this lab environment)
• Revert any intentionally-modified policies (e.g., ESXi rotation back to 90 days if desired) to match your lab's baseline
• Take a post-lab snapshot for future reference: 'vcp-admin-07-complete — All rotations executed, audit trails captured'
Design Reflection (VCDX)
A VCDX panelist examining your credential management strategy would ask: How do you balance rotation frequency (security) with operational risk (failed rotations can break services)? Which accounts can you safely auto-rotate, and which require manual coordination? How do you integrate external vaults to comply with SOC 2 / ISO 27001 audit requirements? Why is vCenter SSO password rotation complex? What is the blast radius of a compromised ESXi root password vs a compromised vCenter SSO password? Be prepared to defend your rotation schedule with risk/benefit analysis, not just follow vendor defaults.
Requirements
- All VCF-managed credentials must be rotatable (manual or automated) at least quarterly
- Rotation operations must not disrupt service availability to management plane or workloads
- Audit trail must be maintained for all credential access and rotations for SOC 2 / ISO 27001 compliance
- Break-glass recovery procedure must exist for accounts that cannot be auto-rotated due to service dependencies
Constraints
- vCenter SSO password is a multi-service dependency — changing it requires coordinated updates to SDDC Manager, NSX, vRealize, and other integrations
- ESXi root password rotation requires network access (SSH) to each host — if a host is down, its password cannot be rotated until it comes back online
- Automated rotation cannot proceed if any dependency service (vCenter, NSX, SDDC Manager) is offline — introduces operational coupling
- Some legacy service accounts may have hardcoded passwords that cannot be auto-rotated — manual rotation required
Assumptions
- SDDC Manager has network connectivity to all managed systems (ESXi hosts, vCenter, NSX) for rotation automation
- All managed systems accept credential updates via secure channels (SSH for ESXi, HTTPS API for vCenter/NSX)
- Operator has administrative credentials to SDDC Manager and understands the rotation sequence and recovery procedures
- External vaults (Hashicorp Vault, Azure Key Vault) are optional; local SDDC Manager credential storage is sufficient for most deployments
Risks
- Failed rotation mid-sequence leaves services in inconsistent state (one has old password, one has new) — IMPACT: authentication failures, MITIGATION: implement rotation validation checks and rollback procedures
- Rotation policy too aggressive (e.g., daily rotation) increases chance of automation failure and service disruption — IMPACT: operational noise and false alarms, MITIGATION: use 30-90 day cycle with testing
- Break-glass account unused for 12+ months becomes stale and unusable when needed in emergency — IMPACT: cannot access system in incident, MITIGATION: test break-glass quarterly
- Compromised SDDC Manager means attacker has access to all stored credentials (ESXi, vCenter, NSX) — IMPACT: lateral movement to entire infrastructure, MITIGATION: use external vault as second line of defense; isolate SDDC Manager network access
Self-Assessment Discussion Prompts
- Why is vCenter SSO password rotation set to 'Manual only' in the default configuration? What would need to change to make it safe to auto-rotate?
- You discover that automated password rotation failed for ESXi-02 root account. What is the first diagnostic step? (Answer: Check if ESXi-02 is reachable from SDDC Manager via SSH. If unreachable, the host may be offline or network partitioned.)
- Compare rotation policies for ESXi root (physical host) vs vCenter SSO (shared service account). Which has a tighter tolerance for rotation failures? Why?
- A security audit requires you to rotate all passwords to 15-day cycles (from 30-day). What are the operational risks? What changes would you recommend to make 15-day rotation safe?
- External vault integration is expensive and complex. Under what circumstances is it worth the investment? (Answer: When you have multiple VCF instances, need to comply with SOC 2, or have a centralized security team managing secrets across multiple products.)
- Design a credential rotation schedule for a production VCF environment with 3 workload domains. Which credentials should be rotated together? Which should be independent?
Extensions
Design and Test a Full Credential Rotation Window
Simulate a production maintenance window where you rotate all credentials (ESXi, vCenter, NSX, service accounts) in sequence. Measure the total time, identify dependencies, and document the rollback procedure if any step fails. Create a runbook for the operations team.
harderIntegrate HashiCorp Vault and Migrate All Credentials
Deploy a HashiCorp Vault instance in your lab (or use Vault Dev mode). Configure SDDC Manager to store all credentials in Vault instead of locally. Verify that rotations are now retrieved from Vault, and audit logs show Vault API calls. This is the enterprise-grade architecture.
harderCredential Compliance Audit: SOC 2 Readiness
Using the credential audit logs from this lab, prepare a SOC 2 Type II report section: 'Credential Management Controls'. Document: (a) Who can access credentials (RBAC), (b) Rotation frequency and method, (c) Audit trail retention, (d) Break-glass procedures, (e) Encryption in transit and at rest. Compare your implementation to SOC 2 CC-6 and CC-7 controls.
harderVCF 5.2 vs VCF 9.0 Credential Architecture Comparison
If you have access to a VCF 5.2 lab, repeat the credential audit (Task 1) and rotation (Task 3) in VCF 5.2. Document differences: UI differences, rotation automation maturity, external vault support, audit trail capabilities. Write a 2-page comparison addressing: which version has better credential lifecycle automation, and what was added in VCF 9.0.
sameReferences
- VMware Cloud Foundation 9.0 Security Guide — Password & Secrets ManagementTier 1 — Official
Official Broadcom documentation covering credential architecture, rotation policies, and external vault integration - SDDC Manager Operations and Day 2 Management — Password Manager ConfigurationTier 1 — Official
Detailed walkthrough of SDDC Manager credential UI, rotation scheduling, and failure recovery - HashiCorp Vault — Enterprise Secret ManagementTier 2 — VMware Press
Reference architecture for external secret storage integration with VCF (if you want to design that extension) - William Lam — VCF Password Manager Deep DiveTier 3 — Expert Blog
Community expert blog with hands-on examples of credential rotation automation and failure scenarios - SOC 2 Type II Compliance — Credential & Access Management (CC-6, CC-7)Tier 2 — VMware Press
AICPA guidance on credential control requirements for SOC 2 audits (useful for designing audit scope)