Password Management in VCF 9.0 (VCF Operations and SDDC Manager)
Objectives
- Understand the VCF 9.0 credential architecture: which local accounts VCF Operations updates and remediates (Table 719 defaults and expiry), what SDDC Manager rotates, and what stays manual
- Execute Update and Remediate from VCF Operations (Fleet Management > Passwords) and Rotate / scheduled rotation from SDDC Manager (Security > Password management)
- Recover from a rotation or synchronisation failure using Remediate and the documented service-account remediation procedures
- Design an external secrets-manager integration around the SDDC Manager credentials API (and the VCF 9.1 VCF Operations password APIs), since 9.0 has no built-in vault integration
- Audit password state with password alerts, SDDC Manager tasks and the credentials API, and identify compliance gaps
Prerequisites
VCF 9.0.x management domain deployed and operational. VCF Operations (fleet management) and SDDC Manager accessible. All four management domain ESX hosts commissioned and Active. vCenter and NSX Manager fully initialised. Lab account must hold the VCF Operations permissions Fleet Management: Passwords:Manage and Passwords:View, and the SDDC Manager Administrator role. No failed or running workflows (Fleet Management > Tasks).
Prior labs: holodeck-02
Required skills:
- VCF Operations console navigation (Fleet Management > Passwords, Tasks, Administration > Integrations)
- SDDC Manager navigation (Security > Password management, Tasks)
- Understanding of the VCF account model (local appliance accounts, SSO accounts, system-managed service accounts and API keys)
- Command-line tools: curl against the SDDC Manager API (GET /v1/credentials), lookup_passwords on the SDDC Manager appliance
- Familiarity with audit and compliance requirements for privileged credentials
Lab Environment
VCF 9.0.x management domain with 4x ESX hosts, SDDC Manager, vCenter, NSX Manager cluster and VCF Operations with fleet management. All components deployed and healthy from a prior lab (holodeck-02 or equivalent). No workload domains required for this lab.
IP Addressing
| Network | Purpose | VLAN |
|---|---|---|
Management (Holodeck Site A untagged Untagged-HoL subnet inside the 10.1.0.0/20 Site A master CIDR, gateway 10.1.1.1; e.g. esx-01a 10.1.1.10) | untagged (Holodeck Site A management network) |
Credentials
| System | Username | Password |
|---|---|---|
| VCF Operations | admin | Set during VCF bring-up (fleet management) |
| SDDC Manager | administrator@vsphere.local | Set during VCF bring-up |
| vCenter Server | administrator@vsphere.local | Managed by VCF; default expiry 90 days (Table 719) |
| NSX Manager | admin | Set during VCF bring-up; managed by VCF (default expiry 90 days) |
| ESX Hosts (root) | root | Set during host commissioning; Update/Remediate in VCF Operations, Rotate in SDDC Manager (no scheduled rotation) |
Tasks
Task 1 Audit the credential inventory in VCF Operations and SDDC Manager
SecurityMap the complete credential landscape: which accounts VCF Operations can update and remediate, which SDDC Manager rotates, which are manual (admin@local, vcf, root on SDDC Manager), and which are system-managed service accounts and API keys. This is the foundation of a defensible credential design.
Log in to the VCF Operations console (https://<vcf_operations_fqdn>) and click Fleet Management > Passwords. Click VCF Management to see the management components (Fleet Management, VCF Operations, VCF Identity Broker, VCF Automation, logs, networks). Then click VCF Instances and your instance or the management domain to see ESX, vCenter, NSX Manager and SDDC Manager (backup account only).
Record the ESX entries: all 4 hosts with the root account. Note status and any password alert. ESX root can be updated or remediated here and rotated on demand in SDDC Manager; scheduled rotation is not supported for ESX accounts (p.2940).
Record the vCenter entries (root) and the SSO account administrator@vsphere.local as listed under the domain. Table 719 (p.1124) gives the defaults: administrator@vsphere.local — update or remediate via VCF Operations, default expiry 90 days; vCenter root — same, 90 days.
Record the NSX Manager entries: root, admin and audit, each updatable/remediable in VCF Operations with a default expiry of 90 days (Table 719). Note which NSX Manager nodes and the VIP they apply to.
Record what is NOT in this list and how it is managed: SDDC Manager admin@local (manual, SDDC Manager API, break-glass, never expires), vcf and root on the SDDC Manager appliance (manual via the OS, 365-day default expiry; root 90 days per p.2941), and the system-managed service accounts and API keys between components (rotated from Administration > Integrations > Accounts in VCF Operations or recreated by VCF Automation — p.2938-2940).
Cross-check with SDDC Manager: log in to SDDC Manager and open Security > Password management to see the same accounts with the Rotate password action. Optionally pull the inventory with the API: curl -k 'https://<sddc_manager_fqdn>/v1/credentials?resourceType=ESXI' -H 'Authorization: Bearer <token>' (also VCENTER, PSC, NSXT_MANAGER). Save the combined inventory (component, account, management path, default expiry, status) to your lab notes.
Validation Gate
Check: Inventory documented from Fleet Management > Passwords (VCF Management and VCF Instance), SDDC Manager Security > Password management and the credentials API, including: 4x ESX root, vCenter root and administrator@vsphere.local, NSX root/admin/audit, SDDC Manager backup; plus the manual accounts (admin@local, vcf, root) and the system-managed service accounts/API keys.
Expected: Complete credential map with the management path (Update/Remediate in VCF Operations, Rotate in SDDC Manager, manual, system-managed) and default expiry for each account.
Common Errors
Task 2 Rotate and schedule rotation from SDDC Manager
SecurityRotation (auto-generated password applied on both the account's server side and every client that uses it) is an SDDC Manager action in 9.0. Scheduled rotation is a design recommendation (VCF-SEC-RCMD-ACT-001) for every account that supports it; ESX accounts do not. This task configures a schedule where the product allows one and documents the exception.
Log in to SDDC Manager (https://<sddc_manager_fqdn>) as an Administrator and click Security > Password management. Review the accounts and the Rotate password action. Note that the rotate options (rotate, rotate all, schedule rotation) are not present in the VCF Operations console in 9.0 (known issue p.269) — SDDC Manager UI or API is the rotation path.
Schedule rotation for an account that supports it (for example the NSX admin account): select the account, open the ellipsis menu and choose Schedule rotation; pick one of the preset intervals offered by the UI. Custom intervals (for example 45 days) are only available through the SDDC Manager API.
Attempt the same for an ESX root account and observe that scheduled rotation is not offered: 'ESX accounts can be rotated, but scheduled rotation is not supported' (p.2940). Record this as a design constraint (VCF-SEC-RCMD-ACT-002: operational practice to rotate on demand for accounts without scheduled rotation).
Review how you will know the schedule ran: SDDC Manager Tasks shows each rotation task; VCF Operations shows password alerts on Fleet Management > Passwords. Decide whether to route alerts by a VCF Operations notification rule (Infrastructure Operations > Configurations > Notifications) to email.
Document the rotation design for the management domain: which accounts are on a schedule (NSX admin/root/audit, vCenter root, administrator@vsphere.local where your change process allows), which are rotated on demand (ESX root), and which are manual (SDDC Manager admin@local, vcf, root). Note Table 719's 90-day default expiry for the SSO and NSX accounts.
Retrieve a rotated password when an operator needs interactive access: SSH to the SDDC Manager appliance as vcf and run lookup_passwords (entity type, then ADMIN credentials), or call GET /v1/credentials. Record how your organisation controls access to this (VCF-SEC-RCMD-ACT-001 implication: new passwords must be retrieved through the API).
Validation Gate
Check: Scheduled rotation configured for at least one supported account in SDDC Manager; ESX root documented as on-demand only; monitoring path and password retrieval procedure documented.
Expected: A rotation design that matches what 9.0 supports: schedules for supported accounts, on-demand for ESX, manual for SDDC Manager local accounts.
Common Errors
Task 3 Execute Update, Rotate and Remediate and validate the result
AvailabilityThree actions, three places: Update (new password you supply, applied server and client side) and Remediate (sync a password already changed on the component) in VCF Operations; Rotate (auto-generated) in SDDC Manager. Knowing which action fixes which situation is what panelists probe.
Preconditions: in the VCF Operations console check Fleet Management > Tasks for failed or running workflows and clear them; make sure no workflow is scheduled during the change (p.2936).
Update: in Fleet Management > Passwords > VCF Instances > management domain, select one ESX host (esx-01a) root and click Update Password. Enter and confirm a new password that meets the recommended requirements (15-20 characters, upper, lower, number, one of ! @ # $ ^ *, no dictionary word, palindrome, >4 monotonic sequences or 3 identical consecutive characters). Click Update Password and follow the progress message.
Validate: SSH to esx-01a with the new password (success) and confirm the host is Active in SDDC Manager Inventory > Hosts and connected in vCenter.
Rotate: in SDDC Manager Security > Password management select esx-02a root and click Rotate password > Rotate. SDDC Manager generates a password per the component policy and re-establishes integrations. Then retrieve it with lookup_passwords or GET /v1/credentials and log in to confirm.
Remediate: simulate an out-of-band change by changing the NSX audit account password directly in NSX Manager (System > Users and Roles). VCF Operations now holds a stale password. In Fleet Management > Passwords select the NSX audit account, click Remediate Password, enter the password you set in NSX and click Remediate Password.
Record for each action: where it ran, what changed on the server side and the client side, how long it took, and how you verified it.
Validation Gate
Check: esx-01a root updated from VCF Operations, esx-02a root rotated from SDDC Manager, NSX audit remediated after an out-of-band change; each verified by login and healthy status.
Expected: Three password operations completed through the correct consoles with evidence.
Common Errors
Task 4 Recover from a synchronisation failure (service accounts and break-glass)
AvailabilityThe failure you will meet in production is not a locked root account but a service account whose password changed on one side only, or an SDDC Manager local account that nobody can reach. VCF 9.0 documents remediation per service account pair; this task walks the documented paths and the break-glass account.
Review the documented manual remediation table (pp.2938-2940) and record the pairs: vCenter service accounts used by SDDC Manager (reset via the vCenter svc-account-mgmt API, then Remediate in SDDC Manager Security > Password management), by NSX (resolve the compute manager in NSX System > Fabric > Compute managers), by VCF Operations/logs/networks/fleet management (Administration > Integrations > Accounts > VCF instance > domain > vCenter tab > System managed credentials > Rotate), by VCF Automation (VCF Instances > vCenter > Rotate service account); NSX service accounts (NSX node users API reset_password, then Remediate in SDDC Manager); SDDC Manager API keys used by VCF Operations (Administration > Integrations > Accounts > VCF instance > VCF Credentials > Rotate).
Exercise one safe path: in VCF Operations go to Administration > Integrations, Accounts tab, expand VMware Cloud Foundation > your VCF Instance > management domain, open the vCenter tab and click Rotate next to System managed credentials. VCF Operations deletes and recreates the service account without affecting the integration (p.2938).
Check the outcome: Administration > Integrations > Accounts shows the account healthy; Fleet Management > Tasks shows the rotation; SDDC Manager Tasks shows the service account creation.
Break-glass review: the SDDC Manager admin@local account is the API break-glass account (manual via the SDDC Manager API, never expires — Table 719). Document who holds it, where it is stored, and the procedure to update it (PATCH /v1/users/local/admin). Also document the vcf and root appliance accounts (passwd vcf / passwd root as root over SSH; root expires after 90 days and an expired root is reset through the vSphere Client console, pp.2941).
Simulate the ESX case conceptually: if an ESX root password becomes invalid for VCF (host reinstalled, password changed at the DCUI), the fix is to set a known password on the host (DCUI or Host Client) and run Remediate Password in VCF Operations for that host. Document this as the ESX recovery path; do not lock accounts in the lab.
Write the runbook: symptom → which side is stale → which documented action (Update, Rotate, Remediate, service-account rotate, break-glass) → verification.
Validation Gate
Check: Service-account remediation map documented; one system-managed credential rotated from Administration > Integrations; break-glass (admin@local, vcf, root) and ESX recovery paths documented; runbook written.
Expected: Recovery procedures grounded in the documented actions, not in invented UI.
Common Errors
Task 5 Design external secrets-manager integration and audit credential operations
ManageabilityEnterprise compliance (SOC 2, ISO 27001) wants central secret custody and audit trails. VCF 9.0 has no built-in vault integration, so the design is an integration around the SDDC Manager credentials API; VCF 9.1 adds VCF Operations password APIs intended for third-party tools such as an enterprise vault, plus fleet and instance password policies. This task is a design exercise with an audit review.
Establish the 9.0 facts: there is no External Vault page in SDDC Manager or VCF Operations in 9.0 (searched the 9.0.1 documentation for 'External Vault' and 'HashiCorp' — neither surfaced as a product feature). Credentials are stored by SDDC Manager and exposed through GET /v1/credentials; VCF Operations shows status and alerts. Record this as the baseline.
Design the integration: an external secrets manager (for example HashiCorp Vault or Azure Key Vault) pulls current passwords through the SDDC Manager API after each rotation (triggered by a scheduled job or by a VCF Operations notification rule webhook on password alerts), stores them under a per-component path, and is the only interactive access path for operators. Define the service account for the API, its SDDC Manager role, network path and audit expectations.
Note the 9.1 change: VCF 9.1 release notes describe VCF Operations password APIs for integration with third-party tools such as an enterprise vault, and fleet/instance password policies. State how your design would move from SDDC Manager API polling to those APIs when the environment is upgraded.
Audit review in 9.0: list the evidence available — SDDC Manager Tasks (rotation, remediate, update), VCF Operations Fleet Management > Tasks and password alerts, and the credentials API. Pull the password-related tasks for the lab period from SDDC Manager Tasks and VCF Operations Tasks and export or screenshot them.
Write a short compliance summary for the lab period: accounts updated, rotated and remediated (with timestamps from the tasks), accounts with alerts, accounts not covered by scheduled rotation (ESX root, SDDC Manager local accounts) and the compensating operational practice (Table 63: VCF-SEC-RCMD-ACT-002/-003).
Recommendations: list the controls you would add — scheduled rotation on every supported account, notification rules for password alerts, API-based vault synchronisation, quarterly break-glass tests, and restricted access to lookup_passwords.
Validation Gate
Check: Baseline statement (no built-in vault in 9.0), integration design with trigger/API/storage/access/audit, 9.1 migration note, evidence set from SDDC Manager and VCF Operations tasks, compliance summary and recommendations.
Expected: A defensible secrets-management design that uses what 9.0 provides and anticipates 9.1.
Common Errors
Final Validation
You have mapped the VCF 9.0 credential landscape, used the right console for each action (Update and Remediate in VCF Operations, Rotate and scheduled rotation in SDDC Manager), recovered from synchronisation failures with the documented service-account procedures, and designed an external secrets-manager integration around the credentials API. You can defend the rotation schedule the product allows and the compensating practices where it does not.
✓ Credential Inventory → Fleet Management > Passwords and SDDC Manager Security > Password management reconciled with the credentials API: ESX root x4, vCenter root and administrator@vsphere.local, NSX root/admin/audit, SDDC Manager backup, plus manual and system-managed accounts
✓ Rotation Design → Scheduled rotation set for a supported account in SDDC Manager; ESX root documented as on-demand only; monitoring path defined
✓ Update / Rotate / Remediate → esx-01a updated (VCF Operations), esx-02a rotated (SDDC Manager), NSX audit remediated after an out-of-band change; all verified
✓ Recovery → Service-account remediation map, one system-managed credential rotated from Administration > Integrations, break-glass and ESX recovery documented in a runbook
✓ Secrets-manager design and audit → Design around the SDDC Manager credentials API with the 9.1 API migration note; evidence from tasks; compliance summary
Cleanup / Restore
Snapshot: vcp-admin-07-complete
• Confirm no password task is still running in Fleet Management > Tasks or SDDC Manager Tasks
• Record the current passwords for esx-01a (updated) and esx-02a (rotated — retrieve with lookup_passwords) in your lab password store
• Decide whether to keep the scheduled rotation on the NSX admin account for later labs or remove it to match the lab baseline
• Take a post-lab snapshot: 'vcp-admin-07-complete — password operations executed, evidence captured'
Design Reflection (VCDX)
A VCDX panelist examining your credential management strategy would ask: Which accounts can VCF rotate on a schedule and which cannot (ESX root), and what compensates? Why do Update/Remediate live in VCF Operations while Rotate stays in SDDC Manager in 9.0, and what does that mean for your runbooks and RBAC? What is the blast radius of a compromised ESX root versus administrator@vsphere.local? How do you give a central secrets manager custody of passwords when 9.0 has no built-in vault integration? Defend your rotation schedule with the product's constraints and your risk analysis, not vendor defaults.
Requirements
- All VCF-managed credentials must be rotatable (manual or automated) at least quarterly
- Rotation operations must not disrupt service availability to management plane or workloads
- Audit trail must be maintained for all credential access and rotations for SOC 2 / ISO 27001 compliance
- Break-glass recovery procedure must exist for accounts that cannot be auto-rotated due to service dependencies
Constraints
- Rotate, rotate all and schedule rotation are SDDC Manager actions in 9.0; they are not present in the VCF Operations console (known issue) — two consoles for one lifecycle
- ESX accounts can be rotated on demand but scheduled rotation is not supported; SDDC Manager admin@local, vcf and root are manual
- Update and Remediate handle one account at a time and require no running or failed workflows; VCF Operations password status refreshes once a day
- No built-in external vault integration in 9.0; integration is API-based (SDDC Manager credentials API; VCF Operations password APIs from 9.1)
Assumptions
- VCF Operations and SDDC Manager have network connectivity to all managed components (ESX, vCenter, NSX) for password operations
- The operator's VCF Operations role includes Fleet Management: Passwords:Manage and Passwords:View, and the operator holds the SDDC Manager Administrator role
- Operators understand which action (Update, Rotate, Remediate) fits which situation and where it runs
- An external secrets manager, if required, is integrated through the API rather than expected as a product feature in 9.0
Risks
- Password changed on one side only (component reset out of band, or rotation interrupted) leaves VCF with a stale credential — IMPACT: integration failures, failed workflows — MITIGATION: Remediate in VCF Operations; for service accounts follow the documented per-pair procedures
- Rotation policy too aggressive (e.g., daily rotation) increases chance of automation failure and service disruption — IMPACT: operational noise and false alarms, MITIGATION: use 30-90 day cycle with testing
- Break-glass account unused for 12+ months becomes stale and unusable when needed in emergency — IMPACT: cannot access system in incident, MITIGATION: test break-glass quarterly
- Compromised SDDC Manager exposes all stored credentials (lookup_passwords, credentials API) — IMPACT: lateral movement across the instance — MITIGATION: restrict SDDC Manager and API access, rotate on a schedule where supported, mirror into an enterprise vault with its own audit, isolate the management network
Self-Assessment Discussion Prompts
- Why does VCF 9.0 keep Rotate in SDDC Manager while moving Update and Remediate to VCF Operations? What does that split mean for RBAC and for your runbooks, and what would you expect to change once the known issue is resolved?
- Automated rotation failed for an NSX service account. What is the first diagnostic step, and which documented remediation path applies (NSX node users API reset, then Remediate in SDDC Manager)?
- Compare ESX root (on-demand rotation only) with administrator@vsphere.local (schedulable, 90-day default expiry). Which has the tighter tolerance for a failed operation, and why?
- A security audit asks for 15-day rotation on all accounts. Which accounts can meet it in VCF 9.0, which cannot, and what compensating controls do you propose?
- VCF 9.0 has no built-in vault integration and 9.1 adds password APIs for third-party tools. When is an external secrets manager worth building around the API now, and when do you wait for 9.1?
- Design a rotation schedule for a production VCF instance with 3 workload domains. Which accounts rotate together, which independently, and where do the SDDC Manager local accounts fit?
Extensions
Design and Test a Full Credential Rotation Window
Simulate a production maintenance window where you rotate all credentials (ESXi, vCenter, NSX, service accounts) in sequence. Measure the total time, identify dependencies, and document the rollback procedure if any step fails. Create a runbook for the operations team.
harderBuild the API-based vault synchronisation
Deploy a HashiCorp Vault instance in your lab (dev mode is fine). Write a job that authenticates to the SDDC Manager API, reads GET /v1/credentials for each resource type and writes the values into Vault KV paths per component, triggered by a schedule and by a VCF Operations notification-rule webhook on password alerts. Verify that after a Rotate in SDDC Manager the new password appears in Vault and that Vault's audit log records every read. Document what changes when the VCF 9.1 VCF Operations password APIs are available.
harderCredential Compliance Audit: SOC 2 Readiness
Using the credential audit logs from this lab, prepare a SOC 2 Type II report section: 'Credential Management Controls'. Document: (a) Who can access credentials (RBAC), (b) Rotation frequency and method, (c) Audit trail retention, (d) Break-glass procedures, (e) Encryption in transit and at rest. Compare your implementation to SOC 2 CC-6 and CC-7 controls.
harderVCF 5.2 vs VCF 9.0 Credential Architecture Comparison
If you have access to a VCF 5.2 lab, repeat the inventory (Task 1) and the rotation (Task 2) in VCF 5.2, where all password management (rotate, rotate all, schedule rotation, update, remediate) lived in SDDC Manager. Document the differences: the 9.0 split between VCF Operations (Update/Remediate, management components in scope) and SDDC Manager (Rotate), the deprecation of the SDDC Manager password API, and what 9.1 adds (password policies, integration APIs). Write a 2-page comparison.
sameReferences
- VMware Cloud Foundation 9.0 — Managing Passwords for VMware Cloud Foundation ComponentsTier 1 — Official
Update, Remediate (VCF Operations), Rotate and scheduled rotation (SDDC Manager), manual service-account remediation table, lookup_passwords, SDDC Manager local accounts (local PDF text pp.2935-2941) - VCF 9.0 Design — Account and Password Management (Table 719, Table 63)Tier 1 — Official
Default expiry and management path per account; design recommendations VCF-SEC-RCMD-ACT-001..003 (local PDF text p.1124) - VCF 9.0 Release Notes — known issue: rotate password options not available in VCF Operations; SDDC Manager password/certificate API deprecationTier 1 — Official
Workaround: use SDDC Manager UI or API for rotation (local PDF text p.269) - VCF 9.1.0.0 Release Notes — Password Management (Integration Ready)Tier 1 — Official
VCF Operations 9.1 password APIs for third-party tools such as an enterprise vault; fleet and instance password policies - AICPA & CIMA — System and Organization Controls: SOC Suite of Services (SOC 2)Tier 3 — Background reading
External (non-VMware) AICPA reference for SOC 2 credential and access-control expectations; useful for designing audit scope