Certificate Rotation Across VCF 9.0 Components

VCF 9.0Advancedadminarchitectvcdx⏱ 120 min

Core certificate architecture applies to VCF 9.0.0 through 9.0.2. VCF 5.2.x uses SDDC Manager CLI-driven rotation — see Extension 1 for 5.2 comparison.

Objectives

Map the VCF 9.0 certificate trust chain architecture (root CA, intermediate CA, instance certificates) and identify trust boundaries across SDDC Manager, vCenter, NSX, and ESXi
Audit certificate inventory, expiry timelines, and issuer trust relationships across a multi-domain VCF stack
Execute automated certificate rotation via VCF Operations Manager (VOM) for the management domain
Perform manual certificate rotation for components outside VOM scope (NSX, ESXi edge nodes, external integrations)
Design and implement certificate monitoring and alerting that fits a 90-day (or shorter) rotation SLA

Prerequisites

VCF 9.0 management and at least 1 workload domain fully deployed and healthy. SDDC Manager, vCenter, NSX Manager, and ESXi hosts all online and accessible via vSphere Client and REST APIs. VCF Operations Manager (VOM) deployed and connected to the VCF instance. At least one domain with 4-6 ESXi hosts and 1-2 NSX components (Manager, Edge nodes if applicable). A test PKI with an intermediate CA capable of issuing certificates is available (internal Microsoft CA, Linux OpenSSL CA, or equivalent). Internet connectivity for certificate validation (OCSP, CRL checks) or offline validation configured.

Prior labs: holodeck-02

Required skills:

Certificate fundamentals (X.509 chain validation, trust roots, CSR generation)
SDDC Manager UI and API navigation
vCenter and NSX certificate management screens
ESXi host certificate management (esxcli, host profiles)
OpenSSL or equivalent CLI tools (openssl x509 -text, openssl verify, etc.)
Basic REST API calls (curl or Postman) for programmatic certificate operations

Lab Environment

VCF 9.0 management domain (SDDC Manager, vCenter 1-2 instances, NSX Manager cluster) + 1 workload domain (4-6 ESXi hosts, 1 NSX Manager or Edge cluster). VCF Operations Manager (VOM) deployed and managing certificate lifecycle for management domain components. Optional: external integration endpoint (e.g., Kubernetes API server) requiring mTLS with VCF-signed certificates.

graph TB
  CA[Internal CA<br/>Microsoft CA or OpenSSL CA]
  CA -->|issues certs| SDDC[SDDC Manager<br/>certificate]
  CA -->|issues certs| VC[vCenter<br/>certificate]
  CA -->|issues certs| NSX[NSX Manager<br/>certificate]
  CA -->|issues certs| ESXi[ESXi Host<br/>certificates]
  VOM[VCF Operations Manager<br/>Certificate Lifecycle] --> SDDC
  VOM --> VC
  VOM --> NSX_Mgmt[NSX Mgmt<br/>via API]
  NSX_Mgmt --> NSX
  Manual[Manual Cert<br/>Rotation] --> NSX_Edge[NSX Edge<br/>Nodes]
  Manual --> ESXi
  Monitor[Certificate<br/>Monitoring<br/>& Alerts] -.->|checks expiry| SDDC
  Monitor -.->|checks expiry| VC
  Monitor -.->|checks expiry| NSX

Credentials

System	Username	Password
SDDC Manager	`administrator@vsphere.local`	From VCF deployment
vCenter Server	`administrator@vsphere.local`	Same as SDDC Manager
NSX Manager	`admin`	From VCF bring-up spec
VCF Operations Manager	`admin`	Deployed with VCF instance, default or custom
Internal CA Admin	`ca-admin or domain user`	For CSR signing — provide your own

Tasks

Task 1 Map and audit the VCF 9.0 certificate architecture and trust chain

Security

Before you rotate a single certificate, you must understand the architecture. In production, you have 50+ certificates across 20+ components. Which ones are self-signed? Which point to an internal CA? Which use Broadcom's intermediate CA? Which trust boundaries would break if you installed the wrong certificate? This task builds a complete certificate map — the foundation for any rotation strategy.

Step 1

In SDDC Manager, navigate to Administration > Certificates. You'll see the management domain certificate inventory. Note: SDDC Manager shows certificates for SDDC Manager, vCenter, and NSX Manager. Document the issuer, subject, and expiry date for each. Create a table: Component | Issuer | Subject | Expiry Date | Days Until Expiry.

Certificate inventory table with at least 5 entries (SDDC Manager itself, vCenter, NSX Manager, possibly others)

Step 2

For the SDDC Manager certificate specifically, click on it to view details. Note the certificate chain (issuer, issuer's issuer, root). Take a screenshot or download the certificate chain (DER or PEM format). Use OpenSSL to inspect the chain: openssl x509 -in <cert-file> -text -noout | grep -A 5 'Issuer:\|Subject:'

Certificate chain showing: Leaf cert (SDDC Manager) -> Intermediate CA (usually Broadcom or your internal CA) -> Root CA

Step 3

Now map trust boundaries. In vCenter (via vSphere Client), navigate to Administration > Certificates > Trusted Root CA Certificates. Verify: Does vCenter trust the CA that issued SDDC Manager's certificate? List all trusted root CAs in vCenter. This is a trust boundary: if SDDC Manager is signed by a CA that vCenter doesn't trust, mTLS between them will fail.

List of trusted root CAs in vCenter (expect 5-20 entries, including VMware CA, internal CA, Broadcom CA)

Step 4

In NSX Manager (https://nsx-manager.vcf.sddc.lab), navigate to System > Certificates > General. View NSX Manager's certificate issuer and compare to vCenter's trusted roots. Document: Does vCenter trust NSX Manager's CA? Does NSX Manager trust the CA that issued vCenter's certificate (for management plane callbacks)?

Trust confirmation: vCenter -> NSX mutual trust documented

Step 5

For ESXi hosts, use SSH to connect to one host (e.g., esxi-1.vcf.sddc.lab). View the host certificate: esxcli system ssl config get or cat /etc/vmware/ssl/rui.crt | openssl x509 -text -noout. Note the issuer (likely 'VMware, Inc.' or your CA). Compare to the trust roots on vCenter — verify vCenter trusts ESXi certificates.

ESXi certificate issuer and expiry documented. Confirmation that vCenter trusts ESXi's CA.

Step 6

Create a certificate architecture diagram or detailed map showing: (a) Leaf certificates (SDDC Manager, vCenter, NSX, ESXi), (b) Their issuers (Broadcom CA, internal CA, etc.), (c) Trust paths (which components trust which CAs). Highlight any trust gaps or mismatches (e.g., 'ESXi issued by VMware but vCenter trusts only internal CA for host management').

Comprehensive certificate architecture map with trust paths and any identified gaps

Validation Gate

Check: (1) Certificate inventory table: all major components listed (SDDC Manager, vCenter, NSX, ESXi) with issuer, subject, expiry, days-to-expiry, (2) Certificate chain verified with OpenSSL for SDDC Manager, (3) Trust roots mapped: vCenter's trusted CAs listed, (4) ESXi certificate issuer verified, (5) Trust paths documented and any gaps identified

Expected: Complete certificate architecture map ready for rotation planning. You can now answer: 'If I rotate SDDC Manager to a new CA, what other components must I update to maintain trust?'

Common Errors

SDDC Manager > Administration > Certificates shows 'Access Denied' or the page is blank

Cause: User role lacks certificate management permissions, or SDDC Manager's cert API endpoint is temporarily unavailable

Fix: Log in as administrator@vsphere.local or a user with Infrastructure Admin role. If still blank, wait 5 minutes (SDDC Manager may be synchronizing cert state). Refresh the page.

OpenSSL command shows 'Unable to load certificate' when inspecting downloaded cert

Cause: Certificate was downloaded in DER format, but OpenSSL expects PEM (text-based). Or file extension mismatch.

Fix: Specify the format: openssl x509 -inform DER -in <cert-file> -text -noout. Or convert: openssl x509 -inform DER -in <cert.der> -out <cert.pem>

NSX Manager certificate issuer shows 'CN=NSX,O=VMware' but vCenter's trusted roots don't include this issuer

Cause: NSX Manager is using a self-signed certificate (issuer = subject, or issued by NSX's built-in CA). vCenter may not yet trust it, or needs NSX CA imported.

Fix: In NSX Manager, export the root CA certificate. In vCenter, import it to Trusted Root CA Certificates. Then retry the trust check.

Task 2 Generate Certificate Signing Requests (CSRs) and configure internal CA issuing

Security

You've mapped the architecture. Now you'll generate CSRs for rotation. This is the operational gateway: you're preparing to replace all leaf certificates with new ones from your internal CA. In production, this requires coordination with your security/PKI team, change control approval, and testing on a lab or pre-production clone.

Step 1

In SDDC Manager, navigate to Administration > Certificates > Generate CSR. Fill in: Subject (CN=sddc-manager.vcf.sddc.lab, O=Your Org, C=US), Validity (730 days is typical for leaf certs to minimize rotation frequency), Key Size (2048 or 4096). Click 'Generate CSR'.

CSR generated and displayed (PEM format, ready to download). SDDC Manager shows 'CSR pending — awaiting certificate installation'.

Step 2

Download and save the CSR (e.g., sddc-manager-new.csr). Repeat for vCenter: in vCenter's vSphere Client, navigate to Administration > Certificates > Certificates. Find the vCenter SSL certificate, click 'Renew' or 'Generate CSR' (UI varies). Generate CSR with Subject: CN=vcenter.vcf.sddc.lab.

Two CSRs saved: sddc-manager-new.csr and vcenter-new.csr

Step 3

For NSX Manager, navigate to System > Certificates > Certificate Management. Click 'Generate CSR'. Generate with Subject: CN=nsx-manager.vcf.sddc.lab. Download CSR.

Three CSRs saved (SDDC Manager, vCenter, NSX Manager). All ready for CA signing.

Step 4

Now configure your internal CA to issue certificates from these CSRs. If using Microsoft CA: submit CSRs via web enrollment (http://ca-server/certsrv), select 'Web Server' or 'Custom' template (to match FQDN subject). If using OpenSSL: openssl ca -in sddc-manager-new.csr -out sddc-manager-new.crt -days 730 -outdir ./certs. Sign each CSR.

Three issued certificates (sddc-manager-new.crt, vcenter-new.crt, nsx-manager-new.crt) in PEM format, signed by your internal CA

Step 5

Verify the issued certificates match the CSR subjects. For each: openssl x509 -in <cert.crt> -text -noout | grep -A 2 'Subject:'. Confirm Issuer is your internal CA (not Broadcom or a self-signed issuer).

Certificate verification: all 3 certs issued by internal CA, subjects match the original CSRs

Step 6

Also generate the full certificate chain (leaf + intermediate + root CA cert). Your CA should provide this as a bundle or you assemble it manually: cat sddc-manager-new.crt intermediate-ca.crt root-ca.crt > sddc-manager-chain.crt. Create chains for all 3 components.

Three certificate chains (sddc-manager-chain.crt, vcenter-chain.crt, nsx-chain.crt) ready for installation

Validation Gate

Check: (1) Three CSRs generated (SDDC Manager, vCenter, NSX), (2) Three certificates issued by internal CA with matching subjects, (3) Certificate chain for each component assembled (leaf + intermediates + root), (4) OpenSSL verification confirms issuer is internal CA, not Broadcom/self-signed

Expected: All certificates ready for installation. CSR requests show 'pending' status in management UIs, waiting for cert installation to proceed.

Common Errors

CSR generation in SDDC Manager fails with 'Certificate key generation error' after 30 seconds

Cause: SDDC Manager's cryptographic service is overloaded or temporarily hung. Rare but has occurred in nested lab environments.

Fix: Restart SDDC Manager's certificate service: SSH to SDDC Manager, run systemctl restart <cert-service> (service name varies by VCF version). Retry CSR generation.

Internal CA rejects the CSR with 'Invalid Subject Name' or 'Key Size Mismatch'

Cause: CSR subject doesn't match your CA's template requirements (e.g., CA requires O=Org but CSR omitted it). Or key size (2048) doesn't match CA's minimum (4096).

Fix: Review your CA's certificate template or policy. Adjust CSR subject to match. If key size mismatch, re-generate CSR with larger key (4096). Resubmit to CA.

Certificate chain assembly shows 'unable to load certificate' when concatenating PEM files

Cause: One of the intermediate or root cert files is in DER format, not PEM. Text concatenation fails.

Fix: Verify all files are in PEM format (text, begins with -----BEGIN CERTIFICATE-----). Convert any DER files: openssl x509 -inform DER -in <file.der> -out <file.pem>. Then concatenate.

Task 3 Install rotated certificates via VCF Operations Manager (VOM) for management domain components

Availability

VCF Operations Manager (VOM) is the orchestration layer that automates certificate rotation for management domain components (SDDC Manager, vCenter, NSX Manager). It handles the sequencing, validates trust chains, and coordinates service restarts with minimal downtime. This task uses VOM to rotate all three components in a single coordinated operation.

Step 1

In VCF Operations Manager (https://vom.vcf.sddc.lab or via SDDC Manager's integrated UI), navigate to Operations > Certificate Management. You'll see the current certificates for management domain components (SDDC Manager, vCenter, NSX Manager).

VOM certificate dashboard showing current cert status, issuers, and expiry dates for all management components

Step 2

Click 'Initiate Certificate Rotation' or 'Update Certificate'. VOM will guide you through uploading the new certificates and chain. For SDDC Manager: upload sddc-manager-chain.crt (the chain you assembled in Task 2). VOM will validate the cert matches SDDC Manager's CSR and is from a trusted issuer.

Certificate upload successful. VOM shows validation: 'Certificate issuer verified. Matches CSR. Trust path confirmed.'

Step 3

Repeat for vCenter: upload vcenter-chain.crt. Then NSX Manager: upload nsx-chain.crt. VOM queues all three for installation.

All three certificates uploaded and validated. VOM shows 'Ready to apply' status.

Step 4

Before clicking 'Apply', review VOM's sequence plan. VOM should propose: (1) Install NSX Manager cert first (lowest impact), (2) Install vCenter cert (impacts host communication), (3) Install SDDC Manager cert last (controls management plane). Confirm the sequence is correct for zero-downtime rotation (NSX is least critical for moment-to-moment operations).

VOM installation sequence displayed and confirmed by user

Step 5

Click 'Apply Certificates'. VOM will begin the rotation. It will: (a) install NSX cert, restart NSX Manager services (5-10 min downtime expected), (b) validate NSX is back up and trust relationships re-established, (c) install vCenter cert, restart vCenter services (2-5 min downtime), (d) install SDDC Manager cert, brief downtime (1-2 min). Monitor the progress in VOM's Activity pane.

Certificate rotation proceeding. Each component rotates in sequence. VOM reports success/failure for each step.

During vCenter restart, vSphere Client sessions may disconnect. Web UI may briefly show 'service unavailable'. This is normal. Do NOT interrupt the process.

Step 6

After all three rotations complete, VOM should show all three certificates with the new issuer and new expiry dates. Verify in SDDC Manager > Administration > Certificates that the issuer is now your internal CA (not Broadcom).

All three management domain components now show new certificates issued by internal CA, with new expiry dates (typically 730 days in future)

Validation Gate

Check: (1) VOM certificate rotation completed successfully (all 3 components rotated), (2) SDDC Manager > Certificates shows new issuer (internal CA, not Broadcom), (3) vCenter and NSX certificates also show new issuer, (4) No alarms or warnings in SDDC Manager/vCenter post-rotation, (5) SDDC Manager can still connect to vCenter and NSX (trust chain validated)

Expected: Management domain certificate architecture successfully rotated to internal CA. All components maintain mutual trust and normal operation.

Common Errors

VOM certificate validation fails with 'Issuer not trusted' or 'Chain validation failed'

Cause: The certificate chain uploaded is incomplete (missing intermediate CA) or the root CA is not in vCenter's trusted roots yet

Fix: Verify the chain includes: leaf cert + all intermediates + root CA. If root is missing, import it to vCenter's Trusted Root CA Certificates first. Then retry certificate upload in VOM.

Certificate rotation begins but stalls during vCenter restart (stuck at 50% for >10 minutes)

Cause: vCenter services are hung or the certificate change triggered a configuration conflict. Rare, but can happen if vCenter's certificate and its SSL port conflict.

Fix: SSH to vCenter and manually restart services: systemctl restart vmware-vpxd vmware-vsan-health vmware-rhttpproxy. Wait 5 minutes. Check vCenter UI is accessible. VOM may auto-continue, or you may need to manually validate and trigger next step.

NSX Manager cert installs successfully, but NSX reports 'Certificate not trusted by SDDC Manager'

Cause: NSX cert's issuer (internal CA root) is not in SDDC Manager's trusted store. VOM should have handled this, but nested labs sometimes have stale cert caches.

Fix: In SDDC Manager, manually import the root CA certificate: Administration > Certificates > Import Root CA. Then have SDDC Manager re-validate NSX's trust relationship via API call or re-run the trust discovery.

Task 4 Perform manual certificate rotation for NSX Edge nodes and ESXi hosts (out-of-band components)

Security

VOM handles management domain (SDDC Manager, vCenter, NSX Manager), but NSX Edge nodes, individual ESXi hosts, and external integration endpoints (Kubernetes, load balancers, syslog collectors) are out-of-scope for VOM. You must rotate these manually. This is a gotcha in production: teams finish VOM rotation thinking they're done, then discover 50 edge nodes still have old certs expiring in 30 days.

Step 1

For NSX Edge nodes (if deployed), navigate in NSX Manager to System > Certificates > NSX Edge Certificates. View current edge node certificates. If edges exist, they should be using NSX Manager's CA (often self-signed or internal). You have two options: (a) rotate edges to your internal CA (requires manual CSR + certificate upload per edge), or (b) keep edges on NSX's internal CA and import NSX's root CA into vCenter/SDDC Manager for trust.

NSX Edge certificates listed with issuers and expiry dates. Decision made: rotate to internal CA (manual, more complex) or keep NSX CA (simpler, but adds complexity to trust chain)

Step 2

For this lab, assume edge nodes keep their NSX-issued certs (simpler path). Instead, focus on ESXi hosts. To rotate ESXi host certificates, you'll use ESXi Certificate Management API or esxcli. Procedure: (1) Generate CSR for each host, (2) Have internal CA sign, (3) Install cert on each host, (4) Restart hostd service.

Decision documented: NSX edges keep internal CA, ESXi hosts will be rotated to internal CA

Step 3

Generate ESXi CSR for host 1. SSH to esxi-1.vcf.sddc.lab. Run: esxcli system ssl config get (to see current cert location, typically /etc/vmware/ssl/rui.crt). Then: esxcli system ssl cert generate --host=esxi-1.vcf.sddc.lab --dir=/tmp (generates CSR + private key). A CSR file will be written to /tmp.

CSR generated in /tmp on esxi-1. Private key saved (do not lose this!)

Step 4

Copy the CSR from esxi-1 to your local machine (scp esxi-1:/tmp/esxi-1.csr .). Submit to your internal CA for signing. Once signed, copy the certificate back to esxi-1: scp esxi-1-signed.crt esxi-1:/tmp/

CSR signed by internal CA. New cert (esxi-1-signed.crt) ready to install.

Step 5

Install the new cert on esxi-1. SSH to the host. Run: esxcli system ssl cert install --cert=/tmp/esxi-1-signed.crt (this installs and automatically updates /etc/vmware/ssl/rui.crt). Then restart the hostd service: /etc/init.d/hostd restart

Certificate installed. hostd service restarted. ESXi host still accessible (if network connectivity is stable).

Restarting hostd briefly disconnects vCenter's connection to the host. Expect 'Host Disconnected' status in vCenter for 1-2 minutes. Hosts will auto-reconnect.

Step 6

Repeat Steps 3-5 for the remaining ESXi hosts in the cluster. For each host: generate CSR, submit to CA, install new cert, restart hostd. Stagger the restarts: do 1 host every 5 minutes to avoid simultaneous host disconnects and potential HA or vMotion issues.

All ESXi hosts updated with new certificates from internal CA. Hosts briefly disconnect and reconnect in staggered fashion (no cluster-wide outage).

Validation Gate

Check: (1) NSX Edge certs audited; decision documented (keep NSX CA or rotate to internal CA), (2) All ESXi hosts successfully rotated to internal CA (verify with esxcli system ssl cert get on each host), (3) Hosts are online and connected to vCenter post-rotation, (4) No alarms in vCenter regarding host connection security

Expected: ESXi host certificates rotated to internal CA. NSX Edge certs either rotated or documented as using NSX CA (with trust path validated). Complete certificate chain updated across all VCF components.

Common Errors

esxcli system ssl cert generate fails with 'Certificate generation requires valid VMware license'

Cause: ESXi host license is missing or in grace period. Some legacy ESXi versions require valid licensing for cert operations.

Fix: Verify host licensing: esxcli system license show. If missing, apply a license (Holodeck environments usually use evaluation licenses). Retry cert generation.

After installing new cert on esxi-1 and restarting hostd, the host remains 'Disconnected' in vCenter for >5 minutes

Cause: vCenter doesn't trust the new cert (issuer not in vCenter's trusted roots). Host's SSL cert is valid, but vCenter rejects it during handshake.

Fix: Import the internal CA root certificate into vCenter's Trusted Root CA Certificates. Then right-click the host > Refresh. vCenter will re-validate trust and the host will reconnect.

ESXi host becomes inaccessible (no SSH, no vCenter connection) after cert restart. SSH still works briefly, then hangs.

Cause: The new certificate and private key are mismatched or the cert has wrong CN/SAN values. hostd can't initialize SSL socket.

Fix: You need recovery access. Boot the host from recovery media or use IPMI/iLO to access emergency console. Roll back cert: cp /etc/vmware/ssl/rui.crt.bak /etc/vmware/ssl/rui.crt, restart hostd. Investigate why cert was invalid (CSR mismatch, signing error, etc.)

Task 5 Implement certificate monitoring, alerting, and automate renewal workflows

Manageability

Rotation is a one-time event. Monitoring is forever. A VCDX design must include a certificate monitoring strategy: automated detection of approaching expiry, alerting to operations teams, and ideally, automated renewal before expiry. In production, a missed certificate expiry cascades into hours of outage. This task builds the monitoring layer.

Step 1

In SDDC Manager, navigate to Administration > Alarms > New Alarm Rule. Create an alarm for certificate expiry. Configure: (a) Event: Certificate_Expiry_Warning (or similar, vary by SDDC Manager version), (b) Condition: Days_To_Expiry < 30, (c) Action: Send Email to sysadmin@yourcompany.com, Log to SDDC Manager event log. Save the rule.

SDDC Manager alarm rule created. Any certificate expiring within 30 days will trigger an alert.

Step 2

For monitoring ESXi host certificates and NSX components not covered by SDDC Manager alarms, implement an external monitoring script. Example: Python script that SSH'es into each ESXi host, runs esxcli system ssl cert get, parses expiry date, and checks Days_To_Expiry < 30. If true, emit alert (Slack message, email, or syslog to monitoring system).

Monitoring script written (pseudocode or actual Python). Example: script iterates through hosts list, connects via SSH, parses cert expiry, and alerts if < 30 days.

Step 3

Deploy the monitoring script as a cron job on a management workstation or monitoring server. Schedule it to run daily at 6 AM. Example cron entry: 0 6 * /opt/scripts/check-vcf-certs.py >> /var/log/vcf-cert-monitor.log 2>&1

Monitoring script deployed and scheduled. Log file configured to track script execution.

Step 4

For NSX certificates specifically, use NSX API to fetch certificate status. Example API call: GET https://nsx-manager/api/v1/cluster/status (returns cert expiry info in response). Create a wrapper script that polls this endpoint daily and alerts if expiry < 30 days.

NSX certificate monitoring integrated (API-based polling)

Step 5

Now design an automated renewal workflow. Ideally, certificates should auto-renew at ~80% of their lifetime (e.g., 730-day cert renews at 584 days = 146 days before expiry). Design (but don't fully implement, as it requires external CA integration): (a) On day 584, generate new CSR for SDDC Manager, (b) submit to internal CA, (c) CA issues new cert automatically (requires CA webhook or scheduled signing), (d) VOM picks up the new cert and installs it (or you implement a custom orchestrator). Document the workflow and what external system integrations would be needed.

Automated renewal workflow designed with clear handoffs: application (SDDC Manager CSR) -> PKI system (CA auto-signing) -> orchestration (VOM or custom) -> installation

Step 6

Test the monitoring system. Manually advance the clock on SDDC Manager or modify a certificate's metadata (in lab only!) to simulate an expiring cert (< 30 days). Verify the alarm fires and the monitoring script detects it. Confirm the alert reaches you (email, Slack, or log file).

Alert received for simulated expiring certificate. Monitoring system working end-to-end.

Validation Gate

Check: (1) SDDC Manager alarm rule created and configured for cert expiry < 30 days, (2) External monitoring script written and deployed (checks ESXi + NSX certs), (3) Script scheduled as daily cron job, (4) NSX certificate monitoring integrated via API, (5) Automated renewal workflow designed with documented handoffs, (6) Alert system tested with simulated expiry — alert received successfully

Expected: Complete certificate lifecycle monitoring in place. Operations team will receive alerts 30 days before any VCF component certificate expires, enabling proactive renewal before impact.

Common Errors

SDDC Manager alarm rule created but never fires, even when testing with < 30 days to expiry

Cause: Certificate_Expiry_Warning event type doesn't exist in this SDDC Manager version, or alarm rule condition syntax is wrong

Fix: Check SDDC Manager documentation for the correct event type name (varies by version). May need to use a generic event (e.g., 'Any Event on Certificate object') and filter in the condition. Test the alarm rule by triggering it manually if possible.

SSH-based monitoring script fails to connect to ESXi hosts

Cause: SSH key/password not configured in script, or firewall blocks SSH from monitoring workstation to ESXi hosts

Fix: Configure SSH keys (ssh-keygen, copy public key to each host's /root/.ssh/authorized_keys). Or update script with ssh credentials (use secure method: read from vault, not hardcoded). Test SSH connectivity: ssh -i key esxi-1.vcf.sddc.lab 'esxcli system ssl cert get'

NSX API call in monitoring script returns 401 Unauthorized

Cause: API credentials (username/password or token) are missing or incorrect in the script

Fix: Verify NSX admin credentials. If using token-based auth, ensure token is still valid (tokens expire). Re-generate token if needed. Test API call manually: curl -u admin:password https://nsx-manager/api/v1/cluster/status

Automated renewal workflow design completed, but integration with internal CA requires CA API that doesn't exist (e.g., Microsoft CA has no auto-signing webhook)

Cause: Your PKI infrastructure doesn't support full automation. Many CAs require manual approval or use aging protocols.

Fix: Design a semi-automated workflow: (1) Application auto-generates CSR, (2) CA admin manually approves and signs (or auto-signs via template), (3) New cert is pulled by your orchestration system, (4) Installed via VOM. This reduces manual effort from 2 hours to 30 minutes per cert.

Final Validation

You have successfully audited VCF 9.0's certificate architecture, rotated all major component certificates (SDDC Manager, vCenter, NSX Manager, ESXi hosts) from Broadcom's default CA to an internal CA, and implemented comprehensive monitoring and alerting to prevent certificate expiry outages. The VCF stack is now managed under your organization's PKI governance.

✓ Certificate architecture mapped: trust chains documented, trust relationships validated → All major components listed with issuer, subject, expiry, trust paths

✓ CSRs generated for SDDC Manager, vCenter, NSX Manager and issued by internal CA → Three issued certs in PEM format with internal CA as issuer

✓ VOM-orchestrated rotation completed: all 3 management components now have certs from internal CA → Zero downtime rotation, all components back online post-installation

✓ NSX Edge and ESXi host certs rotated (or documented trust path if kept on NSX CA) → ESXi hosts manually rotated; Edge certs audited and decision documented

✓ Monitoring and alerting configured: alarms for certs expiring in < 30 days, external monitoring script deployed → Alert system tested and working; monitoring scheduled as daily cron job

✓ Automated renewal workflow designed: documented handoffs between application, PKI, and orchestration layers → Workflow design doc with integration points and responsible teams identified

Cleanup / Restore

Snapshot: management-domain-post-cert-rotation

• Take a snapshot of SDDC Manager, vCenter, and NSX Manager post-rotation: 'management-domain-post-cert-rotation'

• Document and retain: the issued certificates (sddc-manager-new.crt, vcenter-new.crt, nsx-manager-new.crt) and their expiry dates — attach to your lab notebook

• Disable the simulated certificate expiry trigger (if you modified metadata for testing in Task 5 Step 6)

• If you need to revert certificate changes, restore to the 'management-domain-pre-cert-rotation' snapshot (note: this will revert to Broadcom-issued certs)

Design Reflection (VCDX)

A VCDX panelist examining a multi-domain VCF certificate strategy would ask: How do you handle certificate rotation across 50+ workload domains without causing cascading outages? What's your SLA for cert renewal (30 days? 90 days? How does this fit with your change control calendar)? If an internal CA root cert expires, how do you handle renewal of all dependent leaf certs? How would you design a certificate pinning strategy for critical APIs (SDDC Manager ↔ vCenter, NSX ↔ SDDC Manager)? What's your breakglass procedure if an intermediate CA is compromised? A production design isn't just about technical execution — it's about governance, risk, and operational readiness. Be prepared to defend your PKI strategy against questions like: 'Why internal CA vs Broadcom-managed?' and 'What's the audit trail for every cert installation?'

Requirements

All VCF components (SDDC Manager, vCenter, NSX Manager, ESXi hosts) must use valid X.509 certificates with CN and SAN matching their FQDNs
All component-to-component communication (SDDC Manager → vCenter, NSX Manager ↔ SDDC Manager, ESXi ↔ vCenter) must validate certificates to prevent man-in-the-middle attacks
Certificates must be rotated before expiry (ideally 30-90 days in advance) with zero disruption to operational workloads
Audit trail: every certificate installation, renewal, and rotation must be logged with timestamp, operator, and previous/new cert details
Certificate expiry must be detected proactively (alerting at 30 days remaining) and never allowed to lapse undetected

Constraints

Certificate rotation of SDDC Manager or vCenter causes brief service disruption (< 5 min) due to service restarts — must be scheduled in maintenance window
ESXi host certificate installation causes brief hostd restart (1-2 min) and vCenter disconnect — impacts host-level operations but not guest VMs
VOM can only orchestrate management domain components (SDDC Manager, vCenter, NSX Manager) — Edge nodes and individual ESXi hosts require manual or custom orchestration
Trust chain must be complete: if rotating to internal CA, all consumer components must pre-import the root CA before rotation, or brief trust failures occur
Some external integrations (Kubernetes API, load balancers, syslog collectors) may expect specific issuer or cert pinning — rotation may break integrations if not coordinated

Assumptions

Internal CA is available and reachable (not air-gapped) — or offline CSR/cert distribution process exists
VCF Operations Manager is deployed and connected to the VCF instance
All component FQDNs (sddc-manager.vcf.sddc.lab, vcenter.vcf.sddc.lab, etc.) resolve consistently and match certificate CN/SAN
Change control process allows 30-90 day advance scheduling for cert rotations
Operations team has PKI/certificate experience or will be trained on CA CSR submission and cert installation
Monitoring infrastructure (syslog server, email relay, or alerting system) is available for cert expiry alerts

Risks

Certificate expiry: if monitoring fails and a cert expires without renewal, that component becomes unreachable or untrusted. IMPACT: component outage, possible cascading failures (e.g., vCenter cert expires → ESXi hosts cannot connect to vCenter → cluster isolated). MITIGATION: implement dual monitoring (SDDC Manager alarms + external script), alert at 30 and 14 days before expiry, maintain 30-day advance renewal schedule.
Trust chain breakage: if rotating cert issuer (Broadcom → internal CA) but forgetting to import root CA to consumer components, trust validation fails post-rotation and services cannot communicate. IMPACT: management plane outage until certs are reverted or root CA is imported. MITIGATION: pre-import root CA before rotation; in lab, test rotation on 1 component first.
Certificate mismatch: leaf cert subject (CN, SAN) doesn't match component's FQDN, or private key doesn't match public cert. SSL/TLS handshake fails. IMPACT: component unreachable or untrusted. MITIGATION: validate CSR subject before CA signing; validate cert post-installation (openssl verify, openssl x509 -subject).
Cascading rotation failures: if rotating NSX Manager and it briefly goes down, SDDC Manager tries to re-validate NSX trust and times out. If SDDC Manager is also mid-rotation, brief double-down. IMPACT: 2-3 minute management plane disruption. MITIGATION: coordinate rotation sequence (NSX first, then vCenter, then SDDC Manager) to minimize overlap. Test sequence in lab first.
External integrations break: if Kubernetes API server has cert pinning for vCenter's CA cert, rotating vCenter to new CA breaks K8s-vCenter mTLS until K8s is reconfigured. IMPACT: K8s worker nodes may disconnect from vCenter. MITIGATION: maintain inventory of external integrations with cert dependencies; notify teams 2 weeks before rotation; coordinate update of external system certs in parallel with VCF rotation.

Self-Assessment Discussion Prompts

You have 20 workload domains across 3 data centers. Each domain has its own NSX Manager and vCenter. Do you use a single root CA for all 60 components, or separate CAs per data center? What are the governance and operational trade-offs?
Your internal CA root cert expires in 2 years. All 50 leaf certs (SDDC Manager, vCenters, NSX, ESXi) are issued by intermediates under that root. Design the renewal sequence: when do you rotate the root? When do you rotate intermediates? When do you rotate leaf certs? What's the critical path?
A security audit discovers that 5 ESXi hosts were rotated with certs signed by an unauthorized CA (misconfigured automation script). How do you detect this drift? How do you remediate without downtime?
Your monitoring system detects that NSX Manager's cert will expire in 7 days, but you're in production deployment crunch and can't schedule a maintenance window for 2 weeks. What do you do? (This is a real-world scenario.)
Design an automated certificate renewal system that works across 50+ hosts without requiring human intervention. What are the failure modes if CA unavailable? If orchestration system fails mid-installation?
You're moving from Broadcom-issued certs (globally trusted) to internal CA certs (only trusted internally). What external integrations break? (e.g., Broadcom SaaS services, partner APIs, public-facing load balancers)

Extensions

Implement certificate pinning for critical management APIs

Certificate pinning is a security hardening technique where you lock a specific certificate (or its public key hash) into client code, so even a valid cert from a different issuer is rejected. Implement pinning on one critical path: NSX Manager → SDDC Manager API calls. Every NSX API call to SDDC Manager verifies SDDC Manager's cert fingerprint matches a pinned value. If cert rotates without updating the pin, NSX-SDDC Manager communication breaks, alerting you to the rotation (or forcing you to synchronize). Document the pin update procedure and how it integrates into your rotation workflow.

harder

Compare VCF 5.2.x certificate management with 9.0.x workflow

In VCF 5.2, certificate rotation was entirely SDDC Manager-driven (SDDC Manager UI > Cluster Certificates > Renew). VCF 9.0 introduced VCF Operations Manager (VOM) as the orchestration layer. Research and document: (a) VCF 5.2 cert rotation procedure, (b) how 5.2 handles certificate chains, (c) does 5.2 auto-detect approaching expiry? (d) operator time to rotate 1 domain in 5.2 vs 9.0. Quantify the improvements in 9.0 (automation, zero-downtime orchestration, centralized VOM). If you have a 5.2 lab available, perform the 5.2 rotation manually and time it.

same

Design and implement certificate automation for external integrations

Extend your VCF cert infrastructure to external systems: Kubernetes API server, syslog collector, backup system (e.g., Veeam), etc. Generate certs for these systems from your internal CA. Implement a certificate delivery mechanism (e.g., REST API endpoint on a management workstation that external systems can poll daily for updated certs). Test end-to-end: external system requests cert, receives updated cert, loads it (without restart if possible). This simulates the complexity of managing certs across a heterogeneous infrastructure.

harder

Simulate a certificate compromise and execute emergency remediation

Assume your internal CA's private key was compromised (or suspected). All certs issued by that CA are now untrusted. Design and execute an emergency cert rotation: (a) revoke all VCF certs, (b) generate new certs from a new CA (or new intermediate), (c) rotate all components within a 4-hour window to minimize exposure. Document the incident response procedure, decision tree for when to escalate, and communication plan for external stakeholders. This is a real-world security incident that tests your operational readiness.

harder

References

VMware Cloud Foundation 9.0 Certificate Management and Operations ManagerTier 1 — Official
Official Broadcom documentation — authoritative guide for certificate architecture, rotation procedures, and VOM integration
VCF Operations Manager (VOM) Certificate Lifecycle APITier 1 — Official
API documentation for programmatic certificate operations (CSR generation, cert installation, rotation orchestration)
X.509 Certificates and PKI Best PracticesTier 2 — VMware Press
Microsoft documentation on certificate hierarchy, trust chains, and lifecycle management — applicable to any enterprise PKI
ESXi Certificate Management and ReplacementTier 1 — Official
vSphere 8.0 guide for ESXi certificate generation, installation, and validation
NSX Manager Certificate ManagementTier 1 — Official
NSX 4.x and NSX 9.x certificate architecture, root CA and leaf cert management
William Lam — VCF Certificate Rotation and AutomationTier 3 — Expert Blog
Expert blog on certificate automation, VOM integration, and real-world troubleshooting scenarios
Broadcom Community Forum — VCF Certificate ManagementTier 1 — Official
Community discussions on certificate rotation failures, trust chain issues, and automation strategies

Mark complete