vCenter Service Recovery
Objectives
- Map vCenter Server service architecture and startup dependency chains
- Diagnose service failures using systemctl status, log analysis, and vpxd health checks
- Execute correct service restart sequences (PSC/SSO then vpxd then vsphere-ui then remaining)
- Troubleshoot vPostgres database connectivity and disk space exhaustion
- Perform vCenter file-based backup and restore operations
- Document service recovery procedures using RCAR methodology for VCDX defense
Prerequisites
VCF lab environment with operational vCenter Server Appliance (VCSA)
Required skills:
- SSH access to VCSA and ESXi hosts
- Basic Linux service management (systemctl)
- Understanding of vCenter role in VCF management domain
Lab Environment
VCF management domain with VCSA, SDDC Manager, and minimum 4 ESXi hosts
- Hosts needed: 4
- VMs needed: 2
- Storage: vSAN ESA datastore
Tasks
Task 1 vCenter Service Architecture & Dependency Mapping
Build a complete mental model of VCSA service architecture — which services depend on which, the correct startup order, and how service health cascades through the VCF stack.
SSH into the VCSA appliance. Run 'service-control --status' to list all vCenter services and their current state. Document every service name and status (RUNNING/STOPPED).
Map the service dependency chain. Start from the bottom: (1) vmware-vpostgres (database), (2) vmware-stsd (SSO/STS token service), (3) vmware-vpxd (core vCenter daemon), (4) vsphere-ui (web client), (5) remaining services. Draw a dependency diagram showing which services block which.
Check the VCSA VAMI (https://<vcsa>:5480) health dashboard. Document the health status of each subsystem: Database, Overall, SSO, Content Library, etc. Compare VAMI health with service-control output.
Document the VCF management domain impact chain: vCenter down means SDDC Manager loses inventory visibility, NSX Manager cannot sync with vCenter, no new workload provisioning possible, but existing VMs continue running unmanaged. Verify by checking SDDC Manager UI while vCenter is healthy.
Validation Gate
Check: Complete service dependency diagram and document VCF impact chain
Expected: All 20+ VCSA services identified and mapped in dependency order. VCF impact chain documented showing management plane vs. data plane separation.
Common Errors
Task 2 Service Failure Diagnosis & Log Analysis
Develop systematic log analysis skills to identify root cause of vCenter service failures — from vpxd crashes to SSO token expiry to database corruption.
Map the critical log locations for vCenter troubleshooting. SSH to VCSA and verify each path exists: /var/log/vmware/vpxd/vpxd.log (core vCenter), /var/log/vmware/sso/ssoAdminServer.log (SSO), /var/log/vmware/vpostgres/postgresql-*.log (database), /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log (UI), /var/log/vmware/rhttpproxy/rhttpproxy.log (reverse proxy). Document file sizes and last-modified timestamps.
Simulate a common failure scenario: check disk space on the VCSA. Run 'df -h' and examine /storage/db (database), /storage/log (logs), /storage/seat (stats), and root filesystem. Document current usage percentages. Note: vCenter raises a yellow disk-exhaustion warning at 75% continuous use and a red critical alarm at 85% (Broadcom KB 313077).
Practice log analysis for three common failure patterns. Search vpxd.log for: (a) 'ODBC connection failed' indicating database connectivity loss, (b) 'SSO initialization failed' indicating SSO/STS service down, (c) 'Failed to initialize VMware VirtualCenter' indicating critical startup failure. Use grep -i to search for these patterns.
Generate a vCenter support bundle for practice. Navigate to VAMI > Support > Create Support Bundle. Alternatively, from the appliance shell run vc-support to generate the bundle; for a quick service-state snapshot use service-control --status --all (or vmon-cli -l to list vMon services — the VCF 9.0.1 docs show vmon-cli with -h, -l and --restart only, p.5809; a '--get-state-all' option did not surface). Document what is included in the bundle and the typical file size.
Check for STS certificate expiry — a common vCenter 7.x/8.x issue that carries into VCF 9.0 upgrades. Check the STS signing certificate with the Broadcom checksts.py script (KB 'Checking Expiration of STS Certificate on vCenter Server'), not certool --status (which validates a certificate against the VMCA CRL). In a VCF-managed environment review certificate expiry per component in VCF Operations > Fleet Management > Security > Certificates (the SDDC Manager API offers GET /v1/domains/{id}/resource-certificates, 'Get Certificates By Domain', for metadata on the certificates of all resources in a domain — SDDC Manager API Reference Guide, developer.broadcom.com). Document expiry dates for all certificates.
Validation Gate
Check: Document all critical log locations, disk space status, and certificate expiry dates
Expected: Complete log location reference created, disk space verified healthy, all certificates checked with expiry dates documented.
Common Errors
Task 3 Service Recovery Procedures & Database Repair
Execute correct service recovery sequences for different failure scenarios — from simple service restart to database repair to full VCSA restore from backup.
Practice the correct service restart sequence. First, stop all services in reverse dependency order: 'service-control --stop --all'. Wait for confirmation. Then start services: 'service-control --start --all'. Monitor startup progress with service-control --status --all and in the vMon log, /var/log/vmware/vmon/vmon.log (it records each service start request and why a service failed to start). Document the startup time.
Practice individual service recovery for the most common failure: vpxd crash. Stop only vpxd: 'service-control --stop vmware-vpxd'. Check vpxd.log for the last error before crash. Start vpxd: 'service-control --start vmware-vpxd'. Monitor startup in vpxd.log — look for 'VirtualCenter started successfully' message.
Practice vPostgres database health check and repair. Connect to the embedded database: /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB. Run basic health queries: SELECT count(*) FROM vpx_vm; (VM count), SELECT pg_database_size('VCDB'); (DB size). Check for long-running queries: SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state = 'active';
Practice disk space recovery procedures for /storage/db exhaustion. Steps: (1) Reduce stats retention via vSphere Client Administration System Configuration Stats Level; (2) Clean old log files: find /storage/log -name '*.log' -mtime +30 and review before deleting; (3) Purge old events/tasks using vCenter MOB or PowerCLI to remove events older than 30 days; (4) Verify space recovered: df -h /storage/db.
Practice VCSA file-based backup and restore. Configure backup schedule in VAMI Backup Configure Backup Schedule. Set target: SFTP server or NFS share. Run an ad-hoc backup. Verify backup completion and document the backup size and duration. Review restore procedure: VCSA ISO Restore point to backup location.
Validation Gate
Check: Successfully execute service restart, database health check, and backup/restore procedures
Expected: Service restart completed in correct dependency order. Database health verified. Backup completed and stored on external target. Restore procedure documented.
Common Errors
Task 4 vCenter Availability in VCF & VCDX Defense
Understand how VCF protects the vCenter appliance - vSphere HA plus file-based backup and restore, with vCenter HA and vSphere FT explicitly not supported (VCF 9.0 design guidance, Table 670 Methods for Protecting vCenter) - and prepare to defend a vCenter recovery design at a VCDX panel.
Document the VCF-supported methods from Table 670 'Methods for Protecting vCenter': vSphere High Availability - Supported: Yes (ensure sufficient cluster resources so VMs restart on another host). vCenter High Availability (vCenter HA) - Supported: No (services must fail over to the passive node, so no continuous availability; recovery time can be up to 15 minutes; additional networking requirements for the private network; additional resources for the passive and witness nodes; lifecycle management is complicated because the standby VMs must be manually deleted and recreated during a lifecycle operation). vSphere Fault Tolerance - Supported: No (the FT vCPU limit would cap the appliance at Medium; FT needs a dedicated network). Add the stack-level alternative the same section gives: multi-rack management vSphere clusters, vSAN stretched clusters, or third-party synchronous replication (metro clusters) protecting the entire management stack rather than one vCenter.
Analyze the design decision for vCenter availability in a VCF management domain. Document: Requirement (R-001: the management vCenter must be recoverable within the agreed RTO; R-002: RPO equal to the backup interval, with the SDDC Manager and vCenter jobs starting within the same 5-minute window), Constraint (vCenter HA and vSphere FT are not supported in VCF - Table 670), Assumption (the management cluster has N+1 capacity and admission control so the appliance can restart), Risk (a vSphere HA restart during an SDDC Manager workflow leaves the workflow failed and requires a manual retry).
Build a vCenter recovery decision tree: (1) Single service failure: restart that service with service-control (RTO: 2-5 min); (2) vpxd crash loop: analyze logs, restart all services in dependency order (RTO: 5-15 min); (3) Host failure: vSphere HA restarts the appliance on another host automatically (RTO: appliance boot time, measure it in the lab); (4) Database corruption or appliance loss: file-based restore - vCenter ISO installer > Restore, Stage 1 deploys a new appliance (management-domain vCenter on a healthy management ESX host), Stage 2 restores from the SFTP backup (RTO: 30-60 min, RPO: last backup); (5) Full site failure: recovery of the management stack at the secondary site per the stretched-cluster or DR design, or VCF Instance Recovery. Document this as a runbook with commands, expected duration and escalation criteria.
Prepare four VCDX defense responses for vCenter recovery design: (1) 'Why not vCenter HA?' - it is not supported in VCF (Table 670): failover to a passive node with recovery up to 15 minutes, a dedicated private network, extra passive and witness nodes, and standby VMs that must be deleted and recreated for every lifecycle operation; vSphere HA plus file-based backup is the supported model. (2) 'What happens to running VMs when vCenter is down?' - no impact: VMs continue on the ESX hosts; only management operations, SDDC Manager workflows and VCF Operations lifecycle tasks pause. (3) 'How do you protect the management stack against a site failure?' - a vSAN stretched management cluster (witness at a third site) or a multi-rack management cluster, protecting vCenter, NSX Manager and SDDC Manager together rather than vCenter alone. (4) 'What is your vCenter backup strategy for VCF?' - daily file-based backup to an external SFTP server, starting within the same 5-minute window as the SDDC Manager backup, 7-day retention, restore tested on a schedule.
Validation Gate
Check: Table 670 documented, design decision D-001 written, recovery decision tree created as a runbook, four defense responses prepared and practiced.
Expected: vCenter availability design for VCF documented with the supported method (vSphere HA + file-based backup), the rejected alternatives and their reasons, and a graduated recovery runbook.
Common Errors
Final Validation
Complete vCenter service recovery with graduated response procedures and VCDX defense preparation
✓ Service dependency chain documented → All 20+ services mapped with startup order and dependencies
✓ Log analysis capability demonstrated → Critical log locations known, failure patterns identified
✓ Recovery procedures tested → Service restart, database health check, and backup/restore completed
✓ vCenter availability in VCF understood → Table 670 documented (vSphere HA supported; vCenter HA and vSphere FT not supported), with the stack-level alternatives and the file-based restore path
✓ VCDX defense responses prepared → Four defense scenarios documented with concise technical responses
Cleanup / Restore
• Verify all vCenter services are running: service-control --status
• Confirm SDDC Manager shows healthy vCenter connectivity
• Remove any temporary support bundles to reclaim disk space
• Revert to snapshot if any configuration changes were made during practice
Design Reflection (VCDX)
vCenter recovery design demonstrates understanding of management plane vs. data plane separation, graduated response methodology, and the critical role of vCenter in VCF lifecycle management. The design decision to protect the management vCenter with vSphere HA and a coordinated file-based backup strategy - and to reject vCenter HA because VCF does not support it - shows the operational maturity expected at VCDX level.
Requirements
- R-001: vCenter availability >=99.99% (<=52 min/year downtime)
- R-002: vCenter RPO equal to the file-based backup interval (daily, coordinated with the SDDC Manager backup within the same 5-minute window); RTO per the graduated recovery runbook
- R-003: VCF management domain operational continuity during component failures
Constraints
- VCSA must run on management domain — cannot be hosted externally
- VCF lifecycle operations (upgrades, expansion) require vCenter availability
- SDDC Manager, NSX Manager, and the VCF Operations and VCF Automation components all depend on vCenter for inventory
- vCenter HA and vSphere FT are not supported for VCF vCenters (VCF 9.0 design guidance, Table 670); vSphere HA is the only supported availability method
Assumptions
- The management cluster has N+1 capacity with vSphere HA admission control so the vCenter appliance can restart on another host
- External SFTP/NFS target available for file-based backups
- Operations team trained on graduated recovery procedures
Risks
- A vSphere HA restart of vCenter during an active SDDC Manager workflow may leave the workflow in a failed state - requires manual retry
- STS certificate expiry causes cascading authentication failure across all VCF components — monitor proactively
- Database corruption from disk exhaustion requires full restore — implement proactive disk monitoring
Self-Assessment Discussion Prompts
- How would you design vCenter recovery differently for a VCF environment with 500+ hosts across 5 workload domains?
- What metrics would you monitor to predict vCenter failures before they cause outages?
- How does vCenter recovery priority change during a VCF upgrade window?
- If you could only implement one preventive measure for vCenter availability, what would it be and why?
Extensions
vSphere HA Restart Testing for the vCenter Appliance
In your Holodeck lab, identify the ESX host running the management vCenter appliance and fail that host (power off from the outer vSphere layer). Measure the time until vSphere HA restarts the appliance and the vSphere Client responds, verify that SDDC Manager, NSX Manager and VCF Operations reconnect, and document any workflow that needs a retry.
Automated vCenter Health Monitoring
Build a PowerCLI or Python script that checks vCenter service status, disk space, certificate expiry, and database size on a schedule. Output a health report and send alerts when thresholds are breached. Integrate with VCF Operations for centralized monitoring.
Multi-vCenter Recovery Coordination
Design a recovery runbook for a VCF environment with management domain vCenter and 3 workload domain vCenters. Document the recovery sequence, inter-vCenter dependencies, and SDDC Manager coordination required. Practice with Holodeck multi-domain deployment.
⚠ Known Pitfalls
References
- vCenter Server 9.0 Administration Guide — Service Management and Troubleshooting chapter
- VMware KB 2147144 — How to stop, start, or restart vCenter Server services
- VMware VCF 9.0 Administration Guide — Management Domain Backup and Recovery
- VMware KB 2149253 — vCenter Server Appliance file-based backup and restore
- VMware Cloud Foundation 9.0 design guidance - vCenter design, Table 670 Methods for Protecting vCenter (vSphere HA supported; vCenter HA and vSphere FT not supported)