Academy/VCF 9.0 Support (2V0-15.25)/vCenter Service Recovery
This lab targets VCF 9.0

vCenter Service Recovery

VCF 9.0Intermediatevcp-foundationvcp-support⏱ 120 min

Deep-dive into vCenter service architecture, dependency chains, and recovery procedures.

Objectives

  • Map vCenter Server service architecture and startup dependency chains
  • Diagnose service failures using systemctl status, log analysis, and vpxd health checks
  • Execute correct service restart sequences (PSC/SSO then vpxd then vsphere-ui then remaining)
  • Troubleshoot vPostgres database connectivity and disk space exhaustion
  • Perform vCenter file-based backup and restore operations
  • Document service recovery procedures using RCAR methodology for VCDX defense

Prerequisites

VCF lab environment with operational vCenter Server Appliance (VCSA)

Required skills:

  • SSH access to VCSA and ESXi hosts
  • Basic Linux service management (systemctl)
  • Understanding of vCenter role in VCF management domain

Lab Environment

VCF management domain with VCSA, SDDC Manager, and minimum 4 ESXi hosts

Tasks

Task 1 vCenter Service Architecture & Dependency Mapping

This task builds foundational knowledge. Panelists probe: Why does SSO start before vpxd? What happens to SDDC Manager if vCenter is down? How does vCenter HA change recovery procedures?

Build a complete mental model of VCSA service architecture — which services depend on which, the correct startup order, and how service health cascades through the VCF stack.

Step 1

SSH into the VCSA appliance. Run 'service-control --status' to list all vCenter services and their current state. Document every service name and status (RUNNING/STOPPED).

Services include: vmware-stsd (SSO/STS), vmware-vpxd (core vCenter), vsphere-ui (HTML5 client), vmware-vpostgres (embedded database), vmware-rhttpproxy (reverse proxy), vmware-sps (storage profile), vmware-content-library, vmware-eam (ESX Agent Manager), vmware-sca (Service Control Agent), vmware-vmon (service monitor). Total ~20+ services.
The service-control command wraps systemctl for vCenter-specific services. For individual service details, use 'systemctl status vmware-vpxd' format.
Step 2

Map the service dependency chain. Start from the bottom: (1) vmware-vpostgres (database), (2) vmware-stsd (SSO/STS token service), (3) vmware-vpxd (core vCenter daemon), (4) vsphere-ui (web client), (5) remaining services. Draw a dependency diagram showing which services block which.

Dependency chain: vpostgres -> stsd -> vpxd -> [vsphere-ui, sps, eam, content-library, vlcm, wcp]. If vpostgres is down, nothing starts. If stsd is down, vpxd cannot authenticate. If vpxd is down, UI shows 503 but SSO login page still renders.
Key insight for VCDX: service dependency determines recovery priority. A 'vCenter down' ticket could be a simple vsphere-ui restart (30 seconds) vs. a vpostgres corruption (hours of recovery).
Step 3

Check the VCSA VAMI (https://<vcsa>:5480) health dashboard. Document the health status of each subsystem: Database, Overall, SSO, Content Library, etc. Compare VAMI health with service-control output.

VAMI health page shows green/yellow/red indicators for each subsystem. Database health reflects vpostgres state and disk usage. Overall health aggregates all subsystem checks.
VAMI runs independently of vpxd — it's accessible even when vCenter core is down. This makes it the first diagnostic checkpoint.
Step 4

Document the VCF management domain impact chain: vCenter down means SDDC Manager loses inventory visibility, NSX Manager cannot sync with vCenter, no new workload provisioning possible, but existing VMs continue running unmanaged. Verify by checking SDDC Manager UI while vCenter is healthy.

SDDC Manager shows all hosts and VMs inventoried through vCenter. NSX Manager shows transport nodes mapped to ESXi hosts via vCenter. Loss of vCenter means SDDC Manager shows stale inventory but existing workloads are unaffected — ESXi hosts continue running VMs independently.
Critical VCDX point: vCenter is a management plane component, not a data plane component. VM availability is NOT dependent on vCenter availability, but manageability IS.

Validation Gate

Check: Complete service dependency diagram and document VCF impact chain

Expected: All 20+ VCSA services identified and mapped in dependency order. VCF impact chain documented showing management plane vs. data plane separation.

Common Errors

Confusing management plane (vCenter) with data plane (ESXi) — VMs continue running when vCenter is down
Not checking VAMI health before attempting service restarts — VAMI provides critical diagnostic info
Forgetting that SDDC Manager depends on vCenter for inventory — stale state during vCenter outage is expected

Task 2 Service Failure Diagnosis & Log Analysis

Progresses from structured log location knowledge to pattern matching real failure signatures. Panelists ask: Where do you look first? How do you distinguish a service crash from a configuration error? What log entries indicate database corruption vs. disk exhaustion?

Develop systematic log analysis skills to identify root cause of vCenter service failures — from vpxd crashes to SSO token expiry to database corruption.

Step 1

Map the critical log locations for vCenter troubleshooting. SSH to VCSA and verify each path exists: /var/log/vmware/vpxd/vpxd.log (core vCenter), /var/log/vmware/sso/ssoAdminServer.log (SSO), /var/log/vmware/vpostgres/postgresql-*.log (database), /var/log/vmware/vsphere-ui/logs/vsphere_client_virgo.log (UI), /var/log/vmware/rhttpproxy/rhttpproxy.log (reverse proxy). Document file sizes and last-modified timestamps.

Each log file exists and is actively written. vpxd.log is typically the largest (can grow to GB). PostgreSQL logs rotate daily. SSO logs show authentication activity.
For VCDX support scenarios, knowing the exact log file for each symptom saves critical minutes. vpxd.log for inventory/task issues, ssoAdminServer.log for login failures, vpostgres logs for database errors.
Step 2

Simulate a common failure scenario: check disk space on the VCSA. Run 'df -h' and examine /storage/db (database), /storage/log (logs), /storage/seat (stats), and root filesystem. Document current usage percentages. Note: VCSA generates alerts at 75% disk usage.

/storage/db hosts vpostgres data — typically 20-50 GB depending on inventory size and stat retention. /storage/log can grow rapidly with verbose logging. /storage/seat stores performance statistics. Root partition holds OS and application binaries.
Disk exhaustion on /storage/db is the #1 cause of vCenter outages in production. vpostgres stops accepting writes, vpxd crashes, and the entire service chain collapses.
Step 3

Practice log analysis for three common failure patterns. Search vpxd.log for: (a) 'ODBC connection failed' indicating database connectivity loss, (b) 'SSO initialization failed' indicating SSO/STS service down, (c) 'Failed to initialize VMware VirtualCenter' indicating critical startup failure. Use grep -i to search for these patterns.

In a healthy system, these patterns should not appear. Document the grep commands used and verify no errors are present. In a failure scenario, ODBC errors point to vpostgres, SSO errors point to stsd service, initialization failures point to configuration file corruption.
Build a personal troubleshooting cheat sheet: symptom to log file to grep pattern to likely root cause to fix. This is exactly the structured approach VCDX panelists value.
Step 4

Generate a vCenter support bundle for practice. Navigate to VAMI > Support > Create Support Bundle. Alternatively, use the CLI: /usr/lib/vmware-vmon/vmon-cli --get-state-all. Document what is included in the bundle and the typical file size.

Support bundle includes all service logs, configuration files, database state, and system information. Typical size: 500 MB - 2 GB. CLI alternative provides quick service state snapshot without full bundle generation.
In production incidents, generate the support bundle BEFORE attempting fixes — evidence preservation is critical for RCA and VMware SR escalation.
Step 5

Check for STS certificate expiry — a common vCenter 7.x/8.x issue that carries into VCF 9.0 upgrades. Run: /usr/lib/vmware-vmca/bin/certool --status. Also check: for VCF managed environments, verify the SDDC Manager certificate status via API: GET /v1/certificates. Document expiry dates for all certificates.

STS token signing certificate has a 2-year default validity. Machine SSL certificate, Solution User certificates, and VMCA root CA all have separate expiry dates. Expired STS cert causes login failures across all SSO-integrated components.
STS certificate expiry is a sneaky failure — it causes cascading authentication failures across vCenter, NSX Manager, and SDDC Manager simultaneously. Check proactively at least 90 days before expiry.

Validation Gate

Check: Document all critical log locations, disk space status, and certificate expiry dates

Expected: Complete log location reference created, disk space verified healthy, all certificates checked with expiry dates documented.

Common Errors

Not preserving logs before attempting fixes — evidence is lost for root cause analysis
Overlooking /storage/seat partition — stats rollup can consume significant space silently
Forgetting to check STS certificate expiry — causes cascading auth failures across VCF stack
Using 'service-control --stop --all' without checking which services are actually failed — may cause unnecessary downtime

Task 3 Service Recovery Procedures & Database Repair

Builds from least-disruptive (single service restart) to most-disruptive (full restore). Panelists expect you to justify WHY you chose a specific recovery level and when to escalate.

Execute correct service recovery sequences for different failure scenarios — from simple service restart to database repair to full VCSA restore from backup.

Step 1

Practice the correct service restart sequence. First, stop all services in reverse dependency order: 'service-control --stop --all'. Wait for confirmation. Then start services: 'service-control --start --all'. Monitor startup progress in /var/log/vmware/vmon/vmon-startup.log. Document the startup time.

Full service restart takes 5-15 minutes depending on inventory size and database state. vmon orchestrates startup in dependency order. Watch for services that fail to start — they indicate the actual root cause.
service-control --start --all handles dependency ordering automatically. Manual startup of individual services should follow: vpostgres then stsd then vpxd then vsphere-ui then remaining.
Step 2

Practice individual service recovery for the most common failure: vpxd crash. Stop only vpxd: 'service-control --stop vmware-vpxd'. Check vpxd.log for the last error before crash. Start vpxd: 'service-control --start vmware-vpxd'. Monitor startup in vpxd.log — look for 'VirtualCenter started successfully' message.

Individual vpxd restart takes 2-5 minutes. During restart, vsphere-ui shows 503 error. ESXi hosts continue operating normally. SDDC Manager may show warnings about vCenter connectivity.
Always try single-service restart before full restart. It is faster and provides cleaner diagnostic data about the failing component.
Step 3

Practice vPostgres database health check and repair. Connect to the embedded database: /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB. Run basic health queries: SELECT count(*) FROM vpx_vm; (VM count), SELECT pg_database_size('VCDB'); (DB size). Check for long-running queries: SELECT pid, now() - pg_stat_activity.query_start AS duration, query FROM pg_stat_activity WHERE state = 'active';

VCDB contains all vCenter inventory data. VM count should match known inventory. Database size varies: 1-20 GB typical for small-medium environments. Long-running queries (>60 seconds) indicate potential database performance issues.
Never modify VCDB tables directly unless following VMware KB guidance. Direct table manipulation can corrupt the vCenter inventory and void support.
Step 4

Practice disk space recovery procedures for /storage/db exhaustion. Steps: (1) Reduce stats retention via vSphere Client Administration System Configuration Stats Level; (2) Clean old log files: find /storage/log -name '*.log' -mtime +30 and review before deleting; (3) Purge old events/tasks using vCenter MOB or PowerCLI to remove events older than 30 days; (4) Verify space recovered: df -h /storage/db.

Stats level reduction from Level 2 to Level 1 can recover 30-50% of /storage/seat space. Old log cleanup reclaims /storage/log space. Event/task purge reduces VCDB size. Target: keep /storage/db below 70% utilization.
Proactive monitoring: set vCenter alarm for datastore usage on the VCSA VM itself. Alert at 70%, warn at 80%, critical at 85%. This prevents the #1 vCenter outage cause.
Step 5

Practice VCSA file-based backup and restore. Configure backup schedule in VAMI Backup Configure Backup Schedule. Set target: SFTP server or NFS share. Run an ad-hoc backup. Verify backup completion and document the backup size and duration. Review restore procedure: VCSA ISO Restore point to backup location.

File-based backup captures: vPostgres database, configuration files, certificates, and inventory data. Backup size: 2-10 GB depending on inventory. Duration: 10-30 minutes. Restore creates a new VCSA and imports data — requires repointing ESXi hosts to the restored vCenter.
For VCF environments, VCSA backup must coordinate with SDDC Manager backup. Restore order: SDDC Manager first, then vCenter, then NSX Manager. Out-of-order restores cause inventory mismatches.

Validation Gate

Check: Successfully execute service restart, database health check, and backup/restore procedures

Expected: Service restart completed in correct dependency order. Database health verified. Backup completed and stored on external target. Restore procedure documented.

Common Errors

Restarting all services when only one service is failed — causes unnecessary downtime
Directly modifying VCDB tables without VMware KB guidance — voids support and risks corruption
Not coordinating VCSA backup with SDDC Manager backup — causes inventory drift on restore
Forgetting to repoint ESXi hosts after VCSA restore if the IP/FQDN changes

Task 4 vCenter HA Architecture & VCDX Defense

Connects operational recovery knowledge to architectural design decisions. Panelists ask: Why did you choose vCenter HA vs. SRM for vCenter protection? What is the RTO for vCenter HA failover? How does vCenter HA interact with VCF management domain?

Understand vCenter HA (active/passive/witness) architecture, its role in VCF availability design, and prepare to defend vCenter recovery strategies in VCDX.

Step 1

Document vCenter HA architecture. Three nodes: Active (runs all services), Passive (receives continuous replication), Witness (quorum arbitrator, lightweight). Network requirements: management network for client access, HA network (dedicated, isolated, low-latency) for replication. Replication mechanism: vPostgres native streaming replication for database, file-level replication for configuration.

vCenter HA provides automated failover with 5-10 minute RTO. Active node failure triggers automatic promotion of passive node. Witness node prevents split-brain scenarios. HA network should be dedicated VLAN with <10ms latency between nodes.
vCenter HA is different from vSphere HA — vCenter HA protects the vCenter management plane, vSphere HA protects VMs. Both should be enabled in a VCF management domain.
Step 2

Analyze the design decision: vCenter HA deployment in VCF management domain. Document: Requirement (R-001: vCenter RPO=0, RTO<10 min), Constraint (VCF management domain must maintain operational continuity), Assumption (dedicated HA network available between management domain hosts), Risk (HA failover during VCF lifecycle operation may leave SDDC Manager in inconsistent state).

Design decision D-001: Deploy vCenter HA in management domain with dedicated HA network on separate VLAN. Justification: automated failover with near-zero data loss. Alternative considered: manual VCSA file-based restore (RTO: 30-60 minutes, RPO: last backup interval).
VCDX defense: be prepared to explain why vCenter HA is preferred over SRM for vCenter protection. Key points: SRM requires a second vCenter (chicken-and-egg), vCenter HA provides near-zero RPO with automated failover.
Step 3

Build a vCenter recovery decision tree: (1) Single service failure: restart that service (RTO: 2-5 min); (2) vpxd crash loop: analyze logs, restart all services (RTO: 5-15 min); (3) Database corruption: restore from file-based backup (RTO: 30-60 min); (4) VCSA VM failure: vCenter HA automatic failover (RTO: 5-10 min); (5) Full site failure: DR recovery at secondary site (RTO: per SRM/DR design). Document this as a runbook.

Recovery decision tree provides graduated response from least-disruptive to most-disruptive. Each level includes specific commands, expected duration, and escalation criteria.
This graduated recovery approach is exactly what VCDX panelists look for — showing that you do not jump to the most disruptive option first. Always start with the smallest blast radius.
Step 4

Prepare four VCDX defense responses for vCenter recovery design: (1) 'Why not use SRM for vCenter protection?' — SRM requires vCenter running at protected and recovery sites, creating circular dependency; vCenter HA provides independent protection. (2) 'What happens to running VMs during vCenter failover?' — zero impact, VMs continue on ESXi hosts, only management operations are paused. (3) 'How do you handle vCenter HA in a stretched cluster?' — place active and passive on different sites, witness on third site, same as vSAN witness placement strategy. (4) 'What is your vCenter backup strategy for VCF?' — daily file-based backup to external SFTP, coordinated with SDDC Manager and NSX Manager backups, tested quarterly.

Four defense responses documented with clear technical reasoning. Each response demonstrates understanding of VCF architecture, management vs. data plane separation, and operational maturity.
Practice delivering each response in under 60 seconds. VCDX panelists value concise, confident answers that demonstrate deep understanding without rambling.

Validation Gate

Check: Complete vCenter HA architecture documentation and VCDX defense preparation

Expected: vCenter HA architecture documented with network requirements. Recovery decision tree created as runbook. Four VCDX defense responses prepared and practiced.

Common Errors

Confusing vCenter HA with vSphere HA — different technologies protecting different layers
Not accounting for SDDC Manager state during vCenter HA failover — may require SDDC Manager restart
Placing both active and passive vCenter HA nodes on same host — defeats the purpose
Forgetting that vCenter HA requires a dedicated HA network — reusing management network causes split-brain risk

Final Validation

Complete vCenter service recovery with graduated response procedures and VCDX defense preparation

✓ Service dependency chain documented → All 20+ services mapped with startup order and dependencies

✓ Log analysis capability demonstrated → Critical log locations known, failure patterns identified

✓ Recovery procedures tested → Service restart, database health check, and backup/restore completed

✓ vCenter HA architecture understood → Active/passive/witness documented with VCF management domain context

✓ VCDX defense responses prepared → Four defense scenarios documented with concise technical responses

Cleanup / Restore

• Verify all vCenter services are running: service-control --status

• Confirm SDDC Manager shows healthy vCenter connectivity

• Remove any temporary support bundles to reclaim disk space

• Revert to snapshot if any configuration changes were made during practice

Design Reflection (VCDX)

vCenter recovery design demonstrates understanding of management plane vs. data plane separation, graduated response methodology, and the critical role of vCenter in VCF lifecycle management. The design decision to deploy vCenter HA with coordinated backup strategy shows operational maturity expected at VCDX level.

Requirements

  • R-001: vCenter availability >=99.99% (<=52 min/year downtime)
  • R-002: vCenter RPO = 0 for automated failover, RPO <= 24h for backup-based recovery
  • R-003: VCF management domain operational continuity during component failures

Constraints

  • VCSA must run on management domain — cannot be hosted externally
  • VCF lifecycle operations (upgrades, expansion) require vCenter availability
  • SDDC Manager, NSX Manager, and Aria Suite all depend on vCenter for inventory

Assumptions

  • Dedicated HA network available between management domain hosts
  • External SFTP/NFS target available for file-based backups
  • Operations team trained on graduated recovery procedures

Risks

  • vCenter HA failover during active SDDC Manager workflow may leave workflow in failed state — requires manual retry
  • STS certificate expiry causes cascading authentication failure across all VCF components — monitor proactively
  • Database corruption from disk exhaustion requires full restore — implement proactive disk monitoring

Self-Assessment Discussion Prompts

  1. How would you design vCenter recovery differently for a VCF environment with 500+ hosts across 5 workload domains?
  2. What metrics would you monitor to predict vCenter failures before they cause outages?
  3. How does vCenter recovery priority change during a VCF upgrade window?
  4. If you could only implement one preventive measure for vCenter availability, what would it be and why?

Extensions

vCenter HA Failover Testing

Deploy vCenter HA in your Holodeck lab. Simulate active node failure by powering off the active VCSA VM. Measure actual failover time and verify all VCF management services reconnect. Document the failover timeline and any service recovery steps required post-failover.

Automated vCenter Health Monitoring

Build a PowerCLI or Python script that checks vCenter service status, disk space, certificate expiry, and database size on a schedule. Output a health report and send alerts when thresholds are breached. Integrate with Aria Operations for centralized monitoring.

Multi-vCenter Recovery Coordination

Design a recovery runbook for a VCF environment with management domain vCenter and 3 workload domain vCenters. Document the recovery sequence, inter-vCenter dependencies, and SDDC Manager coordination required. Practice with Holodeck multi-domain deployment.

⚠ Known Pitfalls (from Community KB)

Attempting full service restart when only one service is failed — causes unnecessary downtime to all vCenter consumers including SDDC Manager and NSX Manager
Not preserving support bundle and logs before attempting recovery — critical evidence for root cause analysis and VMware SR escalation is lost
Ignoring STS certificate expiry monitoring — certificates expire silently and cause cascading authentication failures across all SSO-integrated VCF components
Restoring VCSA backup without coordinating with SDDC Manager — creates inventory mismatch where SDDC Manager references a vCenter state that does not match the restored state

References

  • vCenter Server 9.0 Administration Guide — Service Management and Troubleshooting chapter
  • VMware KB 2147144 — How to stop, start, or restart vCenter Server services
  • VMware VCF 9.0 Administration Guide — Management Domain Backup and Recovery
  • VMware KB 2149253 — vCenter Server Appliance file-based backup and restore
  • vCenter HA Architecture and Deployment Guide
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.