Lab: SoS Diagnostics & VCF API Troubleshooting
Objectives
- Execute SoS health checks and interpret results
- Collect targeted log bundles for support cases
- Use SDDC Manager REST API for diagnostic queries
- Analyze failed lifecycle tasks via API subtask inspection
- Build a troubleshooting decision tree for common VCF issues
Prerequisites
VCF 9.0 Instance operational, SSH access to SDDC Manager, REST API client (curl, Postman, or similar)
Required skills:
- Linux CLI basics
- REST API concepts
- Log analysis
Lab Environment
VCF 9.0 with SDDC Manager accessible via SSH and HTTPS. At least one completed lifecycle task (domain creation, host addition) in task history for API analysis.
Tasks
Task 1 SoS Diagnostics & API-Driven Troubleshooting
Master VCF's built-in diagnostic tools: SoS for health checks and log collection, SDDC Manager REST API for programmatic diagnostics, and log correlation for root cause analysis — the primary troubleshooting toolkit for VCAP-level VCF administration.
Run SoS health check. SSH to SDDC Manager VM: sudo /opt/vmware/sddc-support/sos --health-check --domain-name <management-domain-name> (or --domain-name ALL for every workload domain; add --short to list only failures and warnings, --force while workflows are running). Health-check options are designed to run on the SDDC Manager appliance. The health check examines: DNS resolution (forward + reverse for all components), NTP synchronization (whether component time is synchronized with the NTP server; sos --ntp-health, VCF 5.2 docs p.609), certificate validity (all certs checked), service status (all VCF services running), component connectivity (SDDC Mgr ↔ vCenter ↔ NSX ↔ ESXi), password validity (credential store vs actual), disk space (all components), vSAN health. Output: written to /var/log/vmware/vcf/sddc-support on the SDDC Manager appliance unless you pass --log-dir (VCF 5.2 docs p.611); review the health-check results there.
Interpret SoS results. Review the health-check output from step 1 (console summary and the files under /var/log/vmware/vcf/sddc-support or your --log-dir; add --json-output-dir <dir> if you want the results as JSON). Each check shows GREEN (normal), YELLOW (warning, attention might be required) or RED (critical, needs immediate attention) (VCF 5.2 docs pp.608-609). Common findings: (a) time not synchronized with the NTP server on specific hosts (ESX Time not GREEN) — time drift can break certificate validation; (b) DNS reverse lookup failure — blocks future lifecycle operations; (c) certificate expiring in 15-30 days (YELLOW: schedule rotation) or in less than 15 days (RED: rotate now) — SoS certificate-health bands, VCF 5.2 docs p.609; (d) disk space low on SDDC Manager — log rotation needed. Document all RED and YELLOW findings as an action list.
Collect targeted log bundle. For a support case, collect logs for one component only instead of everything: sudo /opt/vmware/sddc-support/sos --vc-logs --domain-name <domain> --log-dir /tmp/sos-vc --zip. Component selectors: --vc-logs (vCenter), --nsx-logs, --esx-logs, --sddc-manager-logs, --operations-logs, --operations-fleet-logs, --automation-logs, --api-logs, --vm-screenshots; --collect-all-logs is the everything bundle. A date-range filter is not among the documented SoS options — bundle size is controlled by the component flags, --no-health-check (skip the health check run as part of collection) and --log-dir/--log-folder. Bundle location: the --log-dir you gave (default under /var/log/vmware/vcf/sddc-support/).
Authenticate to SDDC Manager API. Using curl: TOKEN=$(curl -s -k -X POST 'https://<sddc-mgr>/v1/tokens' -H 'Content-Type: application/json' -d '{"username":"admin@local","password":"<password>"}' | python3 -c 'import sys,json; print(json.load(sys.stdin)["accessToken"])'). Verify the token with a documented read call: curl -s -k -H "Authorization: Bearer $TOKEN" 'https://<sddc-mgr>/v1/domains' | python3 -m json.tool. This returns the workload domain inventory (POST /v1/tokens with admin@local: VCF 9.0.1 docs p.3012; GET /v1/domains: p.2844).
Find the failed lifecycle task. Note the task ID of the most recent failure from the SDDC Manager UI task panel (subtask errors are shown in the task panel, VCF 9.0.1 docs p.1454). Then query it: curl -s -k -H "Authorization: Bearer $TOKEN" 'https://<sddc-mgr>/v1/tasks/<task-id>' | python3 -m json.tool (GET /v1/tasks/{id}, p.2845). For a task list call and its filters, use the SDDC Manager API reference for your release on developer.broadcom.com; a filtered list call is not shown in the VCF 9.0.1 documentation.
Deep-dive into the failed subtask. In the GET /v1/tasks/{id} response from step 5, or in the SDDC Manager UI task panel, find the subtask that failed and read its error message — it identifies the exact failure point (p.1454 shows a subtask error displayed in the task panel). Example: 'NSX Transport Node Preparation' failed → look at NSX Manager logs for the specific host.
Correlate API findings with component logs. Based on the failed subtask, SSH to the relevant component and examine logs: (a) If vCenter-related: /var/log/vmware/vpxd/vpxd.log on vCenter; (b) If NSX-related: /var/log/proton/nsxapi.log on NSX Manager; (c) If ESXi-related: /var/log/hostd.log on the specific host; (d) If SDDC Manager: /var/log/vmware/vcf/domainmanager/domainmanager.log. Use the subtask timestamp to find relevant log entries: grep '<timestamp>' <logfile>.
Password validation via API. curl -s -k -H "Authorization: Bearer $TOKEN" 'https://<sddc-mgr>/v1/credentials' → lists all managed credentials with: component, username, credential type, last rotation date. For password-related issues: compare SoS password-check results with API credential inventory. If a password was changed outside VCF, the API shows the old password while the component has the new one — causing authentication failures in lifecycle operations.
Retry a failed task. After resolving the root cause, retry in the SDDC Manager UI: for a failed update, the update status shows Failed with the reason and remediation steps — resolve the issues and retry the update (VCF 9.0.1 docs p.1456); for a failed component upgrade, resolve the issue and retry the failed task (p.1388). For an API-driven retry, take the call and request body from the SDDC Manager API reference for your release on developer.broadcom.com; the retry call is not documented in the VCF 9.0.1 documentation. Monitor: poll GET /v1/tasks/{id} (VCF 9.0.1 docs p.2845) until the status changes to SUCCESSFUL or FAILED again.
Build troubleshooting decision tree. Document the diagnostic workflow as a decision tree: (1) Symptom observed → (2) Run SoS health-check → (3) Any RED findings? → Yes: remediate RED items first → (4) Check API task history for recent failures → (5) Inspect failed subtasks → (6) Correlate with component logs → (7) Resolve root cause → (8) Retry the failed task or update (SDDC Manager UI, or the API call from the API reference) → (9) Verify with SoS health-check post-resolution. This decision tree is a VCAP exam deliverable and a production operations artifact.
Validation Gate
Check: SoS and API diagnostic workflow mastered
Expected: SoS health-check executed and interpreted, log bundle collected with targeted filters, API authentication and queries working, failed task subtasks inspected and correlated with logs, task retry procedure validated
Common Errors
Final Validation
VCF troubleshooting toolkit mastered — SoS, API, and log correlation
✓ SoS health-check → Health-check results reviewed with every check evaluated (GREEN/YELLOW/RED)
✓ Targeted log bundle → Compressed bundle created with component filters and --log-dir (no date filter)
✓ API diagnostics → Token auth, task query, subtask inspection all functional
✓ Log correlation → Failed subtask matched to specific component log entries
✓ Task retry → Failed task or update retried after root cause resolution (SDDC Manager UI, or the API call from the SDDC Manager API reference)
Cleanup / Restore
• Clean up SoS output files from /var/log/vmware/vcf/sddc-support (or the --log-dir you used, for example /tmp/sos-vc)
• Close API session (tokens expire after 6 hours)
Design Reflection (VCDX)
Troubleshooting methodology demonstrates operational maturity. VCDX defense: show you have a systematic approach (SoS → API → logs) rather than random log searching. Discuss: how do you scale troubleshooting across a fleet with 50+ VCF Instances?
Requirements
- Sub-30-minute root cause identification for lifecycle failures
- Targeted log collection for efficient support case resolution
- Retry of the failed task without re-running the entire workflow
Constraints
- SoS health-check takes 10-30 minutes depending on environment size
- Log bundles can be 10+ GB without targeted filtering
- API token expires after 6 hours — long-running scripts need token refresh
Assumptions
- SSH access to SDDC Manager is available
- REST API client (curl) is available on the admin workstation
- Component logs have not been rotated or purged since the failure occurred
Risks
- SoS health-check during active lifecycle operation may show false positives
- Retrying a task without understanding root cause makes the situation worse
- Unfiltered log collection fills the SDDC Manager log partition (or the --log-dir target) and causes SDDC Manager disk space issues
Self-Assessment Discussion Prompts
- What information should be collected before opening a VMware support case?
- How do you troubleshoot a VCF lifecycle failure when SoS itself fails to run?
- When would you use SoS vs direct component log analysis?
Extensions
Script the full diagnostic workflow: SoS → API query → log extraction → email report
Build a monitoring dashboard that runs SoS health-check daily and trends results
Create a PowerShell module for common SDDC Manager API diagnostic queries
⚠ Known Pitfalls
References
- VCF 9.0 Troubleshooting Guide — SoS Utility: techdocs.broadcom.com
- SDDC Manager API Reference: developer.broadcom.com