Troubleshooting Holodeck Deployment Failures

Verify HoloRouter connectivity: ping 10.1.1.1 from HoloConsole PowerShell

SSH into the HoloRouter and check FRR status: ssh admin@10.1.1.1 -c 'systemctl status frr'

If FRR is not running, restart it: ssh admin@10.1.1.1 -c 'systemctl restart frr && sleep 5 && systemctl status frr'

Check FRR logs for errors: ssh admin@10.1.1.1 -c 'tail -50 /var/log/frr/frr.log'

Verify DHCP service on HoloRouter is operational (it assigns IPs to nested ESXi hosts): ssh admin@10.1.1.1 -c 'systemctl status dnsmasq'

Check DHCP lease assignments: ssh admin@10.1.1.1 -c 'cat /var/lib/misc/dnsmasq.leases'

Verify DNS resolution from a nested ESXi host: ssh root@10.1.1.101 -c 'nslookup sddc-manager.site-a.vcf.lab 10.1.1.1'

Create a HoloRouter health check script. In PowerShell on HoloConsole, save the following as check-holorouter.ps1:

$checks = @(
@{ Name='HoloRouter Ping'; Cmd='ping -c 1 10.1.1.1' },
@{ Name='FRR Status'; Cmd='ssh admin@10.1.1.1 systemctl status frr' },
@{ Name='DHCP Status'; Cmd='ssh admin@10.1.1.1 systemctl status dnsmasq' },
@{ Name='BGP Neighbors'; Cmd='ssh admin@10.1.1.1 vtysh -c "show ip bgp summary"' }
)

foreach ($check in $checks) {
Write-Host "\n[$($check.Name)]" -ForegroundColor Cyan
Invoke-Expression $check.Cmd
}

Check the physical ESXi host CPU capabilities: On physical host PowerShell, run: Get-VMHost | Select-Object -Property Name, @{N='CPUModel'; E={$_.ProcessorType}}, CpuCores

Check if legacy CPU support is needed: Review the Holodeck deployment logs for any mentions of 'legacy CPU', 'CPUID', or 'unknown processor'. Run: Select-String -Path '.\logs\*' -Pattern 'legacy|CPUID|unknown processor'

If legacy CPU support is required, check the nested ESXi boot.cfg. From a nested ESXi host, run: ssh root@10.1.1.101 -c 'cat /bootbank/boot.cfg | grep -i allowLegacyCPU'

If allowLegacyCPU is missing, add it. SSH to the nested ESXi host and edit boot.cfg: ssh root@10.1.1.101 -c 'echo "allowLegacyCPU=TRUE" >> /bootbank/boot.cfg && cat /bootbank/boot.cfg | grep allowLegacy'

Verify physical datastore capacity and accessibility. On physical ESXi, run: Get-Datastore | Select-Object -Property Name, @{N='FreeGB'; E={[Math]::Round($_.FreeSpaceGB, 1)}}, @{N='CapacityGB'; E={[Math]::Round($_.CapacityGB, 1)}}

Verify the datastore specified in config.json exists and is accessible. From HoloConsole, run: Get-Content .\templates\config.json | ConvertFrom-Json | Select-Object -Property targetDatastore | Format-List

Check nested ESXi VM disk provisioning. List the nested ESXi VM files on the datastore: ssh root@10.1.1.101 -c 'df -h /'

Create a pre-flight checklist script. In PowerShell on HoloConsole, save the following:

$checks = @(
@{ Check='Physical ESXi CPU'; Cmd='Get-VMHost | Select-Object ProcessorType' },
@{ Check='Datastore Free Space'; Cmd='Get-Datastore | Where-Object Name -eq (Get-Content .\templates\config.json | ConvertFrom-Json).targetDatastore | Select-Object Name, FreeSpaceGB' },
@{ Check='Nested ESXi allowLegacyCPU'; Cmd='ssh root@10.1.1.101 grep allowLegacyCPU /bootbank/boot.cfg' }
)

foreach ($check in $checks) {
Write-Host "[$($check.Check)]" -ForegroundColor Cyan
Invoke-Expression $check.Cmd
}

Monitor Cloud Builder UI during bring-up (or check logs post-failure). From HoloConsole Firefox, navigate to https://10.1.1.200 and log in with admin@local credentials. Watch the 'Bring-up Progress' dashboard.

Check Cloud Builder initialization logs: ssh admin@10.1.1.200 -c 'tail -100 /opt/vmware/vcf/domainmanager/logs/domainmanager.log | grep -i "error\|fail\|warn"'

Check NSX bundle version in the Cloud Builder logs. Search for 'NSX' and 'version': ssh admin@10.1.1.200 -c 'grep -i "nsx" /opt/vmware/vcf/domainmanager/logs/domainmanager.log | grep -i "version\|bundle"'

If NSX version mismatch is detected, verify the content manifest and NSX bundle. From HoloConsole, run: Get-Content .\templates\config.json | ConvertFrom-Json | Select-Object -Property contentManifest | Format-List

Check SDDC Manager service status and logs on the Cloud Builder or deployed SDDC Manager. Run: ssh root@10.1.1.4 -c 'systemctl status domainmanager && tail -50 /opt/vmware/vcf/sddc-manager/logs/domainmanager.log'

Verify DNS and NTP on Cloud Builder (critical for all VCF services). Run: ssh admin@10.1.1.200 -c 'echo "=== DNS ===" && cat /etc/resolv.conf && echo "=== NTP ===" && timedatectl status'

Create a VCF bring-up health check. In PowerShell, save the following:

Write-Host "VCF Bring-up Diagnostics" -ForegroundColor Green
Write-Host "\n[1] Cloud Builder Accessibility"
ping -c 1 10.1.1.200 | Select-Object -Last 1
Write-Host "\n[2] Cloud Builder Service Status"
ssh admin@10.1.1.200 'systemctl status vcf-bringup | grep Active'
Write-Host "\n[3] NSX Bundle Version"
ssh admin@10.1.1.200 'grep -i nsx /opt/vmware/vcf/domainmanager/logs/domainmanager.log | grep version | tail -1'
Write-Host "\n[4] SDDC Manager Status"
ssh root@10.1.1.4 'systemctl status domainmanager | grep Active'
Write-Host "\n[5] DNS and NTP"
ssh admin@10.1.1.200 'timedatectl status | grep -i synchronized'

Create a runbook template in JSON or YAML format (or a markdown table). The runbook should have columns: [Symptom, Where to Look First, Root Cause Check Commands, Likely Causes, Remediation Steps, KB Thread Reference]. Start by documenting 3 failures you've already analyzed (HoloRouter down, ESXi provisioning failure, NSX version mismatch).

Add three more failures to your runbook by reading the KB threads directly. For each of the following KB threads, extract: symptom, first diagnostic command, root cause, fix, and any KB reference:

• Thread 8: ESXi CPU compatibility (allowLegacyCPU)
• Thread 51: Datastore selection failure
• Thread 42: HoloRouter DHCP configuration

Format each as a runbook entry.

Create a decision tree flowchart (ASCII art or mermaid diagram) that shows the diagnostic flow: [Deployment fails] -> [Check phase] -> [Check component] -> [Read logs] -> [Identify root cause] -> [Apply fix]. Map each branch to a runbook entry.

Example pseudocode:
if phase == 'Prepare' then
check HoloRouter ping
if fails: HoloRouter down (see Runbook #1)
else: check FRR status
else if phase == 'Start' then
check Cloud Builder logs
if NSX mismatch: update manifest (see Runbook #5)
else if DNS fails: verify HoloRouter dnsmasq (see Runbook #3)

Test your runbook against a real failure (or a simulated one). Take the snapshot 'holodeck-02-complete' and intentionally introduce a failure:

Option A: Kill the HoloRouter FRR service: ssh admin@10.1.1.1 -c 'systemctl stop frr'
Option B: Delete the NSX bundle from the content directory (to trigger manifest mismatch)
Option C: Misconfigure the datastore name in config.json (to trigger 'Cannot select datastore' error)

Then follow your runbook to diagnose the failure end-to-end.

Document lessons learned. For each failure you diagnosed (real or simulated), record: (a) Time spent diagnosing, (b) Key insight that helped you find the root cause, (c) What you would do differently next time. Add this as a 'Lessons' section to your runbook.

Share your runbook with your study group or save it in your VCDX study materials folder. A well-written runbook is a VCDX panel gold mine — panelists will probe how you would troubleshoot, and your runbook is proof you've done it systematically.