Troubleshooting Holodeck Deployment Failures
Objectives
- Locate and read the Holodeck runtime on the HoloRouter: /holodeck-runtime (templates, config, bin, logs, specs), the per-instance output.json read by Get-HoloDeckInstance, the deployment state file behind idempotency, and the -LogLevel switch
- Diagnose HoloRouter-side faults with the network cmdlets (Get-HoloDeckSubnet, Get-HoloDeckAppNetwork, Get-HoloDeckBGPConfig, Get-HoloDeckDNSConfig, Get-HolodeckServiceIPPools) and the Set-HoloRouter / Reset-HoloRouter pair
- Diagnose nested ESX build failures from the documented pre-check errors (binary not found, no trunk port group, NTP not running) and the SSH-not-responding wait
- Diagnose VCF bring-up failures in the VCF Installer UI and its domainmanager.log, and in the VCF Operations fleet management log
- Build a personal runbook that maps each symptom to its cause, its documented fix, and the resume action (re-run, Reset-HoloDeckState, Reset-HoloRouter + Set-HoloRouter, or Remove-HoloDeckInstance)
Prerequisites
holodeck-02 complete (or at least attempted): a HoloRouter with a configuration and a deployed or partially deployed instance; outer-layer snapshots available.
Prior labs: holodeck-02
Required skills:
- Linux shell basics on Photon OS (ls, tail, grep, journalctl, systemctl)
- PowerShell cmdlet discovery (Get-Help, Get-Command)
- VCF Installer and NSX Manager navigation
Lab Environment
Same as holodeck-02: HoloRouter plus the nested management domain (or a failed partial deployment). All troubleshooting is done from an SSH session on the HoloRouter (bash and pwsh), the VCF Installer UI and appliance shell, and the outer vSphere layer.
Mermaid source
graph LR SSH["SSH root@holorouter"] --> BASH["bash: /holodeck-runtime/logs, journalctl, systemctl"] SSH --> PWSH["pwsh: Get-HoloDeck* cmdlets, Reset-HoloDeckState, Set-/Reset-HoloRouter"] VI["VCF Installer appliance"] --> LOG["/var/log/vmware/vcf/domainmanager/domainmanager.log"] OUTER["Outer vSphere layer"] --> SNAP["nested VM consoles, power state, snapshots"]
IP Addressing
| Network | Purpose | VLAN |
|---|---|---|
10.1.0.0/20 | Default Holodeck Site A master CIDR, subdivided by New-HoloDeckNetworkConfig into the Site A subnets (management, vMotion, vSAN, host overlay, edge uplinks, and others) | VLANs 0 and 10-25 (Site A default) |
10.2.0.0/20 | Default Holodeck Site B master CIDR (dual-site only) | VLANs 40-58 (Site B default) |
<external management network> | HoloRouter external management IP (DHCP or static, set at OVA deployment); the only external IP a Holodeck environment needs | External port group |
Credentials
| System | Username | Password |
|---|---|---|
| HoloRouter (SSH and pwsh) | root | Set in the OVA Additional settings at deployment |
| HoloRouter webtop | (no authentication in 9.0.2) | http://<holorouter-ip>:30000; the docs warn not to expose it externally |
| VCF Installer | admin@local | Holodeck documented default (Post Deployment table) |
| VCF Operations | admin | Holodeck documented default (Post Deployment table) |
| Management vCenter | administrator@vsphere.local | Holodeck documented default (Post Deployment table) |
| SDDC Manager | administrator@vsphere.local | Holodeck documented default (Post Deployment table) |
| Management NSX | admin | Holodeck documented default (Post Deployment table) |
| Nested ESX hosts | root | Holodeck documented default (Post Deployment table) |
Tasks
Task 1 Know the runtime, the logs and the state
ManageabilityEverything Holodeck knows about your deployment lives under /holodeck-runtime on the HoloRouter. Reading it in the right order turns a wall of output into a diagnosis.
SSH to the HoloRouter and list the runtime: ls -la /holodeck-runtime/ and its folders: templates (the config.json template), config (<config-ID>.json per deployment), bin/<version>/ (staged binaries), logs, specs (JSON API specs written by -ProvisionOnly).
List the logs: ls -lt /holodeck-runtime/logs/ and tail the most recent deployment log with tail -n 200. Identify the phase markers you saw in holodeck-02 (pre-checks, networking setup, nested ESX build, VCF Installer, bundles, bring-up).
Re-run a deployment with -LogLevel DEBUG (valid values INFO, DEBUG, SUCCESS, WARN, ERROR; default INFO) when you need more detail; record how much louder DEBUG is and when you would use WARN or ERROR to filter.
In pwsh (config imported) run Get-HoloDeckInstance -InstanceID holo: it reads the instance output.json and returns deployed components, networking and deployment state. Find the output.json under /holodeck-runtime and compare.
Understand idempotency: Holodeck stores the deployment state on the HoloRouter; re-running the same New-HoloDeckInstance command resumes at the failed step. Reset-HoloDeckState clears that state (irreversible) for a clean restart when the state is corrupted. Do not run it now; note where it fits in your runbook.
Read the FAQ note on handled errors: messages during deployment that are not in RED text and do not exit the script are handled. Mark in your log excerpt which lines were handled and which (if any) were fatal.
Open the Holodeck release notes (Known Issues per release) and the GitHub issues page; note the three known issues most relevant to your environment and their workarounds. These are the first places the FAQ sends you for version-specific bugs.
Validation Gate
Check: Runtime map, newest log and phases, LogLevel behaviour, output.json vs Get-HoloDeckInstance, resume/reset rule, handled vs fatal classification, known issues noted.
Expected: You can read a failed run from the HoloRouter without guessing.
Common Errors
Task 2 Diagnose HoloRouter faults: routing, DNS, DHCP, BGP, services
AvailabilityThe HoloRouter provides DNS, DHCP, NTP, L3 routing, firewall, BGP and proxy for the nested estate. When any of them is wrong, every later phase fails in confusing ways. The cmdlets show you what the network manager generated; Set-/Reset-HoloRouter rebuild it.
In pwsh list what the network manager generated for Site A: Get-HoloDeckSubnet -Site a | ft -AutoSize (names such as mgmt-a, vmotion-a with VLAN IDs, subnets and gateways), Get-HoloDeckAppNetwork -Site a (hostname, IP, FQDN per appliance), Get-HoloDeckBGPConfig -Site a (router AS, edge AS numbers, BGP password), Get-HoloDeckOverlaySubnet -Site a, Get-HoloDeckAppIpPools -Site a and Get-HolodeckServiceIPPools -Site a.
Check name resolution on the HoloRouter: Get-HoloDeckDNSConfig (all records; the cmdlet queries the CoreDNS ConfigMap) and Get-HoloDeckDNSConfig -FQDN esx-01a.site-a.vcf.lab. From bash, resolve a nested FQDN against the HoloRouter itself (for example with nslookup or dig if present) and from your console using the HoloRouter external IP as DNS.
Check DHCP and routing: from a nested host console or the webtop, confirm the host received the expected management IP and that the gateway for its subnet is the HoloRouter interface listed by Get-HoloDeckSubnet. Trace a path from the console to a nested IP through the HoloRouter external IP.
Check BGP if an edge cluster exists: compare the Tier-0 BGP neighbour state in NSX with Get-HoloDeckBGPConfig; the HoloRouter is the peer (FRR). From bash, inspect the FRR state (vtysh or the FRR service logs) if available on the appliance.
Inspect the HoloRouter's services: the documentation lists DNS, DHCP, NTP, routing, firewall, BGP, proxy, job scheduling and webtop; the command reference notes DNSMASQ for DNS services and FRR for BGP, and that Reset-HoloRouter needs permissions to modify Kubernetes resources. From bash run kubectl get pods -A and systemctl --failed: since Holodeck 9.0.2 DNSMASQ runs in Kubernetes on the HoloRouter (3 DNS pods + 1 DHCP pod, Holodeck official docs), so expect those pods; if kubectl reports localhost:8080 refused, rerun with KUBECONFIG=/etc/kubernetes/admin.conf (github.com/vmware/Holodeck issue #127). Expect only HoloRouter infrastructure services here; SDDC Manager, vCenter and NSX are VMs on the nested hosts, never pods on the HoloRouter.
Check the proxy if the nested estate reaches the internet through the HoloRouter: confirm the proxy settings given during deployment and that the VCF Installer can reach the depot. Check NTP on the HoloRouter and that nested hosts sync from it.
Rebuild if needed: Reset-HoloRouter removes DNSMASQ, FRR, VLANs, VRF tables and custom routing and restores the original DNS configuration (it disables DNS and routing for the environment until Set-HoloRouter runs again); Set-HoloRouter [-dualsite] [-CIDR <array>] reconfigures it. Remove-HoloDeckInstance -ResetHoloRouter removes the instance networking and reboots the HoloRouter; New-HoloDeckInstance configures networking again.
Write the HoloRouter section of your runbook: symptom, cmdlet to confirm, fix, impact.
Validation Gate
Check: Network inventory, DNS, DHCP/routing, BGP, services, proxy/NTP checked with the documented cmdlets; reset/reconfigure sequence recorded.
Expected: HoloRouter faults can be confirmed and repaired without guesswork.
Common Errors
Task 3 Diagnose nested ESX provisioning failures
RecoverabilityThe nested host build has a short list of documented failure modes; each has a documented fix in the FAQ.
Pre-check: '<Component> binary was not found. Download the binary from the Broadcom support portal and place it in /holodeck-runtime/bin/<version> folder'. Confirm with ls -lh /holodeck-runtime/bin/9.0.2.0/ that both the ESX ISO and the VCF Installer OVA are present and intact.
Pre-check: '[ERROR] No trunk port groups found on the connected server. Please create a trunk port group on the Target Host and run the pre-checks script again.' Verify on the target a port group with VLAN 4095 (or multiple VLANs), Promiscuous Mode, MAC Address Changes and Forged Transmits set to Accept; on a vDS do the same on the distributed port group.
Pre-check: '[ERROR] NTP service is not running on target host <IP>. Enable NTP service before proceeding'. On the ESX host: Manage > System > Time & Date > Edit Settings > NTP with a server, then Start; or via SSH: esxcli network firewall ruleset set -e true -r ntp; esxcli system ntp set -s <server>; esxcli system ntp set -e true; esxcli system ntp start.
esxcli network firewall ruleset set -e true -r ntp
esxcli system ntp set -s pool.ntp.org
esxcli system ntp set -e true
esxcli system ntp startBuild phase: 'WARNING: Host key is not being verified since Force switch is used.' and 'CustomIso[...]: [INFO] SSH service on <IP> is not responding, Trying after 10 seconds'. This is expected while the nested host boots; wait 10-15 minutes. Beyond 15 minutes: check the nested ESX VM is powered on and its console shows the DCUI; confirm networking (trunk security settings, VLANs on a cluster target); try ssh root@<nested-ip> yourself.
Resource symptoms: nested hosts crash, deployment is slow, or vSAN ESA datastore appears fully consumed. Check the FAQ causes: memory tiering enabled on the physical host (not recommended for nested), nested ESA on non-vSAN physical storage, HDD-backed storage, or a host below the requirement table.
Datastore selection problems: if the deployment cannot place VMs, confirm the datastore chosen at the interactive prompt (or $env:datastore_name in -DeveloperMode) has the capacity from the requirement table and is SSD/NVMe backed.
After fixing any of the above, re-run the same New-HoloDeckInstance command and confirm it resumes at the nested ESX phase rather than starting over.
Add each symptom, its documented fix and the resume action to the runbook.
Validation Gate
Check: Three pre-check errors and the SSH wait reproduced or confirmed with their fixes; resource causes mapped; resume verified.
Expected: Host provisioning failures are diagnosed from documented messages.
Common Errors
Task 4 Diagnose VCF bring-up failures in the VCF Installer
RecoverabilityOnce the nested hosts are up, the VCF Installer drives the bring-up. Its UI validation report and the domain manager log on the installer appliance are the documented places to look.
Open https://vcfinstaller-a.site-a.vcf.lab (admin@local). Review the deployment validation report: every failed or warning item names the component and the check.
Bundle download problems: confirm the depot configuration in the installer (online with the Broadcom token through the proxy if configured; or the offline depot IP/port/protocol/credentials). For an offline depot confirm the manifest is the latest from the Broadcom Support Portal.
SSH to the VCF Installer appliance and tail the domain manager log: tail -f /var/log/vmware/vcf/domainmanager/domainmanager.log (VCF 9.0.1 documentation, Log Files and Locations: 'If you experience validation or deployment failures, check the log files for the domain manager service on the VCF Installer appliance').
tail -f /var/log/vmware/vcf/domainmanager/domainmanager.logIf the failure involves VCF Operations or VCF Automation deployment, check /var/log/vrlcm/vmware_vrlcm.log on the VCF Operations fleet management appliance (same documentation page).
Check the nested estate's prerequisites the installer validates: DNS forward and reverse for every appliance (Get-HoloDeckDNSConfig), NTP, and host state (all four nested hosts reachable with vSAN-eligible disks). Fix on the HoloRouter side if the network manager output is wrong.
Retry: in the installer UI retry the failed task after fixing the cause; or re-run the New-HoloDeckInstance command, which resumes and drives the installer again. Record which worked.
Write the bring-up section of the runbook: validation item, log line pattern, cause, fix, retry path.
Validation Gate
Check: Installer validation report, depot configuration, domainmanager.log and vmware_vrlcm.log used; retry path exercised.
Expected: Bring-up failures are diagnosed from the documented logs.
Common Errors
Task 5 Build the troubleshooting runbook
ManageabilityA VCDX candidate shows a repeatable method: symptom, confirming evidence, documented cause, fix, resume action, and the time it costs.
Consolidate the four sections (runtime/logs, HoloRouter, host build, bring-up) into one table with columns: Phase, Symptom (exact message), Where to confirm (cmdlet, log path, UI), Documented cause, Fix, Resume action (re-run / Reset-HoloDeckState / Reset-HoloRouter + Set-HoloRouter / Remove-HoloDeckInstance [-ResetHoloRouter] + redeploy), Time cost.
Add the escalation path: Holodeck release notes Known Issues, GitHub issues (vmware/Holodeck), and the Broadcom VCF community; note that Holodeck is not a VMware supported product (similar to a Fling) so there is no support case for the toolkit itself.
Decide the destructive actions and their guard rails: Reset-HoloDeckState (irreversible), Reset-HoloRouter (removes DNS and routing until Set-HoloRouter), Remove-HoloDeckInstance (deletes the nested hosts and installer VM; -ResetHoloRouter also removes networking and reboots the HoloRouter). Require a snapshot check before each.
Test the runbook once: induce a safe failure (for example stop NTP on the target host or rename the trunk port group), run the deployment pre-checks, follow your own table to the fix, and resume.
Record the lessons as design notes: which failures are design inputs in production (DNS, NTP, trunking, depot access) and which are lab-only.
Revert the induced failure and confirm the environment is healthy (holodeck-02 Tasks 4-5 checks).
Validation Gate
Check: Runbook table complete with resume actions and guard rails; one row tested; design notes written; environment healthy.
Expected: A reusable troubleshooting method for Holodeck 9.0.2 deployments.
Common Errors
Final Validation
The Holodeck runtime, logs and state are understood; HoloRouter, nested host build and VCF bring-up failures are diagnosed with the documented cmdlets, messages and log paths; a tested runbook with resume actions and guard rails exists.
✓ Runtime and logs → /holodeck-runtime map; newest log phases; output.json vs Get-HoloDeckInstance; LogLevel
✓ HoloRouter → Subnet/AppNetwork/BGP/DNS/IP-pool inventories; services; Reset/Set sequence
✓ Host build → Three pre-check errors and SSH wait mapped to fixes; resume verified
✓ Bring-up → Installer validation report, domainmanager.log, vmware_vrlcm.log used
✓ Runbook → 12+ rows, escalation path, guard rails, one row tested
Cleanup / Restore
Snapshot: holodeck-03-complete
• Revert any induced failure
• Keep the runbook with the lab notes
• Confirm the instance is healthy before holodeck-04
Design Reflection (VCDX)
Panels probe method, not memory: when a deployment fails, where do you look first and why, how do you distinguish an infrastructure-service fault (DNS, NTP, routing) from a product fault, and what is your blast radius for each recovery action. Holodeck gives you a compact model: one appliance of infrastructure services, one installer, one state file, one resume rule. Translate it into production language: dependencies, idempotent automation, documented log locations, and guarded destructive operations.
Requirements
- Every failure is diagnosed from documented messages and log paths
- Resume via idempotent re-run before any reset
- Destructive cmdlets guarded by snapshots
Constraints
- Holodeck is not a VMware supported product
- Reset-HoloDeckState is irreversible
- Reset-HoloRouter removes DNS and routing until Set-HoloRouter
- Remove-HoloDeckInstance -ResetHoloRouter reboots the HoloRouter
Assumptions
- SSH access to the HoloRouter and the installer appliance
- Outer-layer snapshots exist
- Console DNS can point at the HoloRouter
- Release notes and GitHub issues reachable
Risks
- Reset instead of resume — Impact: hours lost — Mitigation: runbook rule
- Reset-HoloRouter during a healthy deployment — Impact: estate loses DNS/routing — Mitigation: guard rail
- Chasing Cloud Builder in 9.x — Impact: time lost — Mitigation: installer-first runbook
- Unreported known issue — Impact: repeated failure — Mitigation: release notes check first
Self-Assessment Discussion Prompts
- Why does Holodeck put DNS, NTP, routing and BGP on one appliance, and what does that do to your fault-isolation method?
- How does an idempotent deployment change the recovery runbook compared with a restart-from-scratch tool?
- Which of the documented pre-check errors would be change-management failures in production?
- What is the blast radius of Reset-HoloRouter, and how would you express a similar control for a production DNS change?
- Where do you look first when the installer's validation fails, and why that log?
- What would you tell a panel about using an unsupported toolkit to prepare a supported-platform design?
Extensions
Developer-mode failure injection
Use -DeveloperMode environment variables to script a deployment, inject a wrong datastore_name, and document the failure and resume path.
sameProvision-only diagnosis
Run -ProvisionOnly and perform the VCF Installer bring-up by hand using /holodeck-runtime/specs/, diagnosing each validation failure in domainmanager.log.
harderRelease-notes watch
Set up a weekly check of the Holodeck release notes Known Issues and GitHub issues and fold new items into the runbook.
same⚠ Known Pitfalls
References
- Holodeck 9.0.2 documentation (index): Environment Overview, Holorouter Overview, Concepts, Pre-requisites, Deployment, During Deployment, Post Deployment, Day 2 Operations, How ToTier 1 — Official
Primary source for every HoloRouter and toolkit step in this lab. Fetched 1 Oct 2026. - Holodeck 9.0.2 command referenceTier 1 — Official
New-HoloDeckInstance, Update-HoloDeckInstance, Get-HoloDeckInstance, New/Get/Import-HoloDeckConfig, Reset-HoloDeckState, New-HoloDeckNetworkConfig, Set-/Reset-HoloRouter, Get-HoloDeckSubnet/AppNetwork/BGPConfig/DNSConfig/OverlaySubnet/AppIpPools, Get-HolodeckServiceIPPools. - Holodeck 9.0.2 FAQ and troubleshootingTier 1 — Official
Pre-check errors (binary not found, no trunk port group, NTP not running), ESX build SSH wait, memory tiering, nested vSAN ESA. - Holodeck 9.0.2 downloads and release notes (known issues)Tier 1 — Official
HoloRouter OVA and Offline Depot OVA; release notes at /release_notes/ carry the Known Issues per release. - Holodeck GitHub repository: issues and discussionsTier 2 — Supporting source
Where to report bugs; community troubleshooting. The releases page carries no binaries (OVA downloads are on the Broadcom Support Portal). - VMware Cloud Foundation 9.0 documentation: Log Files and Locations (VCF Installer domainmanager.log; fleet management vmware_vrlcm.log)Tier 1 — Official
VCF 9.0.1 PDF p.1298. Task 4.