Holodeck Lab Setup & Operations

What is Holodeck and Why It Matters for VCDX#

The VCF Holodeck Toolkit (branded 'VCF Lab Constructor' in Broadcom documentation) is an officially-supported automation framework that deploys a complete, nested VMware Cloud Foundation environment on a single physical ESXi host. It creates 4+ nested ESXi virtual machines, a HoloRouter for isolated networking, a HoloConsole for browser-based management access, and orchestrates the full VCF bring-up sequence (Cloud Builder → SDDC Manager → vCenter → NSX → optional VCF Automation).

For VCDX candidates, Holodeck is indispensable because:

1. Hands-on practice at scale: VCDX requires demonstrating deep operational experience with VCF. Holodeck provides a safe, repeatable environment for every deployment, upgrade, and troubleshooting scenario you'll face in the exam.

2. Design validation: Before defending a VCDX design, you can prototype it in Holodeck — test topology choices, failure scenarios, and scaling limits.

3. Failure injection: Holodeck deployments frequently encounter real-world failure modes (networking, resource, versioning) that mirror production issues. Learning to troubleshoot these builds the operational instinct panelists are looking for.

4. Version comparison: With Holodeck, you can run VCF 5.2.x and 9.0.x side-by-side on the same hardware, understanding the architectural evolution — a key VCDX discussion topic.

Key Takeaways

• Holodeck is your primary lab vehicle for all VCDX preparation — treat it as critical infrastructure
• Nested environments have real limitations (performance overhead, single fault domain) that you must articulate in VCDX discussions
• The Broadcom community forum (63 threads) plus internal GChat KB (180 threads, 30 entries) and GitHub issues are your troubleshooting lifeline

Holodeck Architecture Deep Dive#

Understanding Holodeck's architecture is essential because every component maps to a production VCF concept:

HoloRouter — A Photon OS VM running Kubernetes internally. All services run as K8s pods (NOT systemd): dnsmasq-dns (DNS with ~160 pre-provisioned records), dnsmasq-dhcp (DHCP for management and TEP VLANs), FRR (BGP routing with AS 65000 for router, AS 65001/65002 for edge domains), CoreDNS, and webtop (browser-based access). It provides L3 routing via 16+ VLAN subinterfaces (eth1.10 through eth1.25) between the nested environment and the physical host, with VRF 'sitea' isolating NSX Edge uplink VLANs.
The HoloRouter IP is 10.1.1.1 on the management VLAN (external access via the physical host network IP, e.g., 192.168.77.20). In production, this function is served by physical ToR switches and datacenter core routers. The HoloRouter is the most critical single point of failure in the nested environment. IMPORTANT: Use kubectl commands (kubectl get pods, kubectl exec, kubectl logs) to manage services — systemctl commands will NOT work.

HoloConsole / Webtop — Browser-based desktop access to the nested environment, running as a K8s pod (webtop) on the HoloRouter. Accessible at the HoloRouter's external IP (e.g., 192.168.77.20). In production, this maps to a jump host or bastion server in the management network.

Nested ESXi Hosts — VMs running ESXi, configured with virtual NVMe disks (for vSAN), multiple VMkernel adapters (management, vMotion, vSAN, TEP), and nested virtualization enabled. These behave identically to physical hosts from VCF's perspective, with the exception of performance characteristics.

Cloud Builder — The VCF orchestrator that runs the bring-up workflow. In both nested and production, Cloud Builder is a temporary appliance that deploys SDDC Manager, vCenter, and NSX, then hands control to SDDC Manager. After bring-up, Cloud Builder can be powered off.

Content Packs / Manifests — Version-specific bundles containing ESXi ISOs, vCenter OVAs, NSX bundles, and SDDC Manager OVAs. The manifest file tells Holodeck which content to use. Mismatched manifests are the single most common cause of deployment failure (documented in 5+ community threads).

graph TD
  subgraph Physical Host
    ESXi[ESXi 8.0 U3]
    DS[(Datastore 2TB+)]
    PG[Port Groups VLAN 1644-1648]
  end
  subgraph Holodeck Pod
    HR[HoloRouter fa:fa-network-wired]
    HC[HoloConsole fa:fa-desktop]
    N1[Nested ESXi-01]
    N2[Nested ESXi-02]
    N3[Nested ESXi-03]
    N4[Nested ESXi-04]
    CB[Cloud Builder]
  end
  subgraph Management Domain VMs
    SDDC[SDDC Manager]
    VC[vCenter Server]
    NSX[NSX Manager]
  end
  ESXi --> HR
  ESXi --> HC
  ESXi --> N1 & N2 & N3 & N4
  HR --> CB
  CB --> SDDC --> VC --> NSX
  N1 & N2 & N3 & N4 --> |vSAN Cluster| DS

Version Comparison

Key Takeaways

• Every Holodeck component has a production equivalent — learn to translate between lab and production contexts
• The HoloRouter is the single point of failure; know how to troubleshoot FRR, DNS, and DHCP on it
• Content manifest version alignment is non-negotiable — mismatches cause the majority of deployment failures

Community Knowledge Base — Top Deployment Issues#

The Holodeck community knowledge base draws from two complementary sources:

Source 1: Broadcom Public Forums — 63 threads, 247 posts. These are publicly searchable and tend toward initial setup/deployment questions.

Source 2: Holodeck Field Internal GChat — ~180 threads, 843 messages (Nov 2025 – Apr 2026). This internal Broadcom space includes direct responses from the Holodeck dev team (Dhruv Tyagi, Nikhil M Kulkarni, Jatin Purohit) and captures deeper troubleshooting, workarounds, and architectural guidance not available in public forums.

Combined analysis reveals:

80% of threads relate to Deploy / Bring-up phase — getting the management domain up is where most users struggle. Once deployed, the environment is stable.

Top failure categories (forum + GChat combined):

• Datastore selection failures (physical host storage configuration)
• Content manifest / version mismatches (VCF version vs NSX bundle)
• HoloRouter configuration failures (DHCP, FRR, port group VLAN trunking, adapter B/A swap)
• Offline depot authentication bugs (disable auth on HTTP, manual VCF Installer config)
• DNS recursive loops (missing local=/vcf.lab/ directive in dnsmasq)

- ESXi host resource sizing (config.json changes not propagating, VCFA 24→32 CPU)

• Legacy CPU compatibility (boot.cfg allowLegacyCPU flag)
• Certificate failures (SDDC Manager VMCA /etc/hosts fix, NSX signing key expiry KBs)
• State file manipulation for skipping/retrying failed deployment steps
• One-config-per-deployment rule and ProvisionOnly stickiness

25 of 63 forum threads have confirmed resolutions. 28 of 30 GChat entries are resolved — these are captured in the lab's Common Errors sections and linked back to the original sources.

The full community KB is available in the Academy's reference data: holodeck-kb-normalized.json, holodeck-kb-playbook.md (forum + GChat combined), holodeck-kb-issue-index.md, and holodeck-gchat-kb.md (detailed GChat entries with commands and file paths).

Key Takeaways

• Pre-validate config.json, content manifest, and physical host resources BEFORE running Prepare — this eliminates 70%+ of deployment failures
• If stuck, check the community playbook first — between forum and GChat sources, there's a 60%+ chance your exact issue is documented with a resolution
• The internal GChat KB (holodeck-gchat-kb.md) has deeper troubleshooting detail including exact commands, file paths, and step-by-step fixes from the Holodeck dev team
• Keep the community forum bookmarked — it's updated regularly with new issue reports and solutions

Holodeck 9.1 Architecture and Support Matrix (9.1)#

Holodeck 9.1 (docs dated May 2026; GA blog 2026-07-01) is a major toolkit release aligned to the VCF 9.1 GA train, and it resets the baseline assumptions an architect makes about the nested lab platform.

Deployment targets and support matrix. The 9.1 toolkit deploys VCF 9.1.0.0, VVF 9.1.0.0, and the VCF 5.2.4 maintenance line (official docs additionally list 5.2.3, which the community announcement omitted). The full supported deployment range spans VCF 9.1.0.0 down to VCF 5.2, and the minimum nested ESX version is 8.0 U3 — a hard compliance floor that rules out older nested templates, a real concern on shared hosts that carry legacy artifacts. VCF Installer pairing is version-sensitive: Installer 9.1.0.0 is supported for 9.1 deployments, Installer 9.0.2.0 is recommended for the 9.0.x line, and Installer 9.0.0.0 is no longer supported at all. Two product lines continue to coexist: Holodeck 5.2x (VCF 5.2.x only) and Holodeck 9 (5.2.x plus 9.x).

Resource envelope. Requirements step up substantially versus the 9.0 baseline (~30 cores / ~325 GB / ~1.1 TB typical). Published VCF 9.1 figures: 48 CPU cores, 650 GB RAM, 2.2 TB disk for single site, doubling to 96 cores / 1.5 TB / 5 TB for dual site; VVF 9.1 needs 24 cores / 512 GB / 2 TB. New in 9.1, soft validation lets pre-checks warn rather than block when resources fall below recommendation — operationally convenient on constrained shared hosts, but the architect explicitly owns the resulting performance risk. The HoloRouter appliance itself grows to 6 vCPU / 12 GB (from 4 vCPU / 8 GB) to carry its expanded services stack, and no in-place 9.0-to-9.1 router upgrade is documented — plan a clean 9.1 HoloRouter deployment until Broadcom says otherwise.

Design significance. The wide version range turns upgrade-path rehearsal into a first-class lab scenario: one toolkit can stand up a 5.2.4 instance beside a 9.1.0.0 instance and make the 5.2-to-9.x architectural evolution (Cloud Builder vs VCF Installer, NSX 4.x vs 9.x-aligned versioning, licensing and lifecycle changes) a demonstrable exercise rather than a whiteboard claim. What did not change matters equally: a single physical host remains a single fault domain, default networking carries forward (Site A 10.1.0.0/20, Site B 10.2.0.0/20, /20 MasterCIDR rule), and the one-config-one-deployment discipline persists. The 9.0.x guidance in this section remains authoritative for 9.0.x instances; the holodeck91-* lab set is its parallel 9.1 twin, to be enriched during the live W5 deployment.

Key Takeaways

• Targets: VCF 9.1.0.0 / VVF 9.1.0.0 / VCF 5.2.4; supported range VCF 9.1.0.0 down to 5.2; minimum nested ESX 8.0 U3
• Resource floor rises to 48 CPU / 650 GB / 2.2 TB (VCF 9.1 single site); soft validation warns instead of blocking below-spec deploys — you own the performance risk
• HoloRouter grows to 6 vCPU / 12 GB with no documented in-place upgrade — plan a clean 9.1 router deployment
• The 9.1.0.0-to-5.2 range makes side-by-side upgrade rehearsal a first-class lab scenario

VNA Distributed Cluster Networking vs the Single-Router Model (9.1)#

The headline architectural change in Holodeck 9.1 is Virtual Network Appliance (VNA) cluster support for both management and workload domains — a distributed alternative to the single-HoloRouter networking model that has defined every prior Holodeck generation.

The 9.0-era model and its limits. Through 9.0.x, the HoloRouter was the entire network fabric: L3 routing across all VLAN sub-interfaces, DNS, DHCP, NTP, firewall, and the BGP (FRR) peering that NSX Tier-0 gateways depend on. That concentration made it the most critical single point of failure in the nested environment — one appliance failure takes down routing, name resolution, and edge peering for every site simultaneously. It also imposed a scale ceiling: every deployment's north-south path transits one appliance, and edge/transport placement cannot be meaningfully modeled because there is only one place for it to live.

What VNA changes. With 9.1, New-HoloDeckInstance gains -VnaClusterMgmtDomain and -VnaClusterWkldDomain switches that deploy VNA clusters as the networking layer for the respective domain. Supervisor deployment modes are wired to this choice: Centralized mode rides an NSX Edge cluster, while Distributed mode rides a VNA cluster (the -DeploySupervisorMgmtDomain / -DeploySupervisorWkldDomain parameters changed from switches to strings taking 'Centralized' or 'Distributed'). A community-verified 9.1 deployment combined -NsxEdgeClusterMgmtDomain and -VnaClusterMgmtDomain in a single management-only run, so the models can coexist in one instance.

Availability and scalability design reading. For a VCDX candidate this is precisely the centralized-vs-distributed services argument panels probe in production designs. A distributed cluster spreads the networking function across appliances, shrinking blast radius and removing the single choke point — at the cost of more moving parts, more capacity consumed, and a more complex failure-mode analysis (partial cluster degradation instead of clean binary loss).
The lab now lets you rehearse both: model a centralized edge design and a distributed VNA design on identical nested hardware and defend the trade-off with observed behavior instead of doctrine. Note that the HoloRouter itself does not disappear — core infrastructure services (DNS via Technitium, DHCP, BGP, the reverse-proxied service stack) still live there, so the fabric-services fault domain and the domain-networking fault domain must now be analyzed separately.

Open questions. VNA cluster sizing and host-count minimums were not published at release-intake time, and the incremental resource overhead is unquantified — both are W5 live-deployment capture items. Until then, treat VNA sizing as unverified and budget conservatively inside the 9.1 resource envelope.

Key Takeaways

• VNA clusters (-VnaClusterMgmtDomain / -VnaClusterWkldDomain) provide a distributed alternative to the single-HoloRouter model for both domain types
• Supervisor modes map to the networking choice: Centralized = NSX Edge cluster, Distributed = VNA cluster (string-typed parameters, new in 9.1)
• Distributed networking shrinks blast radius and removes the single choke point at the cost of capacity and failure-mode complexity — a rehearsable VCDX trade-off
• VNA sizing and host minimums are unverified — capture during the live W5 deployment

Click-to-Deploy GitOps vs the PowerShell Path (9.1)#

Holodeck 9.1 introduces a second deployment driver alongside the classic PowerShell cmdlets: Click-to-Deploy GitOps. When the HoloRouter OVA is deployed with GitOps enabled (the default), GitLab is automatically installed and configured on the HoloRouter, a repository pre-loaded with the Holodeck deployment pipeline is created, and the HoloRouter registers itself as a GitLab runner with the appropriate tags — letting you trigger a complete Holodeck deployment directly from the GitLab UI with no manual PowerShell.

What each path offers. The PowerShell path is imperative: New-HoloDeckConfig, then New-HoloDeckInstance with explicit switches, verbose console output, and fine-grained control at every step. Its strengths are visibility and intervention — you see each phase, can resume idempotently from failure, and can compose custom automation around the cmdlets. The GitOps path is declarative-by-pipeline: deployment logic lives in a versioned Git repository, execution happens through pipeline runs, and every run leaves a durable record.
Its strengths are repeatability (the pipeline is the same every time; configuration drift between operators disappears) and auditability (pipeline history answers who deployed what, when, with which commit — a change-control story that ad-hoc PowerShell sessions cannot match). For a team lab or any environment where deployments must be reconstructable, that audit trail is the architecturally significant feature, not the convenience.

Interactions and caveats. The cmdlets remain fully available — GitOps is additive, not a replacement — and even GitOps-driven labs still use the PowerShell module for queries and Day-2 operations. Practical caveats at release: a first-boot issue (GH#139) can leave GitLab unreachable until the iptables port 80/443 ACCEPT rules are moved above the DROP rule; the bundled GitLab version and its resource overhead are unpublished; and on a multi-tenant shared host, a UI-triggered full deployment is exactly the kind of high-blast-radius action that needs coordination rules before anyone clicks it. 9.1 also handles build tokens and credentials as secure strings, improving the hygiene of both paths.

Architect framing. This is a repeatability/auditability-versus-control trade-off you should be able to argue both ways in a defense: choose GitOps when the priority is standardized, reviewable, reconstructable deployments across a team; choose PowerShell when the priority is granular control, interactive troubleshooting, or integration into existing automation. Document the choice and its rationale per environment — the deliberateness is what a panel is probing, not the tool.

Key Takeaways

• A GitOps-enabled HoloRouter auto-provisions GitLab with a pre-loaded pipeline and registers itself as the runner — full deployments trigger from the UI
• PowerShell = imperative control and visibility; GitOps = repeatability plus a pipeline-history audit trail; the cmdlets remain fully supported either way
• First-boot caveat GH#139: reorder iptables 80/443 rules if GitLab is unreachable; bundled GitLab version and overhead are unverified
• Defend the driver choice as a deliberate repeatability/auditability-vs-control decision, made per environment

The 9.1 HoloRouter Services Stack — and What It Does Not Yet Supersede (9.1)#

Holodeck 9.1 rebuilds the HoloRouter's service surface into an integrated, secured stack, reachable via the router's management IP behind an HTTPS reverse proxy with automated certificate management (an internal CA at ca.vcf.lab, with certificates auto-provisioned by a Vault backend).

The stack. HashiCorp Vault (:8200, vault.vcf.lab) provides secrets management and automated certificate provisioning for the other services. Authentik SSO (:9443, auth.vcf.lab) is a full identity provider — OIDC, OAuth2, SAML 2.0, and SCIM 2.0 provisioning — and 9.1 ships two new cmdlets, Initialize-Authentik and Set-VCFSSOConfiguration, that bootstrap an OIDC provider and wire VCF Operations to it with SCIM sync, giving the lab a production-shaped SSO chain. Technitium DNS (:5380, dns.vcf.lab) becomes the DNS management surface: zones, records and DHCP scopes are managed through the Technitium web UI, and the Get-/Set-/Remove-HoloDeckDNSConfig cmdlets are deprecated from 9.1 (valid only up to and including 9.0.2). Webtop (:30000, webtop.vcf.lab) — the browser desktop — is now behind authentication, and with GitOps enabled, GitLab and its container registry join the stack.
Infrastructure services always live on the vcf.lab domain regardless of any custom -DNSDomain; only nested VCF component FQDNs follow the custom domain.

Two caveats on 'HTTPS everywhere'. The claim is real but not absolute. Per the docs, "the individual services run on HTTP internally but are exposed securely via HTTPS at the FQDNs above" — and Webtop on http://<ip>:30000 explicitly bypasses the reverse proxy. Separately, the Authentik username is documented inconsistently: the release notes say admin, the architecture page says akadmin. akadmin is the Authentik product default and is the more likely correct value, but this needs verifying on first deploy.

What this does NOT yet supersede — the open DNS question. It is tempting to read Technitium as retiring dnsmasq outright, and an earlier revision of this section said exactly that. The documentation does not support it.

Evidence cuts both ways: for retirement, Technitium is the new management surface and the DNS cmdlets are deprecated; against it, the 9.1 Set-HoloRouter reference still reads "sets up the HoloRouter virtual appliance with DNSMASQ for DNS services, FRR for BGP routing", Reset-HoloRouter still "uninstalls and removes… DNSMASQ DNS, FRR BGP routing", and the #-in-upstream-DNS parsing workaround still targets /holodeck-runtime/dnsmasq/dnsmasq_configmap.yaml.

The working hypothesis is that Technitium is the user-facing DNS UI layered over a still-present dnsmasq/FRR substrate that Set-HoloRouter configures.

Until that is settled on a live 9.1 router, treat the 9.0-era knowledge as still applicable rather than superseded: assume Set-HoloRouter retains its destructive global behaviour on a shared HoloRouter, assume the upstream-forwarder fix is still needed and still re-broken by it, and always supply an IP address, never an FQDN, as upstream DNS — an FQDN is stored as 1.1.1.1#cloudflare-dns.com and dnsmasq reads the # as a port separator.

Design reading. The security posture shift — HTTPS everywhere, authenticated access, centralized secrets, real SSO — moves the lab appreciably closer to a production trust model, which strengthens the security dimension of any lab-derived design narrative. It also concentrates still more critical services on one appliance: the availability analysis of the HoloRouter as a fault domain gets heavier in 9.1, not lighter. And the epistemic lesson is itself defensible material: an architect who says "the docs conflict here, so I re-baselined it in the lab before trusting it" is demonstrating exactly the rigour a panel is testing for.

Version Comparison

Key Takeaways

• Stack: Vault :8200, Authentik SSO/SCIM :9443, Technitium DNS :5380, authenticated Webtop :30000 — behind an HTTPS reverse proxy with an internal CA at ca.vcf.lab
• Initialize-Authentik + Set-VCFSSOConfiguration build a production-shaped OIDC/SCIM SSO chain into VCF Operations
• Technitium is the new DNS management surface and the DNS cmdlets are deprecated — but the docs do NOT establish that dnsmasq is gone; Set-/Reset-HoloRouter still reference it
• Until re-baselined live, treat the 9.0-era dnsmasq forwarder fix and shared-host Set-HoloRouter caution as STILL APPLICABLE — and always give upstream DNS as an IP, never an FQDN
• 'HTTPS everywhere' has exceptions: services are HTTP internally, and Webtop :30000 bypasses the proxy entirely

Day-2 Operations In-Toolkit: Hosts, Automation, Supervisors (9.1)#

Through 9.0.x, Holodeck was overwhelmingly a bring-up tool: Day-0/Day-1 deployment was automated, but growing or extending a running instance meant either full redeployment or manual surgery. Holodeck 9.1 changes the platform's character by making Day-2 lifecycle operations first-class: Update-HoloDeckInstance expands from two to five parameter sets, each an exclusive operation against a running deployment.

The five operations. (1) -AdditionalCluster adds a three-node vSphere cluster to the management or a workload domain. (2) -AddVcfAutomationAllAppsOrg configures the VCF Automation All-Apps organization on an existing VCFA deployment. (3) -DeploySupervisor stands up a Supervisor in the management or workload domain Day-2, with -SupervisorDeploymentMode selecting 'Centralized' (NSX Edge-backed) or 'Distributed' (VNA-backed, VCF 9.1.0.0+ only; default Centralized).
(4) -DeployVcfAutomation adds VCF Automation post-deploy through VCF Operations Manager — the documented pattern is deliberate: deploy the rest of the stack first, verify capacity, then add VCFA. (5) -DeployNewHosts adds nested ESX hosts to a live deployment with explicit -CPU, -MemoryInGb, -Nodes, -vSANMode, and optional per-disk sizing; the older New-HoloDeckESXiNodes cmdlet is retained alongside it, and the relationship between the two paths is a live-verification item.

Why this matters architecturally. First, it keeps Day-0 short and lean: instead of front-loading a 12-hour full-stack deployment, you bring up a minimal management domain and stage capacity-hungry components (VCFA, Supervisors, extra clusters) incrementally, checking headroom at each step — exactly the staged-consumption discipline a capacity-constrained production rollout uses. Second, it makes expansion and lifecycle scenarios rehearsable: add-host workflows, cluster growth, and Supervisor enablement become repeatable lab exercises rather than redeploy-everything events, which pairs directly with the VKS 3.6 content elsewhere in the Academy. Third, for a VCDX defense it supplies operational evidence for Day-2 claims — you can demonstrate the expansion path you designed, not just assert it.

Caveats. Parameter sets are exclusive (one operation per invocation); everything is site-scoped via the mandatory -Site parameter. Known issue GH#143 documents a Day-2 All-Apps Org failure on fresh management-only deployments (an IP Space vs gateway-CIDR mismatch, with a community workaround), and 9.1-specific timings for Day-2 operations are unpublished — capture actuals during the live W5 deployment before quoting maintenance-window numbers in any design artifact.

Key Takeaways

• Update-HoloDeckInstance now has five exclusive Day-2 parameter sets: AdditionalCluster, AddVcfAutomationAllAppsOrg, DeploySupervisor (Centralized/Distributed), DeployVcfAutomation, DeployNewHosts
• Staged consumption becomes the norm: keep Day-0 minimal, verify capacity, then add VCFA / Supervisors / hosts incrementally
• Distributed Supervisor mode requires the VNA path and VCF 9.1.0.0+; Day-2 operations are site-scoped and one-per-invocation
• GH#143 (All-Apps Org IP Space mismatch) has a community workaround; Day-2 timings are unverified until the live W5 deployment