Academy/Holodeck Lab Setup & Operations/External Access, DNS, and Routing Configuration
This lab targets VCF 9.0.2

External Access, DNS, and Routing Configuration

VCF 9.0.2Intermediateadminnetworkvcdx⏱ 60 min

Core workflow applies to VCF 9.0.0 through 9.0.2 with Holodeck 9.0.2.19. HoloRouter networking and DNS forwarding principles are version-agnostic and apply to VCF 5.2.x as well.

Objectives

  • Understand HoloRouter architecture including L3 gateway, DNS forwarder, DHCP, and BGP services
  • Configure static routing from external Windows, Linux, and macOS workstations to reach nested networks
  • Validate network reachability and routing topology using ping, traceroute, and route command tools
  • Configure DNS resolution for nested FQDNs using conditional forwarding and split-horizon DNS techniques
  • Import and validate VMCA root certificate for clean HTTPS browser access to management UIs
  • Design multi-workstation and VPN access patterns with attention to security and RCAR constraints

Prerequisites

Holodeck instance deployed and running with healthy management domain (holodeck-02 completed). HoloRouter and nested ESXi hosts all operational. External workstations on different physical networks or subnets from the Holodeck pod.

Prior labs: holodeck-02

Required skills:

  • Basic networking: IP addressing, CIDR notation, routing tables, DNS concepts
  • Comfort with Windows Command Prompt and PowerShell, or Linux/macOS terminal
  • Understanding of VCF management domain components and their DNS/IP dependencies
  • Familiarity with browser certificate warnings and how to import root CAs
  • Basic troubleshooting: ping, traceroute, nslookup, netstat
📸 Starting State: S3-52 — Management Domain Deployed (VCF 5.2)

Lab Environment

Holodeck pod with deployed management domain (from holodeck-02). HoloRouter (10.1.1.1) acts as L3 gateway, DNS forwarder, DHCP server, and BGP speaker for the entire 10.1.1.0/20 supernet. Nested ESXi hosts on 10.1.1.0/24, management services (SDDC Manager 10.1.1.5, vCenter 10.1.1.10, NSX 10.1.1.20) on 10.1.1.0/20. External workstations on physical networks outside the pod, reaching Holodeck via HoloRouter's management IP (typically on the physical host's external VLAN).

graph TB
  PHY[Physical Workstations<br/>e.g. 192.168.1.100] -->|Static Route<br/>10.1.1.0/20| HR[HoloRouter<br/>10.1.1.1]
  HR -->|DNS Forwarder<br/>Port 53| HC[HoloConsole<br/>10.1.1.100]
  HR --> MGMT[Management Network<br/>10.1.1.0/20]
  HR --> ESXi[ESXi Hosts<br/>10.1.1.0/24]
  MGMT --> SDDC[SDDC Manager 10.1.1.4]
  MGMT --> VC[vCenter 10.1.1.6]
  MGMT --> NSX[NSX Manager 10.1.1.10]
  ESXi --> E1[ESXi-01 10.1.1.101]
  ESXi --> E2[ESXi-02 10.1.1.102]
  ESXi --> E3[ESXi-03 10.1.1.103]
  ESXi --> E4[ESXi-04 10.1.1.104]

IP Addressing

NetworkPurposeVLAN
10.1.1.0/20Management network (SDDC Manager, vCenter, NSX, HoloRouter, HoloConsole)VLAN 1644
10.1.1.0/24ESXi host management VMkernelVLAN 1644
10.1.2.0/24vMotionVLAN 1645
10.1.3.0/24vSANVLAN 1646
10.1.4.0/24NSX Host TEPVLAN 1647
10.1.5.0/24NSX Edge TEPVLAN 1648

Credentials

SystemUsernamePassword
HoloRouteradminSSH credentials set by Holodeck config.json
SDDC Manageradministrator@vsphere.localSet in VCF bring-up spec JSON
vCenter Serveradministrator@vsphere.localSame as SDDC Manager SSO password
NSX ManageradminSet in VCF bring-up spec JSON
API Authenticationadministrator@vsphere.localFor API calls, use doubled password: VMware123!VMware123! (not VMware123!)

Tasks

Task 1 Understand HoloRouter network architecture and services

manageability

The HoloRouter is the linchpin of external access. Understanding its role as L3 gateway, DNS forwarder, DHCP server, and BGP speaker is essential for troubleshooting multi-site deployments and mapping Holodeck concepts to production ToR-based architecture. This mirrors VCDX design discussions about network isolation, management segmentation, and OOB access.

Step 1

SSH into the HoloRouter from HoloConsole or your workstation: ssh admin@10.1.1.1

HoloRouter login prompt. You are now in a Linux-based virtual appliance.
The HoloRouter is a FreeBSD/Linux-based appliance. Standard Linux networking commands apply.
Step 2

View the HoloRouter's network interfaces: ip addr show or ifconfig

Multiple interfaces: eth0 (external/management), eth1-eth5 (internal port groups for management, vMotion, vSAN, NSX TEP, NSX Edge TEP). eth0 has an IP on the physical host's external network; internal interfaces on 10.x.x.x subnets.
The external interface (eth0) is how you reach the HoloRouter from your workstation. Internal interfaces carry tenant traffic.
Step 3

View the routing table: route -n or ip route show

Routes for 10.1.1.0/20, 10.1.1.0/24 (ESXi), 10.1.2.0/24 (vMotion), 10.1.3.0/24 (vSAN), 10.1.4.0/24 (NSX Host TEP), 10.1.5.0/24 (NSX Edge TEP). Default route likely points to the physical host's gateway.
Each internal network has its own route entry. This is where you would add BGP routes in multi-site deployments.
Step 4

Check the DNS forwarder status: systemctl status dnsmasq or systemctl status named (depending on which DNS daemon is configured)

Service running (active/running). DNS daemon listening on port 53.
If DNS is not running, nested services may fail to resolve FQDNs. This is one of the most common issues in external access scenarios (KB thread 24: Waiting for FRR service, thread 38: Holodeck 9.0.1 DNS troubleshooting).
Step 5

Check the BGP daemon status: systemctl status frr

Service running (active/running). FRR (Free Range Routing) is the BGP/OSPF protocol stack.
FRR is used in multi-site deployments for dynamic route advertisement. In single-site labs, it runs but may not be actively used.
Step 6

Check the DHCP server status: systemctl status dnsmasq | grep -i dhcp or check /etc/dnsmasq.conf for DHCP configuration

DHCP enabled and serving IPs on the internal networks. Config shows dhcp-range directives.
DHCP allocates IPs to new VMs on the internal networks. If a VM is not getting an IP, DHCP may have exhausted its pool.
Step 7

Verify the default CIDR assignment: cat /etc/holodeck/config.json | grep -i cidr or check the HoloRouter's config directory

Default CIDR is 10.1.1.0/20 (or custom value if modified during deployment). This represents the entire management domain supernet.
All IP ranges (10.1.1.0/20 for management, 10.1.1.0/24-10.1.5.0/24 for services) fall within this supernet.
Step 8

Document a traffic flow diagram: Write out a single packet flow from external workstation (e.g., 192.168.1.100) to vCenter (10.1.1.6). Include: (1) Source IP 192.168.1.100, (2) Destination 10.1.1.6, (3) Route match 10.1.1.0/20 -> via HoloRouter external IP, (4) HoloRouter L3 forward to 10.1.1.6 via eth1, (5) Return path from 10.1.1.6 via HoloRouter -> external workstation.

Diagram or text description showing the L3 path and HoloRouter's role as the gateway
This exercise prepares you for VCDX discussions about physical network design: HoloRouter ~ production ToR switch, internal interfaces ~ leaf-to-spine fabric, external interface ~ uplink to management network.

Validation Gate

Check: SSH to HoloRouter and verify: (1) ip addr shows multiple interfaces, (2) route -n shows all internal routes, (3) systemctl status dnsmasq/named shows DNS running, (4) systemctl status frr shows BGP running.

Expected: All four services operational. HoloRouter acting as router, DNS forwarder, DHCP, and BGP speaker.

Common Errors

Cannot SSH to HoloRouter (connection refused or timeout)
Cause: HoloRouter not reachable via its external management IP, or SSH service not running
Fix: From HoloConsole or physical ESXi host, verify HoloRouter is powered on. Try ping first. If SSH is not listening, restart it: systemctl restart sshd on the HoloRouter.
📋 KB: General troubleshooting — HoloRouter connectivity
DNS service not running (systemctl status dnsmasq returns inactive)
Cause: DNS daemon crashed during deployment or resource contention
Fix: Restart the DNS service: systemctl restart dnsmasq. Check logs: journalctl -u dnsmasq -n 50. If it crashes again, check /etc/dnsmasq.conf for syntax errors.
📋 KB: Thread 38: Holodeck 9.0.1 DNS troubleshooting
FRR service shows 'failed' or 'dead'
Cause: FRR daemon crashed, likely due to misconfigured BGP peers or insufficient resources
Fix: Restart FRR: systemctl restart frr. Check status of individual daemons: systemctl status bgpd. If still fails, check /var/log/frr/bgpd.log for errors.
📋 KB: Thread 24: Waiting for FRR service

Task 2 Configure static routing from external workstations

manageability

Static routing is the foundational mechanism for external access. In production, this mirrors BGP and static route configuration on physical ToR switches and management network gateways. Learning to add, validate, and troubleshoot routes on Windows, Linux, and macOS prepares you for multi-site VCDX discussions.

Step 1

On your external Windows workstation, open PowerShell as Administrator and run: route add 10.1.1.0 mask 255.255.240.0 <holorouter-external-ip> -p (replace <holorouter-external-ip> with the actual external management IP of the HoloRouter, e.g., 192.168.1.50). The -p flag makes the route persistent across reboots.

Output: 'OK!' or similar confirmation. Route is added to the system routing table.
Without the -p flag, the route disappears after reboot. If you don't know the HoloRouter's external IP, check from HoloConsole: Get-HoloDeckInstance | Format-List | grep -i external-ip. Note: If using Get-HoloDeckInstance cmdlets, first run Import-HoloDeckConfig -ConfigID <config-id> in the same PowerShell session.
Step 2

Verify the route was added (Windows): route print | findstr 10.0.0

Table entry showing: Destination: 10.1.1.0, Netmask: 255.255.240.0, Gateway: <holorouter-external-ip>, Interface: <your-workstation-nic-ip>, Metric: 1
If the metric is very high (e.g., 256), Windows considers it low-priority. Check that your workstation has a direct L2 path to the HoloRouter's external network or that the HoloRouter is reachable.
Step 3

On a Linux workstation, run: sudo ip route add 10.1.1.0/20 via <holorouter-external-ip>

Command completes silently (success) or reports the route is already present.
To make it persistent across reboots, add the same line to /etc/netplan/01-netcfg.yaml (Ubuntu/Debian) or create a file in /etc/sysconfig/network-scripts/ (RHEL/CentOS).
Step 4

On a macOS workstation, run: sudo route -n add -net 10.1.1.0 -netmask 255.255.240.0 <holorouter-external-ip>

Command succeeds with no output or shows 'add net 10.1.1.0: gateway <holorouter-external-ip>'
macOS routes do not persist across reboots by default. For persistence, create a LaunchDaemon script or use a third-party tool like Tunnelblick.
Step 5

From your external workstation (any OS), test basic reachability: ping 10.1.1.1 (HoloRouter). You should get a reply within 1-5ms if on a local network, or higher latency if over a WAN.

Reply from 10.1.1.1: bytes=32 time=<N>ms
If ping fails, verify: (1) the static route was added correctly, (2) there is no firewall blocking ICMP traffic, (3) the HoloRouter is powered on and responding.
Step 6

Test connectivity to the nested management network: ping 10.1.1.4 (SDDC Manager), ping 10.1.1.6 (vCenter), ping 10.1.1.101 (ESXi-01). All should respond.

Replies from all three IPs
If 10.1.1.1 responds but 10.1.1.4 and 10.1.1.6 do not, the HoloRouter's DNS or DHCP may have issues. If only the HoloRouter responds, check that other VMs are powered on and have network connectivity.
Step 7

Use traceroute to visualize the path to nested services. Run: traceroute 10.1.1.6 (or tracert on Windows) to trace the path to vCenter.

Output shows: hop 1 is HoloRouter (10.1.1.1), hop 2 is vCenter (10.1.1.6). Total hops should be 2 or 3 depending on network topology.
If you see more than 3 hops, an intermediate switch or firewall may be introducing latency. This is expected in some corporate networks.

Validation Gate

Check: From external workstation: (1) route print/ip route show confirms 10.1.1.0/20 route via HoloRouter external IP, (2) ping 10.1.1.1, 10.1.1.4, 10.1.1.6 all reply, (3) traceroute 10.1.1.6 shows 2-3 hops ending at vCenter.

Expected: External workstation has full routing to all nested networks via HoloRouter. All ICMP traffic succeeds.

Common Errors

Route added but ping 10.1.1.1 fails with 'Destination host unreachable'
Cause: No L2 connectivity between workstation and HoloRouter's external network, or firewall blocking ICMP
Fix: Verify: (1) Ping the HoloRouter's external IP using its actual IP address (not FQDN) from your workstation. (2) Ensure your workstation NIC is on the same physical network as the HoloRouter's external interface. (3) Check firewall rules: disable host-based firewall or add an ICMP inbound rule.
📋 KB: Thread 3: External Jumpserver Access
Windows route shows very high metric (e.g., 256), or route appears as 'inactive'
Cause: Network interface metric is high, or the target network is unreachable
Fix: Check the metric of your primary network adapter: route print. Ensure your workstation's default gateway is set correctly and that you can reach the HoloRouter's external network.
Linux: route add command fails with 'RTNETLINK answers: File exists'
Cause: Route already exists in the kernel routing table
Fix: Check existing routes: ip route show. If the route is already present, no action needed. If you want to replace it, use: sudo ip route replace 10.1.1.0/20 via <holorouter-external-ip>
macOS: route command fails with 'Operation not permitted'
Cause: Command not run with sudo
Fix: Prefix the command with sudo: sudo route -n add ...

Task 3 Configure DNS resolution for nested FQDNs

manageability

DNS is critical for certificate validation and user experience. Split-horizon DNS (different answers for internal vs external queries) and conditional forwarding are production-grade techniques used in VCDX designs. This task teaches both the 'quick fix' (hosts file) and the 'proper way' (DNS server configuration).

Step 1

From your external workstation, test current DNS resolution: nslookup sddc-manager.site-a.vcf.lab or nslookup vcenter.site-a.vcf.lab

Either: (1) 'Non-existent domain' or 'NXDOMAIN' if DNS is not configured, or (2) IP address if a DNS server is already set up to resolve these names.
If you get an IP other than 10.1.1.x, DNS is resolving to an external server, not the HoloRouter. This is not what you want.
Step 2

Option A (Quick fix for single workstation): Add static hosts file entries. On Windows, edit C:\Windows\System32\drivers\etc\hosts. On Linux/macOS, edit /etc/hosts. Add three lines:
10.1.1.4 sddc-manager.site-a.vcf.lab
10.1.1.6 vcenter.site-a.vcf.lab
10.1.1.10 nsx-manager.site-a.vcf.lab

Entries saved. nslookup sddc-manager.site-a.vcf.lab now returns 10.1.1.4 from your workstation's host resolver (even if network DNS is not configured).
Windows: Requires Administrator privilege to edit the hosts file. Use Notepad Run As Administrator.
Step 3

Verify hosts file DNS: nslookup vcenter.site-a.vcf.lab

Output shows: Name: vcenter.site-a.vcf.lab, Address: 10.1.1.6
nslookup may still query the network DNS first. To ensure hosts file is being used, use: cat /etc/hosts | grep vcenter (Linux/macOS) or type C:\Windows\System32\drivers\etc\hosts | findstr vcenter (Windows).
Step 4

Option B (Proper DNS forwarding for teams): Configure your workstation to use the HoloRouter as a DNS server. On Windows: Control Panel > Network and Sharing Center > Change Adapter Settings > right-click your NIC > Properties > IPv4 > Set DNS servers to 10.1.1.1. On Linux/macOS: Edit /etc/netplan/01-netcfg.yaml (Ubuntu) or /etc/resolv.conf (macOS) and add: nameserver 10.1.1.1

DNS configuration saved. nslookup now queries the HoloRouter DNS forwarder.
If you use the HoloRouter as DNS for ALL queries, you may break resolution of external domains (google.com, etc.) unless the HoloRouter is configured to forward upstream queries. The proper setup is a conditional forwarder: HoloRouter forwards *.site-a.vcf.lab queries internally, others to your ISP DNS.
Step 5

Verify DNS forwarding: nslookup sddc-manager.site-a.vcf.lab and nslookup google.com

sddc-manager.site-a.vcf.lab resolves to 10.1.1.4, google.com resolves to its public IP (or your ISP DNS's answer).
If google.com fails, the HoloRouter is not forwarding upstream queries. This is a configuration issue on the HoloRouter's DNS daemon.
Step 6

Configure HoloRouter DNS forwarding (if needed). SSH to HoloRouter: ssh admin@10.1.1.1. Edit the DNS config: cat /etc/dnsmasq.conf | grep -A5 'address=\|server='

Configuration lines showing local domain resolution (address=/site-a.vcf.lab/...) and upstream servers (server=...)
dnsmasq is the default DNS daemon. Configuration is in /etc/dnsmasq.conf. For BIND/named, check /etc/named.conf instead.
Step 7

Add upstream DNS forwarders to HoloRouter if missing. Edit /etc/dnsmasq.conf: add or uncomment server=8.8.8.8 and server=1.1.1.1 (Google and Cloudflare public DNS). Then restart: systemctl restart dnsmasq

dnsmasq restarts without errors. External DNS queries now resolve via the public nameservers.
In production, use your organization's DNS servers, not public ones, to maintain security and audit trails.
Step 8

Test split-horizon DNS: From your external workstation with HoloRouter as DNS, run: nslookup sddc-manager.site-a.vcf.lab. From HoloConsole (inside the Holodeck pod), also run nslookup sddc-manager.site-a.vcf.lab. Both should return 10.1.1.4, confirming that the name resolves consistently.

Both queries return 10.1.1.4. DNS resolution is split-horizon: same answer regardless of origin.
In complex deployments, you might have different DNS answers for internal vs external queries. This lab uses simple split-horizon: same answer from everywhere.

Validation Gate

Check: From external workstation: (1) nslookup sddc-manager.site-a.vcf.lab returns 10.1.1.4, (2) nslookup vcenter.site-a.vcf.lab returns 10.1.1.6, (3) nslookup nsx-manager.site-a.vcf.lab returns 10.1.1.10, (4) nslookup google.com resolves to a public IP or your ISP's answer.

Expected: All nested FQDNs resolve to correct IPs. External domains still resolvable.

Common Errors

nslookup vcenter.site-a.vcf.lab returns 'Non-existent domain' or times out
Cause: DNS server (HoloRouter or system resolver) is not configured to resolve this zone, or DNS is not running on HoloRouter
Fix: Verify: (1) HoloRouter DNS is running: ssh admin@10.1.1.1 then systemctl status dnsmasq. (2) Your workstation is using HoloRouter as DNS (if using Option B) or has the hosts file entry (if using Option A). (3) Check HoloRouter DNS logs: journalctl -u dnsmasq -n 20.
📋 KB: Thread 38: Holodeck 9.0.1 DNS troubleshooting
nslookup google.com fails but nslookup sddc-manager.site-a.vcf.lab works
Cause: HoloRouter DNS is not forwarding upstream queries. Likely missing or commented-out 'server=' lines in /etc/dnsmasq.conf
Fix: SSH to HoloRouter and edit /etc/dnsmasq.conf: add 'server=8.8.8.8' and 'server=1.1.1.1' if not present. Restart: systemctl restart dnsmasq. Test: nslookup google.com from your workstation.
📋 KB: General DNS configuration troubleshooting
DNS queries very slow (5+ second timeout) or frequent timeouts
Cause: HoloRouter DNS daemon is overloaded or there are too many concurrent queries (thread 21: Maximum number of concurrent DNS queries reached)
Fix: Increase dnsmasq cache size and connection limits in /etc/dnsmasq.conf: cache-size=10000, max-cache-ttl=3600. Restart dnsmasq. Monitor with: watch -n 1 'netstat -an | grep :53'
📋 KB: Thread 21: Maximum number of concurrent DNS queries reached (max: 150)
Windows hosts file edits don't take effect (nslookup still fails)
Cause: DNS client cache not flushed after hosts file change, or file was not saved correctly
Fix: Flush DNS cache: ipconfig /flushdns. Verify hosts file was saved: type C:\Windows\System32\drivers\etc\hosts. Re-run nslookup.

Task 4 Certificate trust and browser access

security

HTTPS access with certificate validation is essential for production-grade management UIs. Understanding self-signed certificates, root CAs, and VMCA (VMware Certificate Authority) is a VCDX-level topic. This task bridges the gap between lab self-signed certs and production PKI, teaching how to build trust without disabling security.

Step 1

From your external workstation, open a browser and navigate to https://10.1.1.6 (vCenter by IP). Observe the certificate warning (ERR_CERT_INVALID, UNABLE_TO_VERIFY_LEAF_SIGNATURE, or similar depending on browser).

Browser displays a security warning: 'Your connection is not private' or 'Certificate not trusted'. The certificate is valid (signed by VMCA) but VMCA root is not in your workstation's trusted CA store.
Do NOT click 'Proceed Anyway' yet — this disables all certificate validation. Instead, we will properly import the VMCA root certificate.
Step 2

Export the VMCA root certificate. From HoloConsole, access vCenter via https://10.1.1.6 (using the webtop browser, not your workstation). Click the certificate warning icon (pad lock with red X) and select 'View Certificate'. Export the root CA certificate (typically the issuer in the cert chain). Save it as vmca-root.crt.

vmca-root.crt file downloaded or saved to HoloConsole. The file is in PEM format (plain text, starting with '-----BEGIN CERTIFICATE-----').
Alternatively, use OpenSSL from HoloRouter: openssl s_client -connect 10.1.1.6:443 -showcerts < /dev/null | grep -A 30 'BEGIN CERTIFICATE' | head -35 > vmca-root.crt
Step 3

Transfer the vmca-root.crt file to your external workstation (use SCP, email, or copy-paste).

vmca-root.crt is now on your workstation.
Step 4

Import VMCA root into your system's trusted CA store. On Windows: Right-click the .crt file > Install Certificate > Local Machine > Place all certificates in: Trusted Root Certification Authorities. Complete the wizard. On macOS: Double-click vmca-root.crt and select 'Always Trust' when prompted. On Linux: Copy to /usr/local/share/ca-certificates/vmca-root.crt and run: sudo update-ca-certificates

Certificate imported successfully. Windows shows a confirmation dialog. macOS shows cert in Keychain. Linux updates /etc/ssl/certs/.
Importing a root CA means you trust ALL certificates signed by that CA. Only do this for lab environments or internal CAs you control.
Step 5

Close and reopen your browser to clear the cert cache. Navigate again to https://vcenter.site-a.vcf.lab (using the FQDN, not the IP).

Browser now displays a clean green lock icon and shows 'Your connection is secure' (or similar). No certificate warnings. You can access vCenter UI without warnings.
Using the FQDN (not IP) ensures the certificate CN (Common Name) matches the URL and browser can fully validate it.
Step 6

Repeat the VMCA certificate import for SDDC Manager (https://sddc-manager.site-a.vcf.lab) and NSX Manager (https://nsx-manager.site-a.vcf.lab). Test each in your browser.

All three management UIs show green locks and are fully accessible without certificate warnings.
If all three components use the same VMCA root, importing once is sufficient. Multiple services can share a single root CA in enterprise environments.
Step 7

Log in to vCenter (https://vcenter.site-a.vcf.lab) using administrator@vsphere.local credentials and verify the management cluster is visible and healthy.

vSphere Client loads successfully. Management cluster shows 4 connected hosts, vSAN health is green.
This confirms that networking, DNS, and certificate setup are all working together for end-to-end HTTPS access.

Validation Gate

Check: From external workstation browser: (1) https://vcenter.site-a.vcf.lab shows green lock and no cert warnings, (2) https://sddc-manager.site-a.vcf.lab shows green lock, (3) https://nsx-manager.site-a.vcf.lab shows green lock, (4) Can log in to vCenter and see cluster health.

Expected: All three management UIs accessible via HTTPS with valid certificates and green browser locks.

Common Errors

After importing VMCA root, browser still shows certificate warning
Cause: Certificate cache not cleared, or VMCA root was not imported correctly
Fix: Clear browser cache: Ctrl+Shift+Delete (Chrome/Windows), Cmd+Shift+Delete (Firefox), Command+Option+E (Safari). Close and reopen browser. Verify cert is in trust store: Windows Cert Manager (certmgr.msc) > Trusted Root Certification Authorities.
📋 KB: General certificate troubleshooting
URL mismatch error: certificate for 10.1.1.6 but browser is accessing vcenter.site-a.vcf.lab
Cause: Certificate CN is set to the IP (10.1.1.6) instead of FQDN (vcenter.site-a.vcf.lab)
Fix: Use the IP address in the URL (https://10.1.1.6) instead of FQDN, or regenerate the certificate with the FQDN in the CN. VMCA can be reissued — contact SDDC Manager to regenerate certs.
📋 KB: Thread 12: Generate and Install VMCA Certificate on SDDC Manager
Cannot access vCenter at all: connection timeout or reset
Cause: vCenter service not running, or network connectivity broken
Fix: Verify from HoloConsole that vCenter is running: ping 10.1.1.6. From vCenter CLI (VM console), check: systemctl status vpxd. If service is down, restart: systemctl restart vpxd.
📋 KB: Task 4 common errors from holodeck-02

Task 5 Multi-workstation and VPN access patterns

security

In production, VCF environments are accessed by teams, not individuals. Understanding how to securely share access, manage IP conflicts, and design access patterns (direct external, VPN, bastion hosts) is a VCDX-level skill. This task exercises RCAR (Requirements, Constraints, Assumptions, Risks) thinking.

Step 1

Document your current access pattern: External workstation > Static route to HoloRouter > vCenter/SDDC Manager/NSX via FQDNs. Write down: (1) Source IP of your workstation, (2) HoloRouter external IP, (3) Nested service IPs. This is the 'direct access' pattern.

Access pattern documented. Example: Workstation 192.168.1.100 -> Route 10.1.1.0/20 via HoloRouter 192.168.1.50 -> vCenter 10.1.1.6 -> Access granted
This pattern works for individual lab access but doesn't scale to teams. It also has zero access controls beyond network routing.
Step 2

Identify the constraints of direct access: (1) HoloRouter is a single point of failure — if it goes down, all external access is lost, (2) All external workstations must have a route to the Holodeck pod, (3) No centralized access logging or audit trail, (4) No encryption of traffic between workstation and HoloRouter (only HTTPS to management UIs), (5) IP conflicts: if your corporate network also uses 10.x.x.x addresses, routing will fail.

List of at least 4 constraints identified
This mirrors production design discussions: Why do we need a bastion host? Why use VPN? These are answers to these exact constraints.
Step 3

Design a VPN access pattern: Imagine a remote team member on a public WiFi (IP 203.0.113.50) who needs to access vCenter. How would they connect? Document: (1) VPN client on workstation, (2) VPN server on physical host's external network, (3) VPN tunnel encrypts traffic, (4) Inside tunnel, traffic routes normally to Holodeck via HoloRouter.

VPN pattern documented. Example: Remote workstation 203.0.113.50 (public WiFi) -> VPN tunnel to VPN server 192.168.1.200 -> Inside tunnel, workstation gets IP 192.168.1.150 -> Route 10.1.1.0/20 via HoloRouter 192.168.1.50 -> vCenter 10.1.1.6
VPN adds three things: (1) Encryption, (2) Centralized access control (VPN server acts as gatekeeper), (3) Audit logging (VPN server logs who connected when).
Step 4

Design a bastion host (jumpserver) pattern: A dedicated VM (HoloConsole already serves this role, but imagine a separate bastion host) sits on the physical host's external network and provides SSH/RDP access. Teams SSH to bastion, then access nested services from inside. Document: (1) Bastion IP (e.g., 192.168.1.100), (2) SSH keys for authentication, (3) Audit log of who logged in when, (4) Nested services only accept connections from bastion (optional firewall rule).

Bastion pattern documented. Example: Team member workstation -> SSH to bastion 192.168.1.100 -> Inside bastion, open tunnel or use webtop browser -> Access vCenter 10.1.1.6 -> All activity logged on bastion
Bastion hosts are common in production. They centralize access control, logging, and allow security teams to monitor all administrative access to infrastructure.
Step 5

Identify IP conflict risks and mitigation. Scenario: Your company's internal network is 10.1.1.0/8 (a common private range). Your workstation is 10.1.50.100. HoloRouter's management network is also 10.1.1.0/20. What happens if both try to use the same IP space? Document: (1) Problem: IP conflict in routing, (2) Solution: Use a different private range for Holodeck (e.g., 172.16.0.0/16 instead of 10.1.1.0/20), (3) Mitigation: In config.json, customize the CIDR before deploying.

IP conflict scenario and mitigation documented. Example solution: Configure Holodeck with CIDR 172.16.0.0/20 Route 172.16.0.0/20 via HoloRouter vCenter becomes 172.16.0.6 (instead of 10.1.1.6)
IP conflicts are a sneaky failure mode. They don't cause deployment errors — they cause intermittent connectivity issues that are hard to debug. This is documented in holodeck-kb-normalized.json as a known pitfall.
Step 6

Document RCAR for your Holodeck external access design. Capture:

  • Requirements: Who needs to access? What services? When? (e.g., 'Lab admin needs 24/7 access to vCenter and SDDC Manager')
  • Constraints: HoloRouter single point of failure, IP space conflicts, team size limits
  • Assumptions: HoloRouter always operational, DNS resolves consistently, VMCA root imported on all workstations
  • Risks: HoloRouter failure -> total access loss, IP conflicts -> routing issues, Certificate revocation -> all browsers fail, Multi-site deployments -> more HoloRouters = more coordination overhead
RCAR document (can be free-form text or structured). At least 3 items in each category.
This RCAR framework is exactly what VCDX panelists expect. Practice articulating requirements, constraints, assumptions, and risks for every design decision.

Validation Gate

Check: Deliver: (1) Access pattern diagram (direct or VPN), (2) Constraints list (4+ items), (3) VPN or bastion design sketch, (4) IP conflict mitigation plan, (5) RCAR document with 3+ items per category.

Expected: Comprehensive design documentation reflecting production-grade access patterns and risk awareness.

Common Errors

Team member cannot access vCenter from their workstation even though another team member can
Cause: IP conflict: their local network uses a different 10.x.x.x range, routing is ambiguous
Fix: Have them use a different CIDR range for their route, or use the bastion host pattern instead of direct external access. Document the IP space requirements in your lab runbook.
📋 KB: General network design — IP planning
VPN connection works, but nested services are unreachable from inside VPN tunnel
Cause: VPN server does not have a route to 10.1.1.0/20 (or custom Holodeck CIDR), or HoloRouter doesn't have return route
Fix: Configure VPN server with route: route add 10.1.1.0 mask 255.255.240.0 <holorouter-external-ip>. Also ensure HoloRouter knows how to reach the VPN clients' IP range (typically 192.168.1.0/24 if VPN uses that pool).
📋 KB: General VPN + routing troubleshooting
Bastion host works for SSH access, but nested services are inaccessible from the bastion
Cause: Bastion host doesn't have the static route to Holodeck network, or firewall blocks traffic from bastion to nested services
Fix: Add static route on bastion: route add 10.1.1.0 mask 255.255.240.0 <holorouter-external-ip> -p. Check firewall rules on nested services (e.g., ESXi host firewall).

Final Validation

External access to the Holodeck nested environment is fully configured and operational. HoloRouter acts as the L3 gateway, DNS forwarder, and network hub. All external workstations can reach nested services via static routes. DNS resolution of FQDNs is consistent. VMCA certificates are trusted and HTTPS access is clean. Multi-workstation access patterns have been designed with attention to security, scalability, and risk.

✓ HoloRouter: ip addr, route, systemctl status dnsmasq, systemctl status frr → All four services operational. Multiple interfaces, correct routing table, DNS and BGP running.

✓ External workstation: route print/ip route show shows 10.1.1.0/20 route → Static route present with correct gateway (HoloRouter external IP)

✓ Reachability: ping 10.1.1.1, 10.1.1.4, 10.1.1.6 from external workstation → All three IPs reply

✓ DNS: nslookup sddc-manager.site-a.vcf.lab, nslookup vcenter.site-a.vcf.lab → Both resolve to 10.1.1.4 and 10.1.1.6 respectively

✓ Browser: https://vcenter.site-a.vcf.lab, https://sddc-manager.site-a.vcf.lab, https://nsx-manager.site-a.vcf.lab → All three load with green locks and no cert warnings

✓ VMCA root certificate imported into workstation trust store → Certificate visible in trust store. Certificate chain validates.

✓ RCAR design document completed for multi-workstation access → Requirements, Constraints, Assumptions, Risks documented. Access patterns sketched.

Cleanup / Restore

Snapshot: holodeck-04-complete

• Snapshots of HoloRouter and all nested management VMs created with label 'holodeck-04-complete' for rapid rollback to this state

• Static routes documented in a lab runbook for reference in future labs or team onboarding

• VMCA root certificate exported and stored in a shared location (e.g., team wiki or shared drive) for other team members to import

• DNS configuration (hosts file entries or HoloRouter DNS forwarder settings) documented

• Network diagram (physical and logical) updated to show external access paths and HoloRouter's role

Design Reflection (VCDX)

A VCDX panelist examining a VCF management network design would probe: Why separate the HoloRouter from the management services? (Answer: isolation, single point of control, mirrors production ToR architecture). What happens if the HoloRouter fails? (Answer: total management plane isolation, no external access, requires recovery/failover plan). How would you extend this to multiple sites or teams? (Answer: VPN, bastion hosts, DNS replication, centralized access control). What are the security implications of DNS? (Answer: DNS spoofing, poisoning — DNSSEC, signed zones, audit logs).
How would you manage certificate lifecycle in production? (Answer: automated renewal via ACME, trusted PKI infrastructure, certificate pinning for critical services). Be ready to articulate the relationship between lab constraints (single HoloRouter) and production reality (redundant ToRs, anycast DNS, PKI infrastructure).

Requirements

  • All external workstations must reach nested management services (vCenter, SDDC Manager, NSX) with full feature access
  • DNS resolution must be reliable and consistent for FQDN-based access (enabling certificate validation)
  • HTTPS traffic to management UIs must be secure without disabling certificate validation (requiring VMCA root trust)
  • Access patterns must scale from individual labs to team-based deployments without architectural changes
  • Network isolation between physical host and nested environment must be maintained (no L2 bridging)

Constraints

  • HoloRouter is a single L3 gateway — failure of HoloRouter = total loss of external access to nested environment
  • Network routing must not conflict with existing corporate IP space; manual CIDR customization required before deployment
  • DNS forwarder (dnsmasq or BIND) has finite query capacity; thread 21 documents exceeding 150 concurrent queries
  • VMCA certificates must be manually imported on each external workstation — no enterprise PKI integration in lab
  • Static route configuration is OS-specific (Windows, Linux, macOS) and not automatically distributed to team members
  • HoloRouter's external interface IP must be reachable from all external workstations (same L2 broadcast domain or routed network)
  • Split-horizon DNS (same answer internal/external) means internal and external clients see the same IP — appropriate for labs but not multi-site designs

Assumptions

  • HoloRouter is always running and responding to ICMP, SSH, and DNS queries
  • External workstations have root/admin privilege to add static routes and import certificates
  • Team members have direct network connectivity to HoloRouter's external interface (no additional firewalls or gateways blocking traffic)
  • VMCA root certificate can be imported on all workstations without PKI policy restrictions
  • DNS resolution via HoloRouter does not conflict with corporate DNS policies or split-DNS configurations
  • Network maintenance windows and HoloRouter restarts are coordinated with team to minimize access loss

Risks

  • HoloRouter failure: IMPACT: total external access loss for entire team, MITIGATION: snapshot HoloRouter regularly, document recovery procedure, design dual-router failover (extension)
  • IP conflict (workstation network overlaps with Holodeck CIDR): IMPACT: routing ambiguity, intermittent connectivity, MITIGATION: pre-flight check IP space, use 172.16.x.x or 192.168.x.x if 10.x.x.x conflicts exist
  • DNS service overload (thread 21): IMPACT: nested services become unreachable (DNS timeouts), MITIGATION: monitor concurrent query count, implement DNS caching on workstations, add upstream DNS servers
  • Certificate not imported on new team member workstations: IMPACT: HTTPS browser warnings, poor user experience, MITIGATION: document VMCA import procedure, provide pre-configured CA bundle, use group policy (Windows) to auto-import
  • HoloRouter external NIC loses connectivity: IMPACT: no external access despite internal Holodeck remaining operational, MITIGATION: monitor external NIC interface, configure network health checks
  • VPN/bastion host required by team: IMPACT: direct access pattern insufficient, requires additional infrastructure, MITIGATION: plan bastion host upfront (extension), design VPN access as evolution not retrofit

Self-Assessment Discussion Prompts

  1. Why is the HoloRouter a single point of failure? In production, what architectural change would eliminate this risk? (Answer: redundant ToR switches, BGP ECMP, VRRP, anycast DNS)
  2. Explain the difference between split-horizon DNS (same answer everywhere) and conditional forwarding (different answers by origin). When would you use each? (Answer: split-horizon for single-site labs; conditional forwarding for multi-site or hybrid cloud)
  3. You're designing external access for a VCF environment shared by 10 engineers. Should you use: (a) direct routes, (b) VPN, (c) bastion host, or (d) all three? Justify your answer. (Answer: bastion host is minimum; VPN adds security for remote access; direct routes only for same-network admins)
  4. VMCA root certificate is imported on all workstations. What is the security risk? How would you mitigate in production? (Answer: Risk is compromised VMCA private key = all management UIs compromised. Mitigation: rotate certificates regularly, audit certificate chain, use hardware security modules for key storage)
  5. Design a multi-site Holodeck lab with two HoloRouters (Site A, Site B) that can reach each other. What additional routing, DNS, and certificate configurations are needed? (Answer: Site-to-site VPN or BGP peering, separate DNS zones or conditional forwarding per site, certificate with SANs covering both sites)
  6. A team member reports 'DNS works fine but https://vcenter.site-a.vcf.lab shows certificate mismatch.' Debug this. (Answer: DNS resolves to correct IP, but certificate CN doesn't match. Either certificate was issued with IP instead of FQDN, or browser is interpreting cert incorrectly. Solution: regenerate cert with FQDN CN, or access via IP and accept cert warning as workaround)

Extensions

Deploy a Dual-Site Holodeck Configuration with Site-to-Site Routing

Create two Holodeck instances (Site A and Site B) on separate physical hosts. Configure site-to-site routing via VPN or BGP peering between the two HoloRouters. DNS zones for each site resolve independently (site-a.site-a.vcf.lab vs site-b.site-a.vcf.lab). Teams can transparently access services on both sites. This exercises the complexity of production multi-site VCF designs and introduces stretched cluster / disaster recovery concepts.

much harder

Implement a Bastion Host with SSH Jump and Webtop Access

Create a dedicated bastion host VM on the physical host's external network. Configure SSH key-based authentication and document jump box procedures (ssh -J bastion management-service). Set up a Guacamole webtop on the bastion for team members without direct network access. Monitor and log all bastion access using auditd. This is a production-grade access pattern.

harder

Configure VPN Server on HoloRouter for Remote Access

Deploy a VPN server (OpenVPN or WireGuard) on the HoloRouter itself. Remote users connect to the VPN and gain access to nested services as if they were on the local network. Test VPN access from different network locations (corporate, home, public WiFi). Document VPN configuration, certificate management, and user onboarding. This mirrors production remote access patterns.

harder

Design and Document a Certificate Lifecycle Management Plan

Document how VMCA certificates will be rotated, renewed, and audited in your lab. Set up automated certificate export and distribution to team members (e.g., weekly email with updated VMCA root). Research ACME (Automated Certificate Management Environment) and how you could integrate Let's Encrypt or internal ACME server into the lab. This prepares you for production PKI discussions.

harder

Implement Conditional DNS Forwarding for Multi-Site Resolution

Extend the lab to use conditional DNS forwarding: *.site-a.site-a.vcf.lab -> HoloRouter Site A, *.site-b.site-a.vcf.lab -> HoloRouter Site B, other queries -> corporate DNS. Implement on workstations or on a central DNS server. Test that a client can seamlessly access services on both sites using FQDNs. This is a production technique for hybrid cloud and multi-cloud environments.

same

⚠ Known Pitfalls (from Community KB)

External Jumpserver Access RESOLVED
Problem: External workstation cannot resolve nested FQDNs or reach nested services despite adding static route.
Resolution: Add static route for IP range, configure DNS to HoloRouter or use hosts file entries, ensure both workstation and HoloRouter are on same network segment or routed network.
Generate and Install VMCA Certificate on SDDC Manager RESOLVED
Problem: VMCA certificate generation fails during VCF bring-up; certificate CN mismatch with FQDN.
Resolution: Ensure bring-up spec includes correct FQDN for each component; regenerate certificates with correct CN via SDDC Manager UI or API.
Maximum number of concurrent DNS queries reached (max: 150) OPEN
Problem: DNS service overloaded; queries timeout and nested services become unreachable.
Resolution: Increase dnsmasq cache-size and max-concurrent settings; add upstream nameservers; monitor query count with netstat.
Waiting for FRR service LIKELY_RESOLVED
Problem: HoloRouter FRR (BGP) service fails to start or hangs during Prepare phase.
Resolution: Restart FRR daemon; check systemctl status bgpd; verify /etc/frr/frr.conf syntax; review HoloRouter logs for BGP errors.
Issue with DNS resolution in Holodeck dual-site deployment OPEN
Problem: DNS resolution fails in multi-site setup where two HoloRouters must serve different domains.
Resolution: Implement conditional forwarding per site; ensure each site's DNS forwarder has correct upstream server; test resolution from both sites independently.
Holodeck 9.0.1 DNS troubleshooting INFORMATIONAL
Problem: Various DNS issues in nested environments (resolution failures, split-brain scenarios).
Resolution: Comprehensive troubleshooting guide covering dnsmasq config, upstream forwarders, zone configuration, and testing procedures.

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.