HoloRouter 9.1 Built-in Services Deep Dive — Vault, Authentik SSO, Technitium DNS, and the Authenticated Webtop Behind the HTTPS Reverse Proxy

VCF 9.1.0.0Intermediateadminarchitectvcdx⏱ 180 min

Written for the Holodeck 9.1 GA HoloRouter appliance (6 vCPU / 12 GB — sized up from the 9.0.x router specifically to carry this service stack). This is the 9.1 parallel twin of holodeck-04 (External Access, DNS, and Routing Configuration). The 9.0.x lab remains valid for 9.0.x routers and is NOT replaced: on 9.0.x the router ran dnsmasq/FRR with flat credential files and an unauthenticated Webtop; on 9.1 the same appliance carries HashiCorp Vault (:8200), Authentik SSO with SCIM (:9443), Technitium DNS (:5380), an authenticated Webtop (:30000), an internal Certificate Authority, and optionally GitLab — all behind an HTTPS reverse proxy with automated certificate management. Service ports, FQDNs, and default credentials below are quoted from the official versioned 9.1 docs (vmware.github.io/Holodeck/9.1/); anything the docs leave open is explicitly marked verify_live.

Objectives

Map the complete HoloRouter 9.1 service topology: which services exist, their direct ports (8200/9443/5380/30000), their vcf.lab FQDNs, and how the HTTPS reverse proxy fronts all of them
Administer Technitium DNS through its web console (:5380): zones, records, DHCP scopes, and upstream forwarders — and verify whether the 9.0-era upstream-forwarder gap is resolved by the dnsmasq-to-Technitium supersession
Authenticate to HashiCorp Vault (:8200) with the documented root token, inventory its secrets engines, and trace the automated-certificate relationship between Vault, the internal CA (ca.vcf.lab), and the reverse proxy
Explore Authentik SSO (:9443) as an OIDC/SAML/SCIM 2.0 identity provider, resolve the documented admin-username discrepancy on your live appliance, and bootstrap it with Initialize-Authentik
Use the authenticated Webtop (:30000) and characterize its two access paths — proxied HTTPS FQDN versus direct-port HTTP bypass — as a security observation
Contrast the 9.1 service model against the 9.0-era dnsmasq/FRR/flat-credential model it supersedes, producing a defense-ready comparison table for secrets, identity, DNS, and transport security

Prerequisites

HoloRouter 9.1 deployed with the GitOps stack provisioned and all four documented service ports answering over HTTPS (holodeck91-02 completed through its Task 3 liveness checks; snapshot 'holodeck91-02-complete' exists). A nested VCF estate is NOT required for Tasks 1-5; the Set-VCFSSOConfiguration half of Task 4 additionally requires a deployed Site A instance with VCF Operations reachable (holodeck91-03) and is explicitly marked deferrable if you have not deployed yet. Operator workstation must be able to use the HoloRouter as its DNS server (temporarily) to exercise FQDN-based service access.

Prior labs: holodeck91-02

Required skills:

Holodeck 9.1 HoloRouter deployment and reverse-proxy access pattern (holodeck91-02)
The 9.0-era HoloRouter service model — dnsmasq, FRR, config.json credentials, open Webtop (holodeck-04) — this lab constantly contrasts against it
Reading TLS certificates in a browser (issuer, SAN, chain) and importing a root CA
DNS fundamentals: zones, A/PTR records, forwarders, conditional forwarding
Conceptual familiarity with secrets managers (Vault) and identity providers (OIDC, SAML, SCIM) — hands-on experience not assumed
Basic kubectl usage (the router hosts services on a single-node Kubernetes cluster)

📸 Starting State: S3-91 — Management Domain Deployed (VCF 9.1.0.0)

Lab Environment

Single HoloRouter 9.1 appliance (6 vCPU / 12 GB / 75 GB, per the official 9.1 sizing) carrying the full infrastructure service stack. All services run HTTP internally and are exposed over HTTPS by the reverse proxy at stable vcf.lab FQDNs; direct ports remain reachable for four of them. The official docs state the service domain is ALWAYS vcf.lab regardless of any custom -DNSDomain — only nested VCF component FQDNs follow the custom domain. Services: HashiCorp Vault (vault.vcf.lab / :8200, secrets + automated certificates), Authentik SSO (auth.vcf.lab / :9443, OIDC/OAuth2/SAML 2.0/SCIM 2.0), Technitium DNS (dns.vcf.lab / :5380, replaces dnsmasq), Certificate Authority (ca.vcf.lab, certsrv extension emulating AD Certificate Services with a Vault backend), Webtop (webtop.vcf.lab / :30000, now authenticated), plus GitLab (gitlab.vcf.lab) and its registry when GitOps is enabled. Authentik runs inside the router's single-node Kubernetes cluster, deployed as part of Set-HoloRouter.

graph TB
  WS[Operator Workstation<br/>DNS -> HoloRouter] -->|HTTPS 443| RP[Reverse Proxy on HoloRouter 9.1<br/>automated SSL via internal CA]
  RP --> VAULT[Vault<br/>vault.vcf.lab / :8200]
  RP --> AUTH[Authentik SSO<br/>auth.vcf.lab / :9443]
  RP --> DNS[Technitium DNS<br/>dns.vcf.lab / :5380]
  RP --> WT[Webtop authenticated<br/>webtop.vcf.lab / :30000]
  RP --> CA[Certificate Authority<br/>ca.vcf.lab - certsrv + Vault backend]
  RP --> GL[GitLab - if GitOps enabled<br/>gitlab.vcf.lab]
  VAULT -.->|issues certs| CA
  CA -.->|automated SSL| RP
  AUTH -.->|OIDC + SCIM sync| OPS[VCF Operations ops-a.site-a.vcf.lab<br/>requires holodeck91-03 estate]

IP Addressing

Network	Purpose	VLAN
`HoloRouter external management IP`	All service access (FQDN via reverse proxy, or direct ports 8200/9443/5380/30000). This is also the DNS server IP your workstation should use for vcf.lab FQDN resolution.	External management port group (as mapped in holodeck91-02 Task 2)
`10.1.0.0/20`	Site A supernet the Technitium zones describe (site-a.vcf.lab). Present in DNS even before the nested estate is deployed.	Site A VLANs 0, 10-25 (9.1 documented defaults)

Credentials

System	Username	Password
HashiCorp Vault (vault.vcf.lab / :8200)	`(token auth — no username)`	Root token is the documented Holodeck master password (doubled form). Source: official 9.1 docs credential matrix. Treat the root token as bootstrap-only; note every place you end up using it.
Authentik SSO (auth.vcf.lab / :9443)	`akadmin per the 9.1 Getting Started page; admin per the 9.1 Release Notes page — a documented discrepancy you resolve on your live appliance in Task 4`	Documented master password (doubled form)
Technitium DNS (dns.vcf.lab / :5380)	`admin`	Documented master password (doubled form)
Webtop (webtop.vcf.lab / :30000)	`admin`	Documented master password (doubled form). Webtop is authenticated in 9.1 — the 9.0-era open-access model is gone.
GitLab (if GitOps enabled)	`root`	Documented master password (doubled form)
HoloRouter (SSH)	`root`	Set during OVA deployment in holodeck91-02 — no product default exists

Tasks

Task 1 Map the reverse proxy and the full service topology

security

Before administering any single service, establish the access architecture they all share. The 9.1 model has three layers that did not exist in 9.0: a reverse proxy terminating HTTPS, an internal CA issuing its certificates automatically, and a DNS dependency (your console must use the HoloRouter as its DNS server for FQDN access). A VCDX panelist reviewing 'HTTPS everywhere' will ask what actually terminates TLS, who issues the certificates, and what breaks when DNS is wrong — this task gives you first-hand answers.

Step 1

Point your workstation's DNS at the HoloRouter external management IP (or run lookups explicitly against it: nslookup vault.vcf.lab <holorouter-mgmt-ip>). Resolve all documented service FQDNs: vault.vcf.lab, auth.vcf.lab, dns.vcf.lab, webtop.vcf.lab, ca.vcf.lab, and (if GitOps is enabled) gitlab.vcf.lab and gitlab-registry.vcf.lab.

Every FQDN resolves to the HoloRouter management IP. The official docs are explicit that FQDN-based access requires your console to use the HoloRouter as its DNS server.

Record which FQDNs resolve BEFORE any nested estate exists. The service zone (vcf.lab) is provisioned by the router itself — this is your first evidence that infrastructure DNS and nested-estate DNS are separate concerns in 9.1.

Step 2

Confirm the dual access model for the four port-published services. For each, test both paths from a browser or curl: https://vault.vcf.lab and https://<holorouter-mgmt-ip>:8200; https://auth.vcf.lab and https://<holorouter-mgmt-ip>:9443; https://dns.vcf.lab and https://<holorouter-mgmt-ip>:5380; https://webtop.vcf.lab and the direct port :30000.

Both paths reach each service. Per the docs, all infrastructure services run HTTP internally and are exposed via HTTPS at the FQDNs — note carefully which direct-port paths are HTTPS and which are not (the docs specifically note http://<ip>:30000 bypasses the proxy for Webtop; you examine that in Task 5).

Record exactly what protocol each direct port speaks on YOUR build (verify_live). The FQDN/proxy path is the documented, certificate-managed path; treat direct ports as diagnostic backdoors, not the design.

Step 3

Inspect the TLS certificate on two proxied FQDNs (e.g. vault.vcf.lab and dns.vcf.lab): issuer, subject/SANs, validity period, and whether both chain to the same issuing CA.

Certificates issued by the appliance's internal CA (the ca.vcf.lab / Vault-backed authority) with per-service or wildcard SANs — record the actual layout. This is 'automated SSL certificate management' made concrete.

Compare against your holodeck91-02 Task 3 certificate notes. If the issuer differs from what you recorded then, the proxy re-issued certificates — note the validity window as a clue to the rotation policy (verify_live).

Step 4

Visit the Certificate Authority endpoint: https://ca.vcf.lab. The docs describe a certsrv extension emulating AD Certificate Services with a Vault backend. Explore what it offers (root CA download, certificate request forms) and download the root CA certificate.

CA web endpoint reachable; root CA certificate downloaded. The certsrv-style interface is the lab analog of the enterprise 'submit a CSR to the PKI' workflow.

This mirrors the 9.0-track VMCA import exercise (holodeck-04 Task 4), but the authority is now the router's own CA rather than VMCA — and it will also sign the nested estate's service certificates workflow later. Keep the file; you import it in step 5.

Step 5

Import the downloaded root CA into your workstation trust store (Windows: certmgr.msc > Trusted Root Certification Authorities; macOS: Keychain 'Always Trust'; Linux: /usr/local/share/ca-certificates + update-ca-certificates). Close and reopen the browser, then revisit two service FQDNs.

Service UIs now load with a clean lock and no certificate warnings.

Importing a root CA means trusting everything it signs. Acceptable for a lab CA you control; in production this decision goes through PKI governance. Note the thought for the design reflection.

Step 6

Build the service topology table in your findings log: [Service, FQDN, Direct port, Protocol on direct port, Auth model, Cert issuer, Runs where (verify with kubectl in Task 4)]. Seed it with the seven documented rows (Vault, Authentik, Technitium, Webtop, CA, GitLab, GitLab Registry).

Complete service topology table with every cell backed by an observation from steps 1-5, not by the docs alone.

Cells you cannot fill from observation stay marked verify_live. This table is the skeleton the rest of the lab fleshes out.

Validation Gate

Check: All documented FQDNs resolve via the HoloRouter; every service answers on both its FQDN and (where published) its direct port; root CA imported and at least two services load warning-free; topology table drafted.

Expected: Access architecture fully characterized: proxy, CA, DNS dependency, and the FQDN-vs-port duality all evidenced in the findings log.

Common Errors

Service FQDNs do not resolve from the workstation

Cause: Workstation is not using the HoloRouter as its DNS server, or queries go to corporate DNS first

Fix: Set the HoloRouter management IP as the DNS server on the active adapter (or use explicit nslookup <fqdn> <router-ip> for testing). The docs are explicit that FQDN access requires the router as your DNS. For a longer-term setup, configure a conditional forwarder for vcf.lab on your normal DNS instead of replacing it wholesale.

📋 KB: Official 9.1 docs — service access notes; carry-over of holodeck-04 Task 3 DNS patterns

https://gitlab.vcf.lab (and possibly other proxied FQDNs) unreachable even though direct ports answer

Cause: Known 9.1 first-boot issue: the reverse proxy's 80/443 iptables ACCEPT rules sit below a DROP rule

Fix: SSH to the router and move the port 80 and 443 ACCEPT rules above the DROP rule (documented workaround for GitHub issue #139); persist via the router's iptables save mechanism. Record whether your build was affected.

📋 KB: Holodeck GitHub issue #139 (workaround provided)

Certificate warnings persist after importing the root CA

Cause: Browser cache, or the service certificate chains to a different issuer than the CA you imported

Fix: Clear the browser SSL/state cache and retry. If the issuer genuinely differs per service, capture each chain — an inconsistent trust chain across services is a significant verify_live finding worth reporting.

📋 KB: Carry-over pattern from holodeck-04 Task 4 (9.0 track)

Task 2 Technitium DNS deep dive — and the dnsmasq supersession test

manageability

Technitium replacing dnsmasq is the most operationally consequential service change in 9.1: the official docs deprecate the HolodeckDNSConfig PowerShell cmdlets outright and declare the Technitium web UI the sole DNS management surface going forward. This task makes you fluent in that surface AND settles a question your own 9.0-era notes left open: the old router shipped without upstream forwarders (the documented one-time fix that Set-HoloRouter kept re-breaking). Does the 9.1 stack resolve that class of problem, or merely relocate it?

Step 1

Log in to the Technitium console at https://dns.vcf.lab (or :5380) as admin with the documented master password. Take inventory of the dashboard: query volume, blocked queries, configured zones.

Technitium admin console loads. Zones for vcf.lab (infrastructure services) and site-a.vcf.lab / site-b.vcf.lab (nested estate) visible — record exactly which zones exist pre-deployment on your build (verify_live).

Contrast the experience immediately: on 9.0 this inventory required SSHing to the router and reading /etc/dnsmasq.conf or kubectl-ing into dnsmasq pods. A web-managed DNS server with query analytics is a different operational tier.

Step 2

Open the vcf.lab zone and enumerate its records. Confirm the service FQDNs from Task 1 (vault, auth, dns, webtop, ca, gitlab) exist as records pointing at the router. Then open the site-a zone and note what is pre-provisioned for the nested estate (the 9.0 track pre-provisioned roughly 160 component records — check the 9.1 equivalent).

Service records confirmed in vcf.lab. Site A component records (vc-mgmt-a, sddcmanager-a, nsx-mgmt-a, esx-01a..., etc. under site-a.vcf.lab) present or generated at deployment time — record which.

Do not edit or delete pre-provisioned records while exploring. The deployment pipeline depends on them; Technitium makes destructive edits one click easy.

Step 3

THE SUPERSESSION TEST. Check Settings > Proxy & Forwarders (and the general recursion settings) for upstream forwarders. Then, from your workstation using the router as DNS, resolve an external name: nslookup broadcom.com <holorouter-mgmt-ip>.

Either external resolution works out of the box (forwarders or recursion pre-configured — the 9.0-era gap is closed) or it fails (the gap survives in new clothing). Record the result with evidence either way.

Your 9.0-era reference documented that both the router's resolv.conf and the dnsmasq configmap lacked upstream forwarders, and that the one-time fix was re-broken by every Set-HoloRouter run. The 9.1 release notes claim 'Fixed DNS lookup and configuration issues with Technitium DNS integration' — this step tests that claim rather than trusting it.

Step 4

If external resolution failed in step 3, configure forwarders in the Technitium UI (Settings > Proxy & Forwarders; use your lab's upstream DNS or public resolvers), save, and re-test. Then — critically — determine persistence: reboot the HoloRouter (or re-run Set-HoloRouter if you are prepared to accept its risks on a dedicated host) and check whether your forwarder setting survives.

Forwarders configured through the supported UI; persistence across reboot verified and logged. Whether Set-HoloRouter still overwrites DNS state under the Technitium model is a headline verify_live finding.

Only test the Set-HoloRouter interaction on a host you own exclusively. The 9.0-era shared-host destructiveness (global router state overwrite) is unresolved as a question under 9.1, and the 9.1 cmd reference still describes Set-HoloRouter in dnsmasq terms — a docs lag that itself tells you the interaction is unverified territory.

Step 5

Inspect DHCP: Technitium manages DHCP scopes in the same UI (the docs state DNS zones, records, AND DHCP scopes are now fully managed through the Technitium web UI). Enumerate configured scopes and map them to the Holodeck subnets you know from the 9.0 track.

DHCP scope inventory recorded. On 9.0 this was dnsmasq dhcp-range directives readable only from config files; on 9.1 it is a managed UI object.

Note whether scopes exist for networks that are not yet deployed — pre-provisioned scopes are further evidence of the config-generation model.

Step 6

Confirm the deprecation on the toolkit side: on the router in pwsh, run Get-Command DNSConfig and Get-Help Set-HoloRouter. Then check whether the old dnsmasq artifacts remain: ls /holodeck-runtime/dnsmasq/ and kubectl get pods -A | grep -i dns (kubeconfig at /etc/kubernetes/admin.conf).

Get-HoloDeckDNSConfig / Set-HoloDeckDNSConfig / Remove-HoloDeckDNSConfig absent or non-functional (deprecated per the 9.1 docs). Presence or absence of dnsmasq remnants (directory, pods) recorded — the docs' own Set-HoloRouter description still mentions DNSMASQ, so what actually serves DNS at runtime on your build is evidence worth capturing.

HONESTY GATE: do not write 'dnsmasq is gone' in your design notes unless you verified no dnsmasq process/pod serves anything on your live router. 'Technitium replaces dnsmasq' is the documented intent; the runtime reality on GA builds is yours to verify.

Step 7

Create and delete one test record to prove the management surface: add an A record test-91-04.vcf.lab pointing at the router IP, resolve it from your workstation, then delete it and confirm NXDOMAIN.

Record round-trip proven: create, resolve, delete, negative-resolve. You now know the change latency and cache behavior of the new DNS surface first-hand.

Note the TTL and whether deletion required a cache flush anywhere. Small operational facts like this are what separate 'I read about Technitium' from 'I operated it' in a defense.

Validation Gate

Check: Technitium console administered: zone and record inventory logged, forwarder/supersession test executed with a recorded verdict, DHCP scopes enumerated, DNS cmdlet deprecation confirmed, dnsmasq-remnant check performed, test record round-trip complete.

Expected: Fluency in the sole supported 9.1 DNS management surface, plus an evidence-backed answer to whether the 9.0-era upstream-forwarder problem class is closed.

Common Errors

External names fail to resolve through the router while vcf.lab names work

Cause: No upstream forwarders configured — the 9.0-era gap surviving into your 9.1 build, or a deliberate closed-resolver default

Fix: Add forwarders via Settings in the Technitium UI (the supported path — do NOT edit files on the router as you did on 9.0). Re-test, then verify persistence across reboot. Log the finding either way; it directly updates your team's network-fix reference for the 9.1 era.

📋 KB: Internal 9.0-era reference: holodeck holorouter + nested DNS fix (superseded-in-9.1 status pending this exact test)

Old Get/Set/Remove-HoloDeckDNSConfig scripts fail on the 9.1 router

Cause: The DNS cmdlet family is deprecated and unsupported in 9.1 — DNS is managed solely via the Technitium UI

Fix: Port any DNS automation to the Technitium HTTP API or accept UI-only management. Do not attempt to restore the old cmdlets; the docs are explicit that the UI is the sole management surface going forward.

📋 KB: Official 9.1 docs — Technitium replaces dnsmasq; DNS cmdlets deprecated

DNS changes made in Technitium disappear after router-level operations

Cause: Possible Set-HoloRouter or lifecycle-cmdlet overwrite of DNS state (the 9.0-era destructive pattern, unverified under 9.1)

Fix: Snapshot the router before any Set-HoloRouter/Reset-HoloRouter run; re-apply DNS settings from your findings log if overwritten; report the reproduction — this is exactly the shared-host risk your team needs characterized under 9.1.

📋 KB: Internal 9.0-era note: shared-holorouter destructiveness (re-validation open under 9.1)

Task 3 HashiCorp Vault — secrets management and the automated certificate chain

security

Vault on the router is the architectural answer to the 9.0 era's flat-credential model, where every password lived in config.json and every service shared the same doubled master password in plaintext files. This task establishes what the 9.1 appliance actually uses Vault FOR (the docs say secrets management plus automated certificate provisioning for all services), how its trust is bootstrapped, and — honestly — how far the lab implementation still is from production secrets discipline, since the documented root token IS the master password.

Step 1

Query Vault's health without authenticating: curl -k https://<holorouter-mgmt-ip>:8200/v1/sys/health (or browse it). Record the JSON verbatim: initialized, sealed, version.

Well-formed health JSON. Vault version string captured — the docs do not publish the bundled version, so this closes an open question for your environment. Seal status recorded.

If sealed=true, record it and continue to step 2 via the UI unseal prompt only if your build documents an unseal path. The 9.1 docs do not describe the unseal model (auto-unseal vs manual) — verify_live, and do not improvise key ceremonies on an appliance you may need to restore.

Step 2

Log in at https://vault.vcf.lab using Token auth with the documented root token (the doubled master password, per the official 9.1 credential matrix — no username).

Vault UI dashboard as root.

Pause on what you just did: the highest-privilege credential in the new secrets manager is the same well-known master password as everything else. That single fact anchors half of this lab's design reflection — write it down now.

Step 3

Inventory the secrets engines and auth methods: in the UI (or via CLI: vault secrets list and vault auth list if the vault binary is available on the router). Identify a PKI engine (certificate automation) and any KV engines holding Holodeck-related secrets.

Engine inventory recorded. Expect a PKI-related engine backing the internal CA (the docs describe the CA's certsrv extension as Vault-backed) — record its mount path, and what if anything the deployment pipeline stores in KV (verify_live).

HONESTY GATE: mount names, roles, and paths are undocumented — record what you SEE, and resist filling gaps with generic Vault defaults in your notes.

Step 4

Trace the certificate chain end to end: in the PKI engine, find the issuing CA and compare its certificate against (a) the root CA you downloaded from ca.vcf.lab in Task 1 and (b) the issuer of the reverse proxy's service certificates.

A documented chain: Vault PKI engine -> ca.vcf.lab authority -> reverse-proxy service certificates. If the chain differs on your build, diagram what you actually observed.

This is 'automated certificates' made auditable. In a defense, being able to name which component ISSUES, which STORES, and which SERVES a certificate is the difference between a feature claim and an architecture.

Step 5

Exercise the secrets surface safely: create a KV secret of your own (e.g. path holodeck-lab/findings, key test, a harmless value), read it back via UI and via API (curl with the token header), then delete it.

Secret round-trip proven through both UI and API. You now know the API access pattern the deployment stack would use for Vault-referenced secrets.

Cross-reference your holodeck91-02 Task 5 observation about where the Click-to-Deploy UI stores deployment secrets. If they were committed to Git in plaintext there, the existence of a working KV engine HERE sharpens the finding: the capability exists, unused. If they were Vault-referenced, find the actual path now.

Step 6

Review Vault's operational posture on the appliance: audit devices enabled? (vault audit list or UI), storage backend, and what happens to Vault across a router reboot — reboot if your window allows and re-check seal state and data.

Audit and persistence posture recorded. Seal-state-after-reboot is the key recoverability fact for the control plane (it determines whether a restored router serves secrets immediately or needs intervention).

Coordinate the reboot if anything else depends on the router. On a shared host, skip the reboot and mark the persistence row verify_live instead.

Validation Gate

Check: Vault health and version captured; root-token login performed; engine inventory recorded; certificate chain traced from PKI engine through ca.vcf.lab to the proxy certs; KV secret round-trip via UI and API; operational posture (audit, seal-after-reboot) logged or explicitly deferred.

Expected: Concrete understanding of what 9.1 uses Vault for, evidenced end to end — plus a sharply documented gap list between the lab's Vault posture and production secrets discipline.

Common Errors

Vault health endpoint returns sealed status and the UI demands unseal keys nobody has

Cause: Vault initialized sealed, or a crash-consistent restore left it sealed; the 9.1 unseal model is undocumented

Fix: Check the router for an auto-unseal mechanism or stored keys (systemd units, K8s secrets — observe, do not guess). If none is discoverable, restore the router from the holodeck91-02-complete snapshot and record the failure mode: 'Vault seal recovery undefined' is itself a defense-grade recoverability finding.

📋 KB: verify_live — Vault bootstrap/unseal model is an open question from the 9.1 release intake

Root token rejected

Cause: Token differs from the documented default on your build, or the doubled-password form was mistyped

Fix: Re-check the exact doubled form against the official 9.1 credential matrix. If genuinely different, check OVF properties and first-boot output for a generated token, and log the mechanism — a deviation from documented defaults is a significant finding.

📋 KB: Official 9.1 docs credential matrix

Task 4 Authentik SSO — identity, SCIM, and the Initialize-Authentik bootstrap

security

The 9.0 lab had no identity layer at all: every service had its own local admin and the same password. Authentik gives the 9.1 lab a real IdP speaking OIDC, OAuth2, SAML 2.0, and SCIM 2.0 — and the new Initialize-Authentik / Set-VCFSSOConfiguration cmdlet pair wires it into VCF SSO with SCIM provisioning. For a VCDX candidate this is a scaled-down rehearsal of enterprise identity design: one authority, federated applications, automated user lifecycle. It also carries a documented ambiguity (admin username) that you resolve empirically.

Step 1

Resolve the documented credential discrepancy: attempt login at https://auth.vcf.lab first as akadmin (per the 9.1 Getting Started page), then as admin (per the 9.1 Release Notes page), with the documented master password.

One of the two works. Record which, with a screenshot — you have just closed a discrepancy the official docs shipped with.

akadmin is Authentik's own conventional bootstrap superuser name, which is why the Getting Started value is the more likely one — but likelihood is not evidence; the login attempt is.

Step 2

Tour the admin interface: Directory > Users and Groups (what exists at bootstrap?), Applications > Providers and Applications (is anything pre-wired?), and System > Brands/Tenants.

Bootstrap inventory recorded: default users/groups, any pre-created providers or applications. Before Initialize-Authentik runs, expect little or no VCF-specific wiring — capture the true baseline (verify_live).

Explore read-only. Deleting or renaming bootstrap objects can strand the appliance's own admin access.

Step 3

Verify where Authentik runs: SSH to the router and inspect the single-node Kubernetes cluster — export KUBECONFIG=/etc/kubernetes/admin.conf; kubectl get pods -A. Identify the Authentik pods (server/worker and any bundled database/cache) and note their restart counts and resource requests.

Authentik workload located in the router's K8s cluster (the docs state it is deployed there as part of Set-HoloRouter). Pod inventory recorded — including what ELSE lives in that cluster, which fills the 'runs where' column of your Task 1 topology table.

Restart counts on first inspection are a free health signal: a crash-looping identity provider on a 12 GB appliance is a capacity smell worth logging early.

Step 4

Study the bootstrap cmdlet before running it: Get-Help Initialize-Authentik -Full on the router. Per the 9.1 docs it takes -AdminPassword, -Site, -UserPassword, and -BootstrapToken (all mandatory), creates the VCF Administrators group and the admin@vcf.lab user, registers the holodeck OIDC provider, binds it to the vcf application, and RETURNS an OIDC provider object (client ID + secret).

Parameter semantics understood and recorded; you can state before execution exactly which objects will appear in the Authentik admin UI afterwards.

The -UserPassword you choose becomes the login credential for admin@vcf.lab in VCF Operations later. Choose deliberately and record WHERE you stored it (ideally: the Vault KV engine from Task 3 — practice the discipline the platform now enables).

Step 5

Run the bootstrap and capture the returned object: $oidcProvider = Initialize-Authentik -AdminPassword '<authentik-admin-pw>' -Site a -UserPassword '<chosen-pw>' -BootstrapToken 'holodeck'. Then re-inspect the Authentik UI: Directory (VCF Administrators group, admin@vcf.lab user) and Applications (holodeck provider bound to the vcf application).

Cmdlet returns an OIDC provider object containing client ID and secret; all four expected objects now visible in the UI. Diff against your step 2 baseline — that diff IS the bootstrap, made auditable.

Treat $oidcProvider with secret discipline: it contains a client secret. Note its properties in your log by NAME only, never value.

Step 6

IF a deployed Site A estate exists (holodeck91-03 completed): complete the SSO integration with Set-VCFSSOConfiguration -Site a -Username admin -Password '<vcf-ops-admin-pw>' -BootstrapToken 'holodeck' -oidcProvider $oidcProvider. Per the docs this creates the SSO realm in VCF Operations, registers Authentik as OIDC/SCIM IdP, generates a SCIM sync token, triggers the initial SCIM sync, and assigns vcf_administrator to the synced group. Then log in to https://ops-a.site-a.vcf.lab as admin@vcf.lab. IF no estate exists yet: mark this step deferred, and revisit after holodeck91-03.

With an estate: VCF Operations login succeeds as admin@vcf.lab — authenticated by Authentik, authorized via the SCIM-synced group with vcf_administrator rights. Without: explicit deferral note in the findings log.

This step is the only part of the lab that touches the nested estate. Everything before it is router-local and safe on a control-plane-only environment.

Step 7

If step 6 ran: inspect the SCIM linkage from both ends. In Authentik, find the SCIM provider/mapping created for VCF; in VCF Operations, find the synced user and group and confirm the role assignment came from the sync rather than manual creation.

Documented two-sided evidence of SCIM provisioning: identity created once in the IdP, materialized in the relying application with correct role, no manual user creation anywhere.

This is the exact provisioning-deprovisioning story enterprise identity designs are defended on: when you disable admin@vcf.lab in Authentik, what happens in VCF Operations, and how fast? If time allows, test it — disable, observe, re-enable.

Validation Gate

Check: Admin username discrepancy resolved empirically; bootstrap baseline and post-Initialize-Authentik diff recorded; Authentik located in the router K8s cluster; OIDC provider object captured (names only); Set-VCFSSOConfiguration either completed with a working admin@vcf.lab login and two-sided SCIM evidence, or explicitly deferred pending holodeck91-03.

Expected: Working, evidenced understanding of the 9.1 identity layer from IdP bootstrap through (optionally) SCIM-provisioned VCF SSO.

Common Errors

Neither akadmin nor admin logs in to Authentik

Cause: Build deviates from both documented values, or Authentik pods are unhealthy

Fix: Check pod health first (kubectl get pods -A on the router — a crash-looping server pod presents as auth failure). If pods are healthy, check for an OVF-property or generated-credential mechanism and document whichever works; you are now the primary source for your environment.

📋 KB: Official 9.1 docs carry the akadmin-vs-admin discrepancy between pages; resolution is per-build evidence

Initialize-Authentik fails partway leaving some objects created

Cause: Partial bootstrap — e.g. group created but provider binding failed (token or connectivity issue mid-run)

Fix: Diff the UI against your step 2 baseline to enumerate exactly what was created, remove the partial objects, and re-run with corrected parameters. Do not run it twice blindly on top of partial state.

📋 KB: verify_live — idempotency of the bootstrap cmdlets is undocumented

Set-VCFSSOConfiguration succeeds but admin@vcf.lab cannot log in to VCF Operations

Cause: SCIM initial sync incomplete, or the OIDC provider object passed was stale (from an earlier bootstrap run)

Fix: Re-check the sync status from the VCF Operations IAM side, confirm the client ID in VCF Operations matches the CURRENT provider in Authentik, and re-run the configuration with a freshly captured $oidcProvider if they diverge.

📋 KB: Official 9.1 docs — Set-VCFSSOConfiguration flow

Task 5 The authenticated Webtop — and the proxy-bypass observation

security

Webtop is the smallest service but carries a disproportionate security lesson. On 9.0 it was an open, unauthenticated browser desktop with implicit full access to the nested estate — a bastion host with no door. 9.1 puts authentication on it and ships Firefox pre-loaded with bookmarks and saved passwords for every nested component. The docs ALSO note the direct port bypasses the proxy over plain HTTP. Characterizing that honestly — hardened front door, documented side door — is precisely the kind of finding a panel expects an architect to surface unprompted.

Step 1

Access Webtop by the designed path: https://webtop.vcf.lab. Authenticate as admin with the documented master password.

Authentication challenge presented BEFORE the desktop loads (the 9.1 change), then the browser-accessible desktop environment.

Note the auth mechanism you actually see: a Webtop-local login, a proxy-level auth gate, or an Authentik-federated redirect. Which layer owns Webtop authentication is undocumented (verify_live) and matters for the identity story from Task 4 — one authority or yet another local account?

Step 2

Inside Webtop, open Firefox and inspect what ships preconfigured: bookmarks for the nested VCF component URLs, and Settings > Passwords for the saved credential set (a documented 9.1 convenience feature).

Pre-loaded bookmarks and saved passwords for nested component logins present.

Register the design tension: the platform added authentication to Webtop and simultaneously concentrated every nested-estate credential inside it. Webtop's password is now effectively the key to everything it stores. Log it for the reflection — convenience features ARE security decisions.

Step 3

Test the documented bypass: from your workstation, open http://<holorouter-mgmt-ip>:30000 (the docs explicitly note this direct path bypasses the proxy). Record protocol, whether authentication still gates access, and any warning behavior.

Direct-port behavior characterized on your build: HTTP vs HTTPS, authenticated vs open. Whatever you find, you now know the real exposure of the side door rather than assuming the front door's properties apply.

HONESTY GATE: the docs say the direct port bypasses the proxy; they do not say it bypasses authentication. Test, record, and quote only your observation.

Step 4

From inside Webtop, verify its operational role for later labs: resolve and reach the service FQDNs (vault.vcf.lab, dns.vcf.lab) and — if an estate exists — a nested component URL. Webtop sits inside the pod network, so it reaches things your workstation may not.

Webtop confirmed as the in-pod bastion: service FQDNs resolve and load from inside without any workstation routing or DNS configuration.

This is the 9.1 answer to the holodeck-04 bastion-host design exercise: the bastion is now a first-class, authenticated platform feature rather than a pattern you had to build.

Step 5

Check the known Webtop defect: if you encounter the error 'Cannot read properties of undefined (reading lastActiveAt)', you have hit the open 9.1 issue; record the exact reproduction and continue via a fresh session.

Either clean operation, or a documented reproduction of the known issue with your session details added to the findings log.

No workaround is published for this issue at authoring time. A fresh browser session or router service restart are the pragmatic recoveries to try — record which worked.

Validation Gate

Check: Webtop reached via authenticated proxied FQDN; auth layer identified (or marked verify_live); Firefox bookmark/password preload confirmed; direct-port bypass behavior characterized with protocol and auth status; in-pod reachability proven; known-issue status recorded.

Expected: Complete picture of the authenticated Webtop as both hardened bastion and credential concentration point, including the honest characterization of the direct-port path.

Common Errors

Webtop UI throws 'Cannot read properties of undefined (reading lastActiveAt)'

Cause: Known open defect in the 9.1 authenticated Webtop stack (GitHub issue #142)

Fix: Start a fresh browser session; if persistent, restart the Webtop service on the router (identify its pod/unit via kubectl or systemctl — record which it is on your build). No official workaround published; your reproduction notes are the KB.

📋 KB: Holodeck GitHub issue #142 (open, no workaround)

Webtop reachable on :30000 but webtop.vcf.lab fails

Cause: Reverse-proxy route or DNS record issue while the underlying service is healthy

Fix: Confirm the FQDN resolves (Task 1), then check the proxy: if other FQDNs work and only Webtop's fails, capture the proxy config for that vhost. If NO FQDNs work, revisit the iptables first-boot issue from Task 1.

📋 KB: Related pattern: GitHub issue #139 (proxy ports blocked on first boot)

Task 6 Consolidate: the 9.0-vs-9.1 service model comparison and control-plane snapshot

manageability

The raw material from Tasks 1-5 becomes defense material only when consolidated into explicit before/after architecture statements. This task produces the comparison table you will actually use in a design discussion — secrets, identity, DNS, transport, access — each row backed by something you observed, then protects the now-fully-explored control plane with a snapshot.

Step 1

Build the supersession table with one row per concern, columns [Concern, 9.0 model, 9.1 model, Evidence (task/step), Gaps remaining]: Secrets (config.json flat files -> Vault, but root token = master password), Identity (per-service local admins -> Authentik OIDC/SCIM, pending Webtop auth-layer finding), DNS (dnsmasq files + deprecated cmdlets -> Technitium UI/API, forwarder-persistence finding), Certificates (per-service self-signed -> internal CA + Vault PKI + automated proxy certs), Transport (mixed HTTP -> HTTPS everywhere via proxy, minus the :30000 bypass finding), Bastion access (open Webtop -> authenticated Webtop with credential preload).

Six-row table, every 9.1 cell citing your own task/step evidence, every gap stated plainly.

The Gaps column is the most valuable one in a defense. 'The platform improved X but left Y' is an architect's sentence; 'the platform improved X' is a marketing sentence.

Step 2

Reconcile the open questions you carried in from holodeck91-02 Task 6: Technitium/dnsmasq coexistence (closed by your Task 2 step 6 evidence?), Authentik default credentials (closed by Task 4 step 1?). Update each with closed/open status and evidence links.

Release-intake open questions updated: the two carried questions resolved for your environment or explicitly still open with what remains unknown.

Never mark closed on inference — closed means observed on the live appliance with captured evidence.

Step 3

Store the credentials and secrets discipline artifacts properly: the admin@vcf.lab password and any tokens you generated go into the Vault KV engine (Task 3), NOT into your findings log. The findings log records retrieval mechanisms and paths only.

Zero secret values in the findings log; Vault KV path(s) recorded as the retrieval mechanism.

You spent this lab documenting the gap between flat-file credentials and managed secrets. Do not end it by writing passwords into a markdown file.

Step 4

Snapshot the router: prefer a clean guest shutdown, then Get-VM -Name '<holorouter-91-vm-name>' | New-Snapshot -Name 'holodeck91-04-complete' -Description 'Service stack explored: Technitium forwarders set, Authentik bootstrapped, CA trusted, findings logged' -Memory:$false. Power on and verify one service (e.g. Technitium login and your forwarder setting intact) after boot.

Boot-verified snapshot capturing the post-exploration service state, including your Technitium forwarder configuration and the Initialize-Authentik objects.

GitLab, Vault, and Authentik's database all dislike crash-consistent snapshots — the clean-shutdown preference from holodeck91-02 applies with more force now that three stateful services live here.

Validation Gate

Check: Six-row supersession table complete with evidence and gaps; carried open questions reconciled; secrets stored in Vault with only mechanisms in the log; snapshot 'holodeck91-04-complete' taken and boot-verified.

Expected: Defense-ready before/after architecture record and a protected control plane reflecting all of this lab's configuration.

Common Errors

After snapshot revert, Authentik or Vault misbehaves (login loops, sealed Vault)

Cause: Crash-consistent snapshot captured stateful services mid-write

Fix: Restore, then prefer clean-shutdown snapshots going forward. Record the recovery behavior of each service — control-plane recoverability evidence feeds directly into this lab's design reflection.

📋 KB: Carry-over from holodeck91-02 Task 6 guidance; general Vault/GitLab operational practice

Final Validation

Every built-in service on the HoloRouter 9.1 has been explored, validated, and used: the reverse proxy and internal CA characterized with the root CA trusted; Technitium DNS administered through its sole supported surface with the dnsmasq supersession and forwarder-persistence questions answered empirically; Vault authenticated, inventoried, and traced through the automated certificate chain; Authentik bootstrapped via Initialize-Authentik (and optionally wired into VCF SSO with SCIM); the authenticated Webtop used and its direct-port bypass honestly characterized. The consolidated 9.0-vs-9.1 supersession table, with evidence and remaining gaps, is the lab's defense artifact.

✓ All service FQDNs resolve via the router; services answer on FQDN and documented direct ports (8200/9443/5380/30000) → Full topology table with observed protocol and auth per path

✓ Root CA from ca.vcf.lab imported; service UIs load without certificate warnings → Trusted chain: Vault PKI -> internal CA -> proxy certificates, as observed

✓ Technitium: zones/records/DHCP inventoried, forwarder test verdict recorded, DNS cmdlet deprecation and dnsmasq-remnant check done → Evidence-backed answer to the 9.0-era DNS gap under 9.1

✓ Vault: version and seal state captured, engines inventoried, KV round-trip via UI and API → Working secrets surface, documented root-token caveat

✓ Authentik: admin username resolved empirically, Initialize-Authentik run with before/after diff; Set-VCFSSOConfiguration done or explicitly deferred → Bootstrapped IdP; SCIM-provisioned VCF SSO if estate exists

✓ Webtop: authenticated access proven, credential preload noted, :30000 bypass characterized → Honest two-path security characterization

✓ Snapshot 'holodeck91-04-complete' boot-verified; supersession table complete → Protected control plane and defense-ready comparison artifact

Cleanup / Restore

Snapshot: holodeck91-04-complete

• Snapshot the HoloRouter as 'holodeck91-04-complete' after clean shutdown (done in Task 6)

• Revert your workstation DNS to its normal configuration (or leave a vcf.lab conditional forwarder in place and document it) — do not leave the lab router as your primary resolver

• Remove the test DNS record and test Vault secret if any exploration artifacts remain (Tasks 2 and 3 round-trips should have cleaned themselves — verify)

• Sign out of all service UIs; do not leave the Vault root token session or Authentik admin session open in a browser

• File the supersession table and findings log with your VCDX evidence; update the team's 9.0-era DNS-fix and shared-router notes with your 9.1 findings (as 9.1-scoped annotations — never overwrite the 9.0 content)

• If on a shared host, brief co-tenants on any router-level changes you made (forwarders, iptables reorder) since router state is shared blast radius until proven otherwise under 9.1

Design Reflection (VCDX)

This lab hands a candidate three defensible architecture narratives, each with a trap for the unwary. (1) SECRETS: the move from flat config.json credentials to Vault is the right architectural direction — centralized storage, API access, audit capability, PKI automation — but the lab implementation bootstraps Vault with a root token equal to the well-known master password.

A panelist will let you praise Vault and then ask 'so what is your effective secret hierarchy?' The strong answer acknowledges that a secrets manager whose root credential is the universal password provides secrets MANAGEMENT without secrets ISOLATION, and describes the remediation path (rotate root, per-consumer AppRole/policy scoping, audit devices on). (2) IDENTITY: Authentik with OIDC and SCIM provisioning replaces per-service local admins with a single authority and automated lifecycle — the enterprise pattern in miniature.

The probe is failure-mode reasoning: the IdP lives on the same single appliance as DNS, secrets, and the deployment pipeline, so identity availability is now coupled to the lab's entire control plane; also note which services genuinely federate versus which retain local admin back doors (your Webtop auth-layer finding), because a directory that covers 80% of services changes operations less than it appears to.

(3) DNS SUPERSESSION: Technitium replacing dnsmasq is a study in how platforms retire operational surface — the cmdlets are deprecated, the UI/API is the sole surface, and your forwarder-persistence test determines whether the old failure class died or moved. The general lesson to articulate: when a platform supersedes a component, the architect's job is to re-verify every workaround and operational assumption attached to the old component, then formally retire them — carrying a dnsmasq-era runbook into a Technitium-era environment is drift in documentation form.

Across all three, the meta-point a panel rewards: 9.1 hardened the front doors (HTTPS, authentication, SSO) while the side doors (direct ports, root tokens, preloaded passwords) remain — and you found them yourself.

Requirements

All HoloRouter services must be reachable over HTTPS at stable vcf.lab FQDNs with certificates chaining to a single importable root CA
DNS for both the service zone (vcf.lab) and nested-estate zones must be manageable through a supported surface, including upstream forwarders that persist across router lifecycle operations
Secrets generated during lab operation (SSO passwords, tokens) must be stored in Vault, not in flat files or findings logs
Identity for VCF Operations must be provided by Authentik via OIDC with SCIM-provisioned users and groups (once an estate exists)
Interactive access to the pod network must require authentication (Webtop), with the unproxied access path explicitly characterized rather than ignored

Constraints

The service domain is fixed at vcf.lab regardless of -DNSDomain — only nested component FQDNs follow a custom domain (documented 9.1 behavior)
The DNS cmdlet family (Get/Set/Remove-HoloDeckDNSConfig) is deprecated and unsupported; Technitium UI/API is the sole DNS management surface
All services share one appliance (6 vCPU / 12 GB): one failure domain and one resource ceiling for secrets, identity, DNS, CA, bastion, and pipeline
Official 9.1 docs carry internal discrepancies (Authentik admin username differs between pages; Set-HoloRouter description still references dnsmasq) — every such point must be field-verified before entering design documentation
Community KB for the 9.1 service stack is embryonic: three relevant GitHub issues (#139 proxy firewall, #142 Webtop, #65 CA trust) and no forum corpus — novel failures have no external answers yet
Shared-host router operations remain governed by the unresolved 9.0-era destructiveness question until explicitly re-validated under the Technitium/proxy model

Assumptions

Documented default credentials (master password in doubled form across services, Vault root token) match the GA build — verified empirically per service during this lab
The reverse proxy and internal CA are provisioned functional by first boot, modulo the known iptables rule-ordering issue
Technitium serves all runtime DNS on the 9.1 appliance (dnsmasq remnants, if present, are inert) — tested in Task 2 rather than assumed
Initialize-Authentik and Set-VCFSSOConfiguration are safe to run once each on a fresh appliance; idempotency on re-run is undocumented and not assumed
Snapshot-based recovery from clean shutdown restores all stateful services (Vault, Authentik, GitLab) to a consistent state

Risks

Control-plane concentration deepens: secrets + identity + DNS + CA + bastion + pipeline on one 12 GB appliance — IMPACT: a single failure simultaneously removes authentication, name resolution, secret retrieval, and deployment capability; MITIGATION: boot-verified snapshots after every configuration milestone, documented rebuild-from-OVA path, findings log as the rebuild reference
Root-token-as-master-password: the secrets manager's highest credential is the universally known lab password — IMPACT: Vault provides no effective isolation; anyone with the lab password owns every secret it will ever hold; MITIGATION: document as a known lab-vs-production gap; optionally rotate the root token and create scoped policies as an extension, recording the procedure
Credential concentration in Webtop: preloaded Firefox passwords make one authenticated desktop the key to the entire nested estate — IMPACT: Webtop credential compromise = estate compromise; MITIGATION: treat the Webtop password with master-password discipline; characterize and, if possible, restrict the :30000 bypass path
DNS single-authority failure: Technitium down means service FQDNs, nested-estate resolution, and (if scoped) DHCP all fail together — IMPACT: broad outage with confusing symptoms (everything 'unreachable'); MITIGATION: know the direct-port access paths as break-glass, monitor the DNS pod/service, snapshot before DNS changes
Docs-vs-build divergence: acting on a documented value that differs on the GA build (admin username, credential defaults, dnsmasq remnants) — IMPACT: fabricated documentation that fails defense scrutiny or misleads the team; MITIGATION: the lab's verify-live discipline — every doc claim exercised empirically before entering the findings log
Deprecated-surface drift: 9.0-era DNS automation or runbooks reused against a 9.1 router — IMPACT: scripts fail or, worse, partially apply; MITIGATION: formal retirement of the DNS cmdlet runbooks for 9.1 environments, replaced by Technitium API equivalents, with version-scoped documentation kept separate per the coexistence policy

Self-Assessment Discussion Prompts

The 9.0 model stored every credential in config.json; 9.1 ships Vault whose root token is the same master password. Argue whether the security posture has actually improved, and specify the minimum set of changes that would make the Vault deployment meaningfully better than the flat files.
Authentik gives the lab one identity authority — running on the same appliance as DNS, secrets, and the deployment pipeline. In production you would never co-locate these. Enumerate what you would separate first, second, and third, and justify the order with failure-mode reasoning.
SCIM provisioning created admin@vcf.lab in VCF Operations from the IdP. Walk a panel through the deprovisioning story: an administrator leaves, you disable them in Authentik — what happens, how fast, and what residual access paths (local admins, saved Webtop passwords, Vault tokens) survive the deprovisioning?
Technitium supersedes dnsmasq and the DNS cmdlets are deprecated. Describe your process for retiring the 9.0-era DNS runbooks: what do you re-verify, what do you rewrite, and how do you prevent a colleague from applying the old dnsmasq fix to a 9.1 router during an incident?
The platform claims HTTPS everywhere, and you found the :30000 path that bypasses the proxy. How do you report a finding like this in a design document without either overstating it (it is a lab appliance) or burying it? Draft the exact risk statement you would write.
The internal CA emulates AD Certificate Services with a Vault backend, and you imported its root into your trust store. What is the blast radius if that CA's key is compromised, and how does your answer differ between this lab and a production PKI with the same architecture?
One appliance now carries six security-relevant services. A panelist asks: 'What is your monitoring minimum for this control plane?' Name the five signals you would watch and the failure each one catches first.

Extensions

Vault Hardening Pass

Rotate the Vault root token, enable an audit device, and create a scoped policy plus AppRole for a hypothetical pipeline consumer. Document each step and its effect on the deployment stack (does anything break when the root token changes? — that answer reveals what actually consumes Vault). This converts the lab's biggest documented gap into a practiced remediation.

harder

Federate a Second Application Through Authentik

Wire one more service into Authentik yourself — e.g. protect Technitium or GitLab behind an Authentik OIDC/proxy provider. You built SSO consumption in the lab's guided path; building a federation yourself proves you understand providers, applications, and bindings rather than just the bootstrap cmdlet.

harder

Technitium API Automation

Recreate the retired DNS cmdlet capabilities against the Technitium HTTP API: scripted record create/read/delete and a forwarder-configuration function, with token-based auth stored in Vault. Produces the 9.1-era replacement for your team's 9.0 DNS automation and closes the deprecation gap with working code.

same

Break the Control Plane on Purpose

With the holodeck91-04-complete snapshot proven, induce failures one at a time — stop the Technitium service, kill the Authentik pods, seal Vault — and record the symptom each produces from an operator's perspective (what LOOKS broken vs what IS broken). Builds the diagnostic index for a service stack with no community KB yet.

harder

Shared-Host Re-Validation Under the Service Stack

On a host carrying another Holodeck instance, run the coordinated experiment: does Set-HoloRouter on the 9.1 appliance still overwrite global DNS/routing state now that Technitium and the reverse proxy replaced dnsmasq/direct services? This closes the standing multi-tenant safety question your team has carried since the 9.0 era.

much harder

⚠ Known Pitfalls (from Community KB)

[9.1 known issue] Reverse proxy 80/443 blocked by iptables ordering on first boot OPEN (workaround provided)

Problem: Proxied FQDNs (notably gitlab.vcf.lab) unreachable after fresh HoloRouter boot because the ACCEPT rules for 80/443 sit below a DROP rule.

Resolution: Move the port 80 and 443 ACCEPT rules above the DROP rule and persist; verify all service FQDNs afterwards. (GitHub #139)

[9.1 known issue] Authenticated Webtop throws lastActiveAt undefined error OPEN

Problem: The new authenticated Webtop stack can fail with 'Cannot read properties of undefined (reading lastActiveAt)'; no workaround published.

Resolution: Fresh session or Webtop service restart; capture the reproduction — your notes are the seed KB. (GitHub #142)

[9.1 docs discrepancy] Authentik admin username differs between official pages OPEN until resolved per build

Problem: Getting Started documents akadmin; Release Notes documents admin. Both cannot be right for one build.

Resolution: Resolve empirically at first login (Task 4 step 1) and record the answer with evidence.

[9.1 docs lag] Set-HoloRouter description still references DNSMASQ OPEN

Problem: The cmd reference describes Set-HoloRouter configuring dnsmasq although Technitium is the documented DNS server — either docs lag or a runtime remnant.

Resolution: Check for dnsmasq processes/pods and the /holodeck-runtime/dnsmasq/ directory on the live router (Task 2 step 6); record what actually serves DNS.

[Carried 9.0-era question] Shared-host router state destructiveness under the new stack OPEN under 9.1

Problem: On 9.0, Set-HoloRouter overwrote global dnsmasq/FRR state and broke co-tenants. Whether Technitium + reverse proxy changes this is unverified.

Resolution: Treat router-level operations as destructive on shared hosts until the re-validation extension is executed; snapshot co-tenant routers first.

[Carried 9.0-era issue] HoloRouter-to-vCenter connections blocked by CA trust OPEN

Problem: Router connections to vCenter-managed datastores can fail on certificate trust (GitHub #65) — now more visible since the 9.1 router carries its own CA and trust store.

Resolution: See the 9.0.x reference for detail; on 9.1 verify the router's trust store handling when using vCenter targets.

References

Holodeck 9.1 Documentation — Getting Started (services, FQDNs, credentials)Tier 1 — Official
Authoritative source for the service table (ports, FQDNs, credentials), the vcf.lab domain rule, the Technitium/dnsmasq supersession statement, and the Initialize-Authentik / Set-VCFSSOConfiguration cmdlets. Use the versioned /9.1/ URLs only — the un-versioned pages serve stale legacy content.
HoloDeck 9.1 Release Announcement (verbatim capture) Holodeck-9.1/01-release-announcement.mdlocal fileTier 1 — Official
Primary capture for the four service ports (8200/9443/5380/30000), HTTPS-everywhere claim, and SCIM mention.
HoloDeck Toolkit 9.1 — Structured Reference & Changelog Holodeck-9.1/02-structured-changelog.mdlocal fileTier 1 — Official
Carries the section 7 open questions this lab closes for Technitium/dnsmasq coexistence and Authentik defaults.
VCF 9.1 Release Intake — rn-9-1-017 (HoloRouter built-in services) academy-v2/content/releases/vcf-9.1.jsonlocal fileTier 1 — Official
Canonical changelog entry for this lab: services behind the HTTPS reverse proxy, re-validation flags for the 9.0-era DNS fix and shared-router notes.
Holodeck GitHub Issues #139, #142, #65Tier 1 — Official
The three 9.1-relevant service-stack defects at authoring time: proxy 80/443 iptables ordering on first boot (#139, workaround provided), authenticated Webtop lastActiveAt error (#142, open), router-to-vCenter CA trust (#65, carried from 9.0 era).
holodeck-04 — External Access, DNS, and Routing Configuration (9.0 twin) holodeck-04.jsonlocal fileTier 2 — VMware Press
The superseded model this lab contrasts against: dnsmasq/FRR, VMCA import workflow, bastion design exercise. Remains authoritative for 9.0.x routers.
HashiCorp Vault DocumentationTier 2 — VMware Press
Reference for secrets engines, auth methods, PKI engine, audit devices, and seal/unseal concepts used in Task 3 and the hardening extension.
Authentik DocumentationTier 2 — VMware Press
Providers, applications, SCIM provisioning, and the akadmin bootstrap convention referenced in Task 4.
Technitium DNS Server DocumentationTier 2 — VMware Press
Web console, HTTP API (used in the automation extension), forwarders, zones, and DHCP scope management.
SCIM 2.0 — RFC 7644 (System for Cross-domain Identity Management)Tier 3 — Expert Blog
The provisioning protocol behind the Authentik-to-VCF-Operations user sync — background for the identity design_reflection prompts.

Mark complete