Academy/Holodeck Lab Setup & Operations/Holodeck Instance Lifecycle Management
This lab targets VCF 9.0.2

Holodeck Instance Lifecycle Management

VCF 9.0.2Intermediateadminarchitectvcdx⏱ 90 min

Lifecycle concepts apply across VCF 9.0.0 through 9.0.2. Side-by-side testing in Task 4 can use VCF 5.2.x content manifest for version comparison.

Objectives

  • Design and execute a multi-checkpoint snapshot strategy for Holodeck lab environments
  • Execute Remove-HoloDeckInstance and verify complete cleanup of nested infrastructure
  • Redeploy from an existing config.json and measure redeploy time vs snapshot revert time
  • Version-control config.json templates and demonstrate config overlay patterns for different VCF versions
  • Deploy and compare side-by-side VCF 5.2.x and VCF 9.0.x instances to articulate version-specific architectural differences
  • Automate Holodeck operations via PowerShell and document best practices for lab reset and scheduling

Prerequisites

Holodeck pod with VCF 9.0.2 management domain deployed and healthy (holodeck-02 completion snapshot available). Sufficient datastore space for 2+ snapshots per VM (minimum 200 GB free). PowerShell access to Holodeck Toolkit runtime directory.

Prior labs: holodeck-02

Required skills:

  • Holodeck instance deployment and basic PowerShell scripting
  • vSphere VM snapshot creation and revert operations
  • Understanding of VCF component versioning and manifest structure
  • File system navigation and JSON editing
📸 Starting State: S3-52 — Management Domain Deployed (VCF 5.2)

Lab Environment

Primary Holodeck pod (VCF 9.0.2) from holodeck-02. Secondary instance (VCF 5.2.x) deployed in Task 4 on the same physical host using separate config. Both pods isolated via HoloRouter networking (Site-A: 10.1.1.0/20, Site-B: 10.10.0.0/20 if dual-site comparison is performed).

graph TB
  PHY[Physical ESXi Host 512+ GB RAM] --> HR[HoloRouter-A 10.1.1.1]
  PHY --> HR2[HoloRouter-B 10.10.0.1]
  HR --> MGMT-A[VCF 9.0.2 Instance]
  HR2 --> MGMT-B[VCF 5.2.x Instance]
  MGMT-A --> SNAP1[Snapshot: post-deploy]
  MGMT-A --> SNAP2[Snapshot: post-config]
  MGMT-A --> SNAP3[Snapshot: post-cleanup]

IP Addressing

NetworkPurposeVLAN
10.1.1.0/20Site-A primary instance (VCF 9.0.2)VLAN 1644
10.10.0.0/20Site-B secondary instance (VCF 5.2.x, optional)VLAN 1650

Credentials

SystemUsernamePassword
Physical ESXi HostrootConfigured during ESXi installation
HoloConsole-AholodeckDefault Holodeck password from Toolkit docs
SDDC Manageradministrator@vsphere.localFrom VCF bring-up spec. API authentication uses doubled password: VMware123!VMware123!
vCenteradministrator@vsphere.localFrom VCF bring-up spec

Tasks

Task 1 Design and execute a multi-checkpoint snapshot strategy

recoverability

Snapshots are your lab reset button. In production, they're not allowed, but in a test environment, snapshots enable rapid rollback after experiments fail. VCDX panelists expect you to articulate when to snapshot (post-major-milestone), what to snapshot (which VMs), and what trade-offs exist (memory vs non-memory, disk overhead, consolidation timing). This task builds operational discipline.

Step 1

Open PowerShell on the HoloConsole or your physical host management workstation. Set the working directory to the Holodeck runtime: cd C:\Holodeck\holodeck-runtime

PowerShell prompt at C:\Holodeck\holodeck-runtime
Step 2

Review the current instance state: Get-HoloDeckInstance | Format-Table -Property InstanceId, State, VcfVersion

Instance with State: Running, VcfVersion: 9.0.2
Step 3

List all Holodeck VMs currently deployed: Get-VM -Name 'Holo-*' | Format-Table -Property Name, MemoryMB, NumCpu, ProvisionedSpaceGB

Output shows HoloRouter, HoloConsole, nested ESXi hosts, Cloud Builder, SDDC Manager, vCenter, NSX Manager
Step 4

Check datastore free space to estimate snapshot overhead: Get-Datastore | Where-Object {$_.Name -like 'holodeck'} | Format-Table -Property Name, FreeSpaceGB, CapacityGB

Datastore showing >200 GB free (sufficient for multi-snapshot strategy)
Step 5

Create a non-memory snapshot at the post-deployment milestone: Get-VM -Name 'Holo-*' | New-Snapshot -Name 'lcm-post-deploy-9.0.2' -Description 'Post VCF 9.0.2 management domain deployment' -Memory:$false -Quiesce:$false

Snapshots created on all VMs
Non-memory snapshots are much faster to revert (seconds) vs memory-inclusive snapshots (minutes) but require guest OS reboots after revert. For lab work, non-memory is preferred.
Step 6

Verify snapshot creation: Get-VM -Name 'Holo-*' | Get-Snapshot | Format-Table -Property VM, Name, Created, SizeGB | Sort-Object Name

All VMs show lcm-post-deploy-9.0.2 snapshot with creation timestamp
Step 7

Now simulate a major lab change: perform a non-disruptive operation (e.g., document current SDDC Manager state). Then take a second checkpoint: Get-VM -Name 'Holo-*' | New-Snapshot -Name 'lcm-post-validation-9.0.2' -Description 'After operational validation' -Memory:$false -Quiesce:$false

Second set of snapshots created, representing a savepoint at a different phase
Step 8

Examine snapshot metadata including growth: $snapshots = Get-Snapshot -Name 'lcm-*'; $snapshots | Format-Table -Property VM, Name, SizeGB, Created

Two snapshots per VM, showing size growth and timestamps
Step 9

Review snapshot consolidation implications. Run: Get-VM -Name 'Holo-*' | Get-Snapshot | Measure-Object SizeGB -Sum

Total snapshot size reported (should be <50% of available free space for safety)
Step 10

Document your naming convention in a text file: echo 'Snapshot Naming: <component>-<phase>-<version>. Examples: lcm-post-deploy-9.0.2, lcm-post-wld-9.0.2, lcm-post-config-9.0.2' > C:\Holodeck\holodeck-runtime\SNAPSHOT_NAMING.txt

Text file created with naming convention documented

Validation Gate

Check: Verify: (1) At least 2 snapshots exist per VM, (2) snapshot names follow the convention <component>-<phase>-<version>, (3) all snapshots are non-memory, (4) total snapshot size < 50% of free datastore space

Expected: Multi-checkpoint snapshot strategy in place with documented naming and growth tracking

Common Errors

New-Snapshot command fails with 'Insufficient free space'
Cause: Datastore has insufficient free space for snapshots, or other VMs on the host are consuming space
Fix: Check Get-Datastore free space. Delete old snapshots: Get-Snapshot -Name 'old-*' | Remove-Snapshot -Confirm:$false. Alternatively, consolidate existing snapshots: Get-VM | Get-Snapshot | Remove-Snapshot
📋 KB: Datastore space management — common in nested environments
New-Snapshot with -Quiesce:$true hangs or fails
Cause: VMware Tools not responsive in nested VMs, or timeout during quiescence
Fix: Use -Quiesce:$false for nested environments. Nested VMs often have slow I/O which causes quiescence to timeout.
Snapshot size larger than expected (>100 GB per VM)
Cause: Memory snapshots were created with -Memory:$true, or datastore is using thin provisioning and snapshot size is inflated
Fix: For thin-provisioned datastores, snapshot size reflects changed blocks, not provisioned capacity. Monitor actual free space, not reported sizes.

Task 2 Destroy and redeploy an instance to verify cleanup and measure redeploy time

recoverability

The destroy-redeploy cycle is the ultimate validation that your instance is reproducible and that the Toolkit properly cleans up resources. VCDX panelists will probe: How do you ensure repeatability? Can you detect orphaned resources? How would you automate this in CI/CD? This task answers those questions.

Step 1

Before destroying, document the instance configuration. Run: Get-HoloDeckInstance | ConvertTo-Json | Out-File C:\Holodeck\holodeck-runtime\instance-backup-before-destroy.json

JSON file created with instance metadata
Step 2

Power off all Holodeck VMs to prepare for destruction. Run: Get-VM -Name 'Holo-*' | Stop-VM -Kill -Confirm:$false

All VMs powered off (or skipped if already off)
Step 3

Record pre-destruction datastore state: $datastorePre = Get-Datastore | Where-Object {$_.Name -like 'holodeck'}; $datastorePre | Format-Table -Property Name, FreeSpaceGB, CapacityGB

Datastore metrics before cleanup
Step 4

Execute instance removal. Get your instance ID first: $instanceId = (Get-HoloDeckInstance).InstanceId; Write-Host 'Destroying instance: '$instanceId; Remove-HoloDeckInstance -InstanceID $instanceId -Verbose

Progress output showing VM deletion, port group cleanup, snapshot cleanup. Final message: 'Instance removed successfully'
This step takes 10-20 minutes. Do not interrupt. The Toolkit must sequence VM deletions and port group cleanup correctly.
Step 5

Verify cleanup: (a) No VMs named Holo- should exist: Get-VM -Name 'Holo-' -ErrorAction SilentlyContinue | Measure-Object

Count: 0 (no VMs found)
Step 6

Verify (b) Port groups cleaned: Get-VirtualPortGroup | Where-Object {$_.Name -match '10\.0\.'} | Measure-Object

Count: 0 or minimal (only physical infrastructure port groups remain)
Step 7

Verify (c) Datastore space reclaimed: $datastorePost = Get-Datastore | Where-Object {$_.Name -like 'holodeck'}; $spaceReclaimed = $datastorePost.FreeSpaceGB - $datastorePre.FreeSpaceGB; Write-Host 'Space reclaimed: '$spaceReclaimed' GB'

FreeSpaceGB increased (typically 300-500 GB for 9.0.2 instance)
Step 8

Now redeploy from scratch using the same config.json. First, prepare: $stopwatch = [System.Diagnostics.Stopwatch]::StartNew(); New-HoloDeckInstance -ConfigFile .\templates\config.json -Verbose

Prepare phase completes with all validations passing. Record the elapsed time.
Step 9

Start the full deployment: Start-HoloDeckInstance -InstanceID ((Get-HoloDeckInstance).InstanceId) -Verbose

Full bring-up sequence progresses. Record the time when 'deployment completed successfully' message appears.
Step 10

Compare redeploy time vs snapshot revert time. Redeploy typically takes 90-150 minutes. Snapshot revert (if you had reverted instead of destroying) would take ~5-10 minutes. Document this in your lab notebook.

Time comparison documented

Validation Gate

Check: After destroy: (1) No VMs named Holo-* exist, (2) management port groups removed or minimal, (3) datastore space significantly recovered. After redeploy: (1) Instance returns to 'Running' state, (2) All management plane components healthy, (3) Redeploy time measured and logged

Expected: Complete lifecycle cycle demonstrated: destroy validates cleanup, redeploy validates reproducibility

Common Errors

Remove-HoloDeckInstance hangs during 'Removing Virtual Machines'
Cause: One or more VMs are locked by snapshots, vMotion operations, or backup jobs
Fix: Manually identify and remove snapshots: Get-Snapshot -VM 'Holo-*' | Remove-Snapshot -Confirm:$false. Force power off: Get-VM -Name 'Holo-*' | Stop-VM -Kill. Then retry Remove-HoloDeckInstance.
Datastore space not reclaimed after Remove-HoloDeckInstance completes
Cause: Snapshots were not cleaned up before VM deletion, or datastore reclamation is delayed
Fix: Wait 5-10 minutes and check again. If space is still not reclaimed, manually delete any remaining Holo-* snapshots and files.
Redeploy fails with 'Cannot select datastore' even though datastore has >1.5 TB free
Cause: Fragmentation or cached datastore metadata
Fix: Rescan the datastore: Get-Datastore | Rescan-DataStore. Wait 1-2 minutes, then retry Prepare.

Task 3 Version-control config.json templates and demonstrate config overlay patterns

manageability

Configuration management at scale: how do you manage multiple scenarios without copying and pasting? VCDX architectural thinking includes template reuse, inheritance, and override patterns. This task applies those principles to Holodeck configs.

Step 1

Create a config directory structure for version control: mkdir -p C:\Holodeck\holodeck-runtime\configs\{base,overlays,deployed}

Directory structure created
Step 2

Copy your current config.json as the base template: Copy-Item .\templates\config.json .\configs\base\config-base-9.0.2.json

Base config file created
Step 3

Create overlay configs for different scenarios. First, create an overlay for management-only deployment: @{ managementOnly = $true; vcfAutomationRequired = $false } | ConvertTo-Json | Out-File .\configs\overlays\overlay-mgmt-only.json

Overlay file created
Step 4

Create an overlay for different resource tiers (e.g., 3-host compact vs 4-host standard). Read the base config: $baseConfig = Get-Content .\configs\base\config-base-9.0.2.json | ConvertFrom-Json

Base config loaded into PowerShell object
Step 5

Modify the config for a 3-host scenario (compact): $baseConfig.holodeck_sddc.'Site-A'.management.esxi.NumofNode = 3; $baseConfig.holodeck_sddc.'Site-A'.management.esxi.memory = 128; $baseConfig | ConvertTo-Json -Depth 20 | Out-File .\configs\deployed\config-vcf902-mgmt-3host.json

Modified config saved for 3-host scenario
Step 6

Verify the 3-host config: Get-Content .\configs\deployed\config-vcf902-mgmt-3host.json | ConvertFrom-Json | Select-Object -ExpandProperty holodeck_sddc | Select-Object -ExpandProperty 'Site-A' | Select-Object -ExpandProperty management | Select-Object -ExpandProperty esxi | Format-Table -Property NumofNode, memory

NumofNode: 3, memory: 128 (values correctly modified)
Step 7

Create naming convention documentation: @'EOM
Holodeck Config Naming Convention:
---
(1) base configs: config-base-<vcf-version>.json
Example: config-base-9.0.2.json

(2) deployed configs: config-vcf<version>-<scenario>.json
Example: config-vcf902-mgmt-only.json
Example: config-vcf902-mgmt-3host.json
Example: config-vcf921-full-wld.json

(3) overlays: overlay-<feature>.json (for composition patterns)
Example: overlay-mgmt-only.json
Example: overlay-vsan-compact.json

(4) dated archives: config-<scenario>-<date>.json.bak
Example: config-vcf902-mgmt-only-2026-04-17.json.bak
EOM
' | Out-File .\configs\CONFIG_NAMING.md

Naming convention documented
Step 8

Create a version history file: Add-Content C:\Holodeck\holodeck-runtime\configs\VERSION_HISTORY.txt -Value @"
2026-04-17: config-vcf902-mgmt-only.json deployed successfully (90min)
2026-04-17: config-vcf902-mgmt-3host.json validated (85min)
2026-04-16: config-base-9.0.2.json created as baseline
"@

Version history log created
Step 9

Demonstrate config validation before deployment. Create a validation function in PowerShell: function Validate-HoloDeckConfig { param([string]$ConfigFile); $config = Get-Content $ConfigFile | ConvertFrom-Json; if (-not $config.vcf.version) { Write-Host 'ERROR: vcf.version not set' } else { Write-Host 'VCF version: '$config.vcf.version }; if (-not $config.network.management_cidr) { Write-Host 'ERROR: management_cidr not set' } else { Write-Host 'Management CIDR: '$config.network.management_cidr } }; Validate-HoloDeckConfig -ConfigFile .\configs\deployed\config-vcf902-mgmt-only.json

Validation output showing version and network settings
Step 10

Create a config snapshot feature: function Export-ConfigSnapshot { param([string]$ConfigId); $timestamp = Get-Date -Format 'yyyyMMdd-HHmmss'; Copy-Item .\configs\deployed\$ConfigId .\configs\base\archive\$ConfigId'.snapshot-'$timestamp'.json }; Export-ConfigSnapshot -ConfigId 'config-vcf902-mgmt-only.json'

Snapshot of config created in archive directory

Validation Gate

Check: Verify: (1) Base config exists at configs/base/, (2) At least 2 deployed configs at configs/deployed/, (3) Naming convention documented, (4) Version history log populated, (5) Validation function works correctly

Expected: Config management framework in place with templates, overlays, naming convention, and version history

Common Errors

Config modification breaks PowerShell JSON parsing (ConvertFrom-Json fails)
Cause: Manual JSON editing introduced syntax error (missing comma, unmatched brace)
Fix: Use PowerShell objects instead of manual JSON editing: $config = ConvertFrom-Json; $config.field = value; $config | ConvertTo-Json. This avoids syntax errors.
Deployed config works in test but fails in production scenario
Cause: Config overlay was not fully applied, or base config had stale values not overridden
Fix: Always merge overlays explicitly: $merged = $baseConfig; foreach ($key in $overlay.PSObject.Properties) { $merged | Add-Member -MemberType NoteProperty -Name $key.Name -Value $key.Value -Force }. Document which overlays are active.

Task 4 Deploy side-by-side VCF 5.2.x and 9.0.x instances and document version differences

manageability

This is VCDX gold. Demonstrating deep knowledge of version differences shows architectural maturity. By running 5.2.x and 9.0.x simultaneously, you can articulate what changed between releases (component versions, bring-up sequence, SDDC Manager UI, feature set). Panelists will probe your design choices when considering upgrades.

Step 1

Verify your VCF 5.2.x content manifest is available. List content manifests: Get-ChildItem .\content\manifests\ | Where-Object {$_.Name -like '5.2*'} | Format-Table Name

At least one manifest file for VCF 5.2.x (e.g., vcf-5.2.1-manifest.json)
If no 5.2.x manifest exists, download it from Broadcom support portal and stage in the content/ directory.
Step 2

Create a VCF 5.2.x config by copying the base config and modifying version: $config521 = Get-Content .\configs\base\config-base-9.0.2.json | ConvertFrom-Json; $config521.vcfVersion = '5.2.1'; $config521.contentManifest = '.\content\manifests\vcf-5.2.1-manifest.json'; $config521 | ConvertTo-Json -Depth 20 | Out-File .\configs\deployed\config-vcf521-mgmt-only.json

VCF 5.2.1 config file created with updated version and manifest path
Step 3

Update the Site-A naming in the config to avoid conflicts with the 9.0.2 instance. Modify the site identifier: $config521.holodeck_sddc | Add-Member -MemberType NoteProperty -Name 'Site-B' -Value $config521.holodeck_sddc.'Site-A' -Force; $config521.holodeck_sddc.PSObject.Properties.Remove('Site-A'); $config521 | ConvertTo-Json -Depth 20 | Out-File .\configs\deployed\config-vcf521-mgmt-only.json

Config updated with Site-B identifier
Step 4

Prepare the VCF 5.2.1 instance: New-HoloDeckInstance -ConfigFile .\configs\deployed\config-vcf521-mgmt-only.json -Verbose

Prepare phase completes. Note the content version strings logged in the verbose output.
Step 5

Start the VCF 5.2.1 deployment in parallel (or sequentially if resources are limited): Start-HoloDeckInstance -InstanceID ((Get-HoloDeckInstance | Where-Object {$_.VcfVersion -eq '5.2.1'}).InstanceId) -Verbose

Deployment begins. Monitor for 90-150 minutes. Note any differences in the bring-up sequence compared to 9.0.2.
Running two large Holodeck instances simultaneously requires careful resource management. Monitor physical host RAM and CPU. If contention occurs, complete 9.0.2 first, snapshot it, then start 5.2.1.
Step 6

Once 5.2.1 deployment completes, validate both instances. Get the VCF 5.2.1 SDDC Manager IP (different Site-B network) and compare: Write-Host 'VCF 9.0.2 instances:'; Get-HoloDeckInstance | Where-Object {$_.VcfVersion -eq '9.0.2'} | Format-Table -Property InstanceId, State, VcfVersion; Write-Host 'VCF 5.2.1 instances:'; Get-HoloDeckInstance | Where-Object {$_.VcfVersion -eq '5.2.1'} | Format-Table -Property InstanceId, State, VcfVersion

Two instances listed with different versions and states
Step 7

Access both SDDC Managers and document differences. For 5.2.x, navigate to https://10.10.0.4 (Site-B network). For 9.0.2, navigate to https://10.1.1.4 (Site-A network). Compare: (a) SDDC Manager UI layout and menu structure, (b) Component versions listed in System > About, (c) Workload Domain configuration workflow

Screenshots or notes documenting UI differences between versions
Step 8

Compare component versions. Access vCenter on both instances and check: Administration > System Configuration > Servers. Document the vCenter version for each instance (typically vCenter 7.x for 5.2.x, vCenter 8.x for 9.0.x).

Component version comparison table created
Step 9

Document 5+ specific architectural or operational differences. Create a comparison document: @'EOM
VCF 5.2.1 vs VCF 9.0.2 Differences:
---

  1. SDDC Manager UI:

5.2.x: Simplified dashboard, fewer Day 2 operations visible
9.0.x: Enhanced UI with VCF Automation integration, expanded Day 2 menu

  1. vCenter Version:

5.2.x: vCenter 7.0.x
9.0.x: vCenter 8.0.x

  1. NSX Version:

5.2.x: NSX 4.x
9.0.x: NSX 9.0.x

  1. Cloud Builder Role:

5.2.x: Cloud Builder required for bring-up, then can be removed
9.0.x: Cloud Builder retained for SDDC lifecycle operations

  1. Licensing Model:

5.2.x: Older licensing mechanism, limited evaluation trial
9.0.x: New licensing portal integration, extended evaluation trial
EOM
' | Out-File C:\Holodeck\holodeck-runtime\COMPARISON_5.2-vs-9.0.md

Comparison document created
Step 10

Create a summary discussing VCDX-relevant insights: 'From a design perspective, VCF 9.0.x introduces tighter integration between SDDC Manager and lifecycle operations. This affects how you plan upgrade windows, capacity planning, and automation. A VCDX candidate should understand these version-specific constraints when designing infrastructure.'

Design reflection written

Validation Gate

Check: Verify: (1) Two instances deployed (VCF 5.2.1 and 9.0.2), both in Running state, (2) Both SDDC Managers accessible, (3) Component version comparison table created, (4) At least 5 documented differences, (5) VCDX design reflection written

Expected: Side-by-side comparison complete with architectural insights documented for panelist discussion

Common Errors

VCF 5.2.1 deployment fails at Cloud Builder initialization with 'Unsupported component version'
Cause: Content manifest has version mismatch or incorrect bundle references
Fix: Verify the manifest JSON explicitly lists VCF 5.2.1, vCenter 7.0.x, NSX 4.x bundles. Download fresh content pack if bundles are mismatched.
📋 KB: Similar to Thread 23: Holodeck 9.0.1 VCF Deployment fails (version mismatch pattern)
Both instances compete for resources; one deployment stalls
Cause: Insufficient physical host RAM or CPU for two concurrent large deployments
Fix: Complete one deployment fully, take a snapshot, then start the second. Alternatively, reduce nested ESXi host count in one config (3 hosts instead of 4).

Task 5 Automate Holodeck operations and build a lab reset script

manageability

Operational maturity means repeatability and automation. VCDX-level thinking includes building frameworks that others (or your future self) can execute without manual intervention. This task applies software engineering best practices (idempotency, error handling, logging) to lab operations.

Step 1

Create a PowerShell module for common Holodeck operations. Create file: C:\Holodeck\holodeck-runtime\HoloDeckOps.psm1

PowerShell module file created
Step 2

Add a function to deploy an instance with logging: @'
function New-HoloDeckDeployment {
param(
[string]$ConfigFile,
[string]$LogPath = 'C:\Holodeck\holodeck-runtime\logs'
)
$timestamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$logFile = Join-Path $LogPath "deployment-$timestamp.log"

Write-Host "[$(Get-Date)] Starting deployment with config: $ConfigFile" | Tee-Object -FilePath $logFile -Append

try {
New-HoloDeckInstance -ConfigFile $ConfigFile -Verbose 2>&1 | Tee-Object -FilePath $logFile -Append
$instanceId = (Get-HoloDeckInstance | Where-Object {$_.State -eq 'Prepared'}).InstanceId
Write-Host "[$(Get-Date)] Prepared instance: $instanceId" | Tee-Object -FilePath $logFile -Append

Start-HoloDeckInstance -InstanceID $instanceId -Verbose 2>&1 | Tee-Object -FilePath $logFile -Append
Write-Host "[$(Get-Date)] Deployment completed successfully" | Tee-Object -FilePath $logFile -Append
return $instanceId
}
catch {
Write-Host "[$(Get-Date)] ERROR: $_" | Tee-Object -FilePath $logFile -Append
throw
}
}
export-modulemember -function New-HoloDeckDeployment
' | Out-File C:\Holodeck\holodeck-runtime\HoloDeckOps.psm1 -Encoding UTF8

Deployment function added to module
Step 3

Add a snapshot management function: @'
function New-HoloDeckCheckpoint {
param(
[string]$CheckpointName,
[string]$Description
)
Write-Host "Creating checkpoint: $CheckpointName"
Get-VM -Name 'Holo-*' | New-Snapshot -Name $CheckpointName -Description $Description -Memory:$false -Quiesce:$false | Out-Null
Write-Host "Checkpoint created: $CheckpointName"
}

function Revert-HoloDeckCheckpoint {
param(
[string]$CheckpointName
)
Write-Host "Reverting to checkpoint: $CheckpointName"
Get-VM -Name 'Holo-*' | Get-Snapshot -Name $CheckpointName | Foreach-Object { Set-VM -VM $_.VM -Snapshot $_ -Confirm:$false } 2>&1
Write-Host "Checkpoint reverted: $CheckpointName"
}
export-modulemember -function New-HoloDeckCheckpoint, Revert-HoloDeckCheckpoint
' | Add-Content C:\Holodeck\holodeck-runtime\HoloDeckOps.psm1 -Encoding UTF8

Snapshot functions added to module
Step 4

Create a lab reset script: @'

HoloDeck Lab Reset Script

Usage: .\Reset-Lab.ps1 -TargetSnapshot 'holodeck-02-complete'

param(
[string]$TargetSnapshot = 'holodeck-02-complete',
[switch]$Confirm = $true
)

Import-Module C:\Holodeck\holodeck-runtime\HoloDeckOps.psm1

Write-Host "Lab Reset: Reverting all VMs to snapshot: $TargetSnapshot"

if ($Confirm) {
$response = Read-Host "Confirm reset? (yes/no)"
if ($response -ne 'yes') { Write-Host 'Cancelled'; exit }
}

Get-VM -Name 'Holo-*' | Where-Object { Get-Snapshot -VM $_ -Name $TargetSnapshot -ErrorAction SilentlyContinue } | Foreach-Object {
Write-Host "[$(Get-Date)] Reverting: $($_.Name)"
Get-Snapshot -VM $_ -Name $TargetSnapshot | Set-VM -Snapshot -Confirm:$false | Out-Null
}

Write-Host "[$(Get-Date)] Lab reset complete. Starting VMs..."
Get-VM -Name 'Holo-*' | Start-VM -Confirm:$false | Out-Null
Write-Host "Lab ready"
' | Out-File C:\Holodeck\holodeck-runtime\Reset-Lab.ps1 -Encoding UTF8

Lab reset script created
Step 5

Create a scheduled snapshot script. This script runs weekly to create a clean checkpoint: @'

Scheduled Weekly Snapshot

This runs every Friday at 18:00 via Windows Task Scheduler

$timestamp = Get-Date -Format 'yyyyMMdd'
$checkpointName = "auto-weekly-$timestamp"

Write-Host "[$(Get-Date)] Creating weekly checkpoint: $checkpointName"
Get-VM -Name 'Holo-*' | New-Snapshot -Name $checkpointName -Description "Auto-generated weekly snapshot" -Memory:$false -Quiesce:$false | Out-Null

Write-Host "[$(Get-Date)] Snapshot created"

Clean up snapshots older than 4 weeks

$cutoffDate = (Get-Date).AddDays(-28)
Get-VM -Name 'Holo-*' | Get-Snapshot | Where-Object { $_.Created -lt $cutoffDate } | Remove-Snapshot -Confirm:$false

Write-Host "[$(Get-Date)] Old snapshots cleaned up"
' | Out-File C:\Holodeck\holodeck-runtime\ScheduledSnapshot.ps1 -Encoding UTF8

Scheduled snapshot script created
Step 6

Test the reset script (dry-run): Read the script to verify syntax, then test on a single VM: Get-VM -Name 'Holo-Console' | Get-Snapshot | Measure-Object

Script readable, snapshot exists on at least one VM
Step 7

Create a CI/CD integration example. This script validates a config before deployment: @'
function Test-HoloDeckConfig {
param([string]$ConfigFile)

Write-Host "Validating config: $ConfigFile"
$config = Get-Content $ConfigFile | ConvertFrom-Json

Validation checks

$checks = @(
('vcfVersion defined', -not [string]::IsNullOrEmpty($config.vcfVersion)),
('contentManifest exists', (Test-Path $config.contentManifest)),
('ESXi count > 2', $config.holodeck_sddc."Site-A".management.esxi.NumofNode -ge 2)
)

$passed = 0
foreach ($check in $checks) {
$status = if ($check[1]) { 'PASS' } else { 'FAIL' }
Write-Host " [$status] $($check[0])"
if ($check[1]) { $passed++ }
}

return ($passed -eq $checks.Count)
}
export-modulemember -function Test-HoloDeckConfig
' | Out-File C:\Holodeck\holodeck-runtime\HoloDeckValidation.psm1 -Encoding UTF8

Config validation module created
Step 8

Create a deployment status dashboard script. This shows health of all instances: @'
Write-Host "Holodeck Deployment Status"
Write-Host "============================="
Get-HoloDeckInstance | Format-Table -Property @(
'InstanceId',
'State',
'VcfVersion',
@{Label='SDDC Manager';Expression={Get-VM -Name 'SDDC-Manager*' -ErrorAction SilentlyContinue | Measure-Object | Select-Object -ExpandProperty Count}}
) -AutoSize

Write-Host ""
Write-Host "Datastore Usage"
Write-Host "==============="
Get-Datastore | Where-Object {$_.Name -like 'holodeck'} | Format-Table -Property Name, @{Label='Free GB';Expression={[Math]::Round($_.FreeSpaceGB, 0)}}, @{Label='Total GB';Expression={[Math]::Round($_.CapacityGB, 0)}} -AutoSize

Write-Host ""
Write-Host "Snapshots"
Write-Host "========="
Get-VM -Name 'Holo-*' | Get-Snapshot | Group-Object -Property VM | Format-Table -Property Name, @{Label='Snapshot Count';Expression={$_.Count}} -AutoSize
' | Out-File C:\Holodeck\holodeck-runtime\Show-LabStatus.ps1 -Encoding UTF8

Status dashboard script created
Step 9

Test the modular functions: Import-Module C:\Holodeck\holodeck-runtime\HoloDeckOps.psm1; New-HoloDeckCheckpoint -CheckpointName 'test-automation' -Description 'Testing automation script'

Checkpoint created successfully using the automated function
Step 10

Document the automation framework in a README: @'

Holodeck Automation Framework

Available Functions

Deployment
  • New-HoloDeckDeployment -ConfigFile <path>: Deploy instance with logging
Checkpoints
  • New-HoloDeckCheckpoint -CheckpointName <name>: Create snapshot
  • Revert-HoloDeckCheckpoint -CheckpointName <name>: Revert to snapshot
Utilities
  • .\Reset-Lab.ps1 -TargetSnapshot <name>: Full lab reset
  • .\Show-LabStatus.ps1: Display current health
  • Test-HoloDeckConfig -ConfigFile <path>: Validate config before deployment

Scheduling

Schedule weekly snapshots via Windows Task Scheduler:
Action: PowerShell -ExecutionPolicy Bypass -File C:\Holodeck\holodeck-runtime\ScheduledSnapshot.ps1
Trigger: Weekly on Friday at 18:00

Idempotency

All scripts are designed to be re-runnable. If a step fails, you can run again without side effects.
' | Out-File C:\Holodeck\holodeck-runtime\AUTOMATION_README.md -Encoding UTF8

Automation framework documentation created

Validation Gate

Check: Verify: (1) HoloDeckOps.psm1 module exists with at least 3 functions, (2) Reset-Lab.ps1 script created and readable, (3) ScheduledSnapshot.ps1 created, (4) Config validation module created, (5) Status dashboard script created, (6) Module functions tested successfully, (7) Automation README documented

Expected: Complete automation framework in place for repeatable, idempotent lab operations

Common Errors

PowerShell module fails to import: 'PSModuleAutoloadingPreference not set'
Cause: Module folder structure or manifest missing
Fix: Ensure the .psm1 file is in a folder matching the module name. For simplicity, use dot-sourcing instead: . C:\path\to\script.ps1
Reset script fails: 'Cannot find snapshot on VM'
Cause: Snapshot name doesn't match exactly, or doesn't exist on all VMs
Fix: Use Get-VM | Get-Snapshot to list available snapshots. Update the script to handle missing snapshots: Get-Snapshot ... -ErrorAction SilentlyContinue

Final Validation

The Holodeck instance lifecycle is fully managed: snapshots checkpoint progress, destroy-redeploy cycles validate reproducibility, config templates enable multi-version testing, side-by-side instances demonstrate version differences, and automation scripts enable repeatable operations. You have operational maturity equivalent to production systems — the skills transfer directly.

✓ Snapshots: At least 2 named checkpoints per VM with multi-checkpoint strategy documented → Naming convention follows <component>-<phase>-<version> pattern

✓ Destroy-redeploy: Instance removed completely + redeployed from config, times measured → Redeploy time ~2-3x slower than snapshot revert; datastore space reclaimed

✓ Config management: Base + deployed + overlay configs organized in version-controlled structure → At least 2 deployed configs with naming convention and history log

✓ Version comparison: VCF 5.2.x and 9.0.x deployed, 5+ differences documented → VCDX-relevant architectural insights written (UI, components, licensing)

✓ Automation: 5+ PowerShell functions + reset script + validation + status dashboard → All scripts tested and runnable; module imports successfully

Cleanup / Restore

Snapshot: holodeck-07-complete

• Take final snapshot after all tasks complete: Get-VM -Name 'Holo-*' | New-Snapshot -Name 'holodeck-07-complete' -Description 'Post lifecycle management lab' -Memory:$false -Quiesce:$false

• Consolidate intermediate snapshots to reduce datastore overhead (keep only holodeck-02-complete and holodeck-07-complete): Get-Snapshot -Name 'lcm-*' | Remove-Snapshot -Confirm:$false

• Archive config files: Copy-Item .\configs\deployed\* .\configs\base\archive\

• Back up automation scripts to external location for future reference

Design Reflection (VCDX)

A VCDX panelist examining your operational maturity would probe: How do you manage multiple lab scenarios simultaneously? What happens when a deployment fails mid-way — can you roll back? How would you standardize Holodeck deployments across a team? Can you articulate the trade-offs between snapshots (fast, state-dependent) and redeploy (slower, reproducible)? How do version-specific differences affect your design when planning multi-version testing? This lab demonstrates that you think operationally — not just about deployment, but about repeatability, auditability, and automation. That's what separates architects from operators.

Requirements

  • Support rapid environment reset for iterative design testing
  • Enable concurrent testing of multiple VCF versions on shared infrastructure
  • Provide reproducible, version-controlled configurations for auditing and compliance
  • Detect and cleanup orphaned resources to prevent resource exhaustion
  • Automate routine operations (snapshot, reset, redeploy) to reduce human error

Constraints

  • Snapshots are I/O and storage-intensive — only practical for lab, not production
  • Concurrent instances severely constrain physical host resources (512 GB RAM limits to 1-2 full instances)
  • Nested virtualization overhead means performance characteristics don't reflect production
  • Config templating requires discipline — unused configs accumulate and complicate management

Assumptions

  • Holodeck Toolkit is the source of truth for instance lifecycle (Prepare, Start, Remove)
  • PowerShell is the automation layer (vs REST API, which is limited)
  • Snapshots are non-memory to reduce I/O overhead and enable faster revert
  • Configs are version-controlled and organized (not scattered across multiple directories)

Risks

  • Snapshot chain corruption causes restore failures — IMPACT: environment loss, MITIGATION: periodic consolidation, test restores regularly
  • Config drift (local edits not committed) causes reproducibility failures — IMPACT: deployment fails in CI/CD, MITIGATION: enforce config validation before deploy
  • Resource exhaustion during multi-instance testing cascades to physical host failure — IMPACT: total lab loss, MITIGATION: monitor resource usage, implement early warning alerts
  • VCF version incompatibilities (e.g., NSX bundle mismatch) cause silent failures — IMPACT: incomplete deployments, MITIGATION: automated manifest validation, test content packs before production use

Self-Assessment Discussion Prompts

  1. When would you snapshot vs redeploy? What are the trade-offs in terms of time, disk space, and accuracy?
  2. You have 512 GB RAM and need to test VCF 9.0.2, VCF 5.2.1, and a future VCF release. How would you manage the environment?
  3. Snapshot consolidation is manual and labor-intensive. How would you automate it in a production lab environment?
  4. A junior team member misses a step in the reset-lab script and leaves the environment in an inconsistent state. How would you prevent this?
  5. How do version-specific config differences (e.g., NSX 9.0.x vs NSX 4.x) affect your testing methodology?
  6. If you needed to provision Holodeck labs for 10 VCDX candidates simultaneously, what changes to your automation strategy would be needed?

Extensions

Build a Holodeck-as-Code (HaC) Framework

Extend the config templating into a full Infrastructure-as-Code pattern. Use Terraform or Ansible to orchestrate Holodeck provisioning. This transforms lab setup from manual scripts to declarative configuration — a production-grade capability.

much harder

Implement Automated Health Checks and Alerting

Build a monitoring dashboard that continuously checks Holodeck instance health (SDDC Manager availability, NSX fabric status, datastore space). Trigger automated remediation (snapshot revert, restart failed VMs) on detected anomalies.

harder

Create a Multi-Tenant Lab Hosting Service

Design a system where multiple users can provision independent Holodeck instances on shared infrastructure. This requires quota management, RBAC, automated cleanup of abandoned instances, and cost tracking. A VCDX-level challenge.

much harder

Integrate CI/CD Pipeline for Automated Lab Testing

Build a GitOps workflow where pushing a config to a Git repo automatically triggers Holodeck deployment, runs validation tests, and reports results. This demonstrates integration of lab infrastructure into development workflows.

harder

Deploy Holodeck Across Multiple Physical Hosts

Extend the framework to span multiple physical ESXi hosts (e.g., two separate lab pods). Implement cross-site networking, DNS resolution, and coordinated lifecycle management. This is production-equivalent complexity.

much harder

⚠ Known Pitfalls (from Community KB)

Encounter Issues when Modified default ESXi Host CPU & Memory RESOLVED
Problem: Config modifications don't apply consistently to all hosts — first host gets new values, others retain defaults
Resolution: Always customize config before first Prepare. If already partially deployed, destroy instance and re-prepare with clean config.
Start-HoloDeckInstance -InstanceID xxxxxx command failed. Cannot find instance OPEN
Problem: Start command fails after successful Prepare, Stop works but Start fails with missing appliances error
Resolution: Instance JSON file missing Appliance property array. Verify Get-HoloDeckInstance lists instance and appliances are populated.
Holodeck vcf 9 - Cannot select datastore RESOLVED
Problem: Datastore selection fails during instance creation even with adequate free space
Resolution: Verify physical datastore name matches config, clean up orphaned VMs from failed attempts, rescan datastore.
Waiting for FRR service LIKELY_RESOLVED
Problem: Prepare phase hangs indefinitely waiting for BGP routing daemon to start on HoloRouter
Resolution: SSH to HoloRouter and restart FRR: systemctl restart frr. If still stuck, destroy and re-prepare.
VCF Installer is not ready yet RESOLVED
Problem: Start phase hangs waiting for Cloud Builder initialization; services stuck
Resolution: Check Cloud Builder DNS, NTP, and service status via SSH. Fix root cause (usually DNS) and restart vcf-bringup service.

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.