Academy/NSX 4.x Network Virtualization Professional (2V0-41.24)/Lab: Create Overlay Segment & Verify Tunnel Endpoints
This lab targets VCF 9.0

Lab: Create Overlay Segment & Verify Tunnel Endpoints

VCF 9.0Intermediatevcp-foundation⏱ 90 min

NSX 9.0.x (feature set inherited from the NSX 4.2 line; NSX 4.2 docs remain a valid technical reference) overlay networking — GENEVE encapsulation (UDP 6081), VDS-based host switch (N-VDS is not supported on ESX transport nodes since NSX 4.0), TEP validation

Objectives

  • Create an overlay segment on an NSX overlay transport zone
  • Understand GENEVE encapsulation and VNI allocation
  • Verify Tunnel Endpoint (TEP) establishment between ESXi hosts
  • Validate VM-to-VM connectivity across the overlay fabric
  • Inspect host-switch (VDS) forwarding tables and ARP suppression

Prerequisites

VCF 9.0 management domain operational, NSX Manager deployed, ESXi hosts prepared as transport nodes with overlay transport zone membership and TEP IP pools configured

Required skills:

  • NSX Manager UI navigation
  • SSH access to ESXi hosts
  • Basic GENEVE/overlay networking concepts

Lab Environment

VCF 9.0 management domain with minimum 2 ESXi hosts configured as NSX transport nodes. Each host has vmk10 (TEP vmkernel) on a dedicated VLAN-backed TEP segment. NSX Manager cluster (3-node or single for lab) operational.

Tasks

Task 1 Create Overlay Segment and Verify TEP Connectivity

Production overlay designs must consider TEP VLAN sizing (MTU of at least 1700 required on TEP VLANs and gateways — Geneve needs 1600 and 1700 leaves headroom — with 9000 recommended on host ports), TEP IP pool exhaustion, and multi-TEP for bandwidth scaling. Prefer static IP pools over DHCP for host TEP addressing (VCF-VNI-RCMD-CFG-001 is a recommendation: it removes the external DHCP dependency; DHCP is still accepted).

Build an overlay segment from scratch and validate the entire forwarding path — from NSX Manager control plane down to the ESXi kernel data plane — ensuring GENEVE tunnels are established, VNI is allocated, and VM-to-VM traffic traverses the overlay correctly.

Step 1

Pre-flight: Verify transport node status. In NSX Manager → System → Fabric → Transport Nodes, confirm all ESXi hosts show 'Success' for Configuration State and 'Up' for Node Status. If any host shows 'In Progress' or 'Failed', resolve before proceeding (common cause: TEP vmkernel port missing or MTU mismatch).

Step 2

Verify TEP connectivity between hosts. SSH to esxi-01, run: vmkping ++netstack=vxlan -d -s 8972 <esxi-02-TEP-IP>. The -d flag sets the DF bit; -s 8972 tests a 9000-byte path (the VCF recommendation, VCF-NET-RCMD-SWT-004). If the underlay is built to the 1700-byte minimum (VCF-NET-REQD-MTU-006), test with -s 1672 instead. Success confirms the underlay carries the Geneve encapsulation overhead. If this fails, check the MTU on every hop carrying the TEP VLAN (1700 or more required, 9000 recommended; VCF 9.0.1 docs p.585) and the TEP VLAN configuration.

Step 3

Create overlay segment. NSX Manager → Networking → Segments → Add Segment. Configure: Name='prod-web-seg', Connected Gateway=None (L2-only for now), Transport Zone=select your overlay TZ (e.g., 'tz-overlay'), Subnets=leave empty (no gateway yet), Admin State=Up. Click Save → the segment receives an auto-allocated VNI (e.g., 72001).

Step 4

Verify segment realization. After saving, click the segment name → check Realization status = 'Success'. Navigate to Related Resources → Transport Nodes tab — all hosts in the overlay TZ should show the segment as realized. Note the VNI value from the Overview tab — you will verify this on the host.

Step 5

Verify on ESXi host via CLI. SSH to esxi-01, run: nsxcli → get logical-switches. Confirm your segment appears with the correct VNI. Then run: get logical-switch <VNI> arp-table and get logical-switch <VNI> mac-table — both should be empty (no VMs attached yet). Run: get logical-switch <VNI> vtep-table — should list TEP IPs of all other transport nodes in the TZ.

Step 6

Attach test VMs. In vSphere Client, edit VM 'test-web-01' on esxi-01 → Network Adapter 1 → Browse → select 'prod-web-seg'. Repeat for 'test-web-02' on esxi-02. Power on both VMs and assign IPs: test-web-01=10.10.1.11/24, test-web-02=10.10.1.12/24 (no gateway needed for this L2 test).

Step 7

Validate overlay connectivity. From test-web-01 console: ping 10.10.1.12. Traffic flows: VM → vNIC → VSIP filter (DFW) → NSX-prepared VDS port → GENEVE encapsulation (src TEP → dst TEP, UDP 6081, VNI header) → physical underlay → destination host reverse path. Verify the ping succeeds.

Step 8

Inspect forwarding tables post-traffic. SSH to esxi-01, run: nsxcli → get logical-switch <VNI> mac-table — should now show test-web-02's MAC with its TEP IP as the destination tunnel endpoint. Run: get logical-switch <VNI> arp-table — should show 10.10.1.12 → MAC mapping (ARP suppression cache). These tables prove the overlay data plane is fully converged.

Step 9

Packet capture (optional but valuable). On esxi-01, capture GENEVE-encapsulated traffic: pktcap-uw --uplink vmnic0 --capture UplinkSndKernel -o /tmp/geneve.pcap. Transfer the pcap and open in Wireshark — filter on 'udp.port==6081'. You should see the outer IP header (TEP-to-TEP), GENEVE header with VNI, and inner Ethernet frame (VM-to-VM). This is exam-critical knowledge.

Step 10

Segment profile exploration. In NSX Manager → Networking → Segments → Segment Profiles, review the default profiles: Segment Security (BPDU filter, DHCP snooping), MAC Discovery (MAC learning, unknown unicast flooding), IP Discovery (ARP snooping, ND snooping), SpoofGuard (disabled by default). For production, enable SpoofGuard and ARP snooping at minimum. Note: VCF applies these defaults automatically.

Validation Gate

Check: Complete overlay data path verification

Expected: Overlay segment created with auto-allocated VNI, TEP tunnels UP between all hosts (TEP MTU path verified), test VMs communicate across hosts via GENEVE encapsulation, MAC/ARP tables populated correctly on both hosts

Common Errors

TEP vmkping fails with jumbo frames
Fix: Every hop carrying the TEP VLAN must have an MTU of at least 1700 (VCF-NET-REQD-MTU-006); 9000 is recommended. Match the vmkping -s value to the MTU you built (1672 for 1700, 8972 for 9000). Check: esxcfg-nics -l (host NIC MTU), then verify each switch hop in the physical path.
Segment shows 'Partial Success' in realization
Fix: One or more transport nodes failed to realize the segment. Check System → Fabric → Transport Nodes for the failing host — common causes: host disconnected from NSX Manager, or NSX VIB installation incomplete.
VM connectivity works on same host but fails cross-host
Fix: TEP routing issue — ensure TEP IPs on different hosts can reach each other via the underlay. If TEPs are on different subnets, a Layer 3 route must exist between TEP VLANs.
ARP table not populating despite successful pings
Fix: ARP suppression may be disabled on the segment profile. Verify IP Discovery profile has ARP Snooping enabled. Without it, ARP floods as BUM traffic — functional but inefficient.

Final Validation

Overlay segment fully operational with verified TEP tunnels and cross-host VM connectivity

✓ Segment realization status → Success on all transport nodes in the overlay TZ

✓ TEP tunnel health → All TEP-to-TEP tunnels show BFD status UP (Edge: get bfd-sessions; ESX: nsxdp-cli bfd sessions list)

✓ VM-to-VM ping across hosts → 0% packet loss on overlay segment

✓ MAC/ARP tables populated → Remote VM entries present with correct TEP destination

✓ GENEVE encapsulation verified → Packet capture shows UDP 6081 with correct VNI in header

Cleanup / Restore

• Disconnect test VMs from segment (revert to original network)

• Delete 'prod-web-seg' segment from NSX Manager (Networking → Segments → Delete)

• Verify segment removed from hosts: nsxcli → get logical-switches

Design Reflection (VCDX)

In a VCDX defense, be prepared to justify overlay vs. VLAN segment choices. Key arguments: overlay provides workload mobility without L2 stretching at the physical layer, supports micro-segmentation natively, and scales beyond 4096 VLAN limit (16M VNIs). Discuss TEP design — single vs. multi-TEP per host, TEP VLAN sizing, and failure domain isolation (separate TEP pools per rack for blast radius containment).

Requirements

  • East-west VM communication across hosts without VLAN trunking changes
  • Support for >4096 logical networks
  • Workload mobility without physical network reconfiguration

Constraints

  • Physical underlay MTU of at least 1700 end to end on TEP VLANs and gateways for Geneve (VCF-NET-REQD-MTU-006); 9000 recommended
  • TEP IP pools sized and pre-provisioned (VCF recommends static IP pools over DHCP for host TEPs, VCF-VNI-RCMD-CFG-001)
  • Maximum 16 TEPs per host (practical limit for bandwidth scaling)

Assumptions

  • All physical switches in the TEP path support jumbo frames
  • TEP VLAN is a dedicated, low-latency network (not shared with vMotion)
  • NSX Manager cluster is healthy and all hosts are connected

Risks

  • TEP MTU mismatch causes silent packet drops — hard to diagnose without pktcap-uw
  • TEP IP pool exhaustion during scale-out prevents new hosts from joining overlay
  • Single TEP VLAN creates a network failure domain — consider multi-VLAN TEP design for resilience

Self-Assessment Discussion Prompts

  1. Why does VCF recommend static IP pools over DHCP for host TEP addressing, and what dependency does DHCP add?
  2. How would you design TEP networks for a 64-host cluster across 4 racks?
  3. What is the impact of enabling ARP suppression vs. allowing ARP flooding?
  4. When would you choose head replication vs. hierarchical replication for BUM traffic?

Extensions

Configure multi-TEP (2 TEPs per host) and verify ECMP load balancing across both uplinks

Create a VLAN-backed segment on the same host and compare forwarding behavior (no GENEVE encap)

Test BUM traffic handling: send a broadcast from one VM, capture on the uplink of all hosts to observe head replication

⚠ Known Pitfalls

Using MTU 1500 on the physical underlay — Geneve adds at least 50 bytes of overhead (more with options), so full-size VM frames no longer fit; small pings succeed while large transfers fail, which is hard to diagnose. Build at least 1700 (9000 recommended)
Forgetting to check TEP vmkernel stack — TEP traffic uses the 'vxlan' netstack, not the default stack; regular vmkping won't test the right path
Creating segments on wrong transport zone — overlay segments on VLAN TZ (or vice versa) will fail realization

References

  • NSX 4.2 documentation (landing page) — Administration Guide: segments and BUM replication modes: https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2.html
  • VCF 9.0 documentation (landing page) — design: infrastructure VLAN MTU (VCF-NET-REQD-MTU-006, VCF 9.0.1 docs p.585) and host TEP IP assignment (VCF-VNI-RCMD-CFG-001, p.952); ESX transport nodes use VDS only (p.1957): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.