Lab: Create T0/T1 Logical Routing Topology

VCF 9.0Intermediatevcp-foundation⏱ 120 min

NSX 9.0.x (feature set inherited from the NSX 4.2 line; NSX 4.2 docs remain a valid technical reference) Tier-0/Tier-1 gateway architecture — BGP peering, route redistribution, N-S traffic flow

Objectives

Deploy a Tier-0 gateway with Active-Active HA mode on an Edge cluster
Configure BGP peering between T0 and a physical router (or simulated peer)
Deploy Tier-1 gateways and connect overlay segments
Understand route redistribution between T0 and T1
Validate north-south traffic flow through the routing topology
Compare Active-Standby vs Active-Active T0 HA modes

Prerequisites

VCF 9.0 with NSX Manager operational, NSX Edge cluster deployed (minimum 2 Edge nodes), overlay transport zone with TEPs verified (Lab net-virt-01 completed), physical router or VyOS VM available for BGP peering

Prior labs: net-virt-01

Required skills:

BGP fundamentals (ASN, neighbor, route advertisement)
NSX Edge architecture concepts
IP subnetting

Lab Environment

VCF 9.0 management domain with 2+ Edge nodes in an Edge cluster. Edge uplinks connected to a VLAN segment that peers with a physical ToR switch or VyOS router VM. Overlay transport zone with at least 2 ESXi hosts.

Tasks

Task 1 Build T0/T1 Routing Topology with BGP

T0 HA mode selection is a critical design decision. Active-Active (ECMP) doubles north-south bandwidth but does not support stateful services (NAT, firewall, VPN, load balancer). Active-Standby supports all stateful services but uses only one Edge for forwarding. In VCF, the management domain T0 is typically Active-Standby (needs NAT for SDDC services); workload domain T0s can be Active-Active if stateful services are on T1.

Construct the complete NSX logical routing hierarchy — T0 for north-south connectivity with BGP peering to the physical network, T1 for tenant/application-level routing — and validate the full traffic path from overlay VMs to external networks.

Step 1

Pre-flight: Verify Edge cluster health. NSX Manager → System → Fabric → Edge Transport Nodes — all Edge nodes should show Configuration State 'Success' and Connectivity 'Up'. Note the Edge cluster name (e.g., 'edge-cluster-01'). Verify Edge nodes have uplink interfaces on the external VLAN segment.

Step 2

Create Tier-0 Gateway. NSX Manager → Networking → Tier-0 Gateways → Add Tier-0 Gateway. Configure: Name='T0-Lab', HA Mode=Active-Active (ECMP for this lab — no stateful services needed), Edge Cluster='edge-cluster-01'. Click Save. Note: In production VCF, SDDC Manager creates the management T0 automatically — this lab simulates a workload domain T0.

Step 3

Configure T0 uplink interfaces. Edit T0-Lab → Interfaces → Add Interface. For each Edge node: Name='uplink-edge01', Type=External, IP=192.168.100.1/24 (Edge node 1), Edge Node=edge-01, Segment=select the VLAN uplink segment. Repeat for edge-02: IP=192.168.100.2/24. These IPs face the physical router. In Active-Active mode, both interfaces are active simultaneously (ECMP).

Step 4

Configure BGP on T0. Edit T0-Lab → BGP → Enable BGP, Local AS=65100. Add BGP Neighbor: IP=192.168.100.254 (physical router), Remote AS=65200, Keep Alive Timer=4s, Hold Down Timer=12s. Enable BFD for fast failure detection (recommended interval: 500ms × 3). If using VyOS, configure the peer side: 'set protocols bgp 65200 neighbor 192.168.100.1 remote-as 65100' and same for .2.

Step 5

Verify BGP peering establishment. In T0-Lab → BGP → BGP Neighbors, status should show 'Established' for each peer. If stuck at 'Active' or 'Connect': (a) verify physical/VLAN connectivity with ping from Edge to router, (b) check ASN mismatch, (c) verify no firewall blocking TCP 179. From Edge CLI: get logical-router <T0-SR-UUID> bgp neighbor summary.

Step 6

Create Tier-1 Gateway. NSX Manager → Networking → Tier-1 Gateways → Add Tier-1 Gateway. Configure: Name='T1-App-Prod', Linked Tier-0 Gateway='T0-Lab', Edge Cluster='edge-cluster-01' (required for services like NAT/LB on T1), Route Advertisement=toggle ON: 'All Connected Segments & Service Ports'. Click Save.

Step 7

Connect overlay segments to T1. Create or reuse overlay segments: 'seg-web' (subnet 10.10.1.1/24), 'seg-app' (10.10.2.1/24), 'seg-db' (10.10.3.1/24). For each: edit segment → Connected Gateway='T1-App-Prod', Subnet=<gateway IP>/24. The T1 becomes the default gateway for VMs on each segment. Verify: Networking → Tier-1 Gateways → T1-App-Prod → Route Table should show connected routes for all three subnets.

Step 8

Configure route redistribution. Edit T0-Lab → Route Redistribution → Add Route Redistribution. Name='redistribute-t1-connected', Sources: select 'Tier-1 Connected', 'Tier-1 LB VIP', 'Tier-1 NAT'. This tells the T0 to include T1 connected routes in BGP advertisements to the physical router. Verify: the physical router's BGP table should now contain 10.10.1.0/24, 10.10.2.0/24, 10.10.3.0/24.

Step 9

Validate north-south traffic. Deploy a test VM on seg-web with IP 10.10.1.11, gateway 10.10.1.1. From the VM, ping 192.168.100.254 (physical router) — this validates: VM → T1 DR (distributed on ESXi host) → T1 SR (on Edge if stateful services, or backhaul to T0) → T0 SR (on Edge) → physical router. Then ping an external destination beyond the router to validate full path.

Step 10

Inspect routing on ESXi host. SSH to the host running the test VM, run: nsxcli → get logical-routers (lists DR and SR instances). Then: get logical-router <T1-DR-UUID> route-table — should show connected routes for local segments and a default route pointing to T0. This proves the Distributed Router is handling first-hop routing directly on the hypervisor (no hairpin to Edge for east-west traffic).

Step 11

Test east-west vs north-south path difference. Deploy a second VM on seg-app (10.10.2.11). From the seg-web VM, ping 10.10.2.11 — this traffic stays entirely within the ESXi data plane (T1 DR routes between segments locally, never touches the Edge). Compare with the north-south ping from step 9 — T0 SR on the Edge is only involved for traffic exiting the NSX domain. Use Traceflow to visualize both paths (see Lab net-virt-04).

Validation Gate

Check: Complete T0/T1 routing topology with BGP and traffic validation

Expected: T0 created with Active-Active HA and BGP peering established to physical router. T1 connected with 3 overlay segments. Routes redistributed to physical network. North-south and east-west traffic validated. Distributed routing confirmed on ESXi host.

Common Errors

BGP neighbor stuck in 'Active' state

Fix: TCP 179 connectivity issue. Verify: (1) Edge uplink IP can ping the physical router, (2) no ACL blocking BGP on the physical side, (3) ASN numbers match the configuration. Use Edge CLI: get logical-router <SR-UUID> bgp neighbor <IP> to see detailed state.

T1 connected routes not appearing in BGP table on physical router

Fix: Route redistribution not configured on T0, or 'Tier-1 Connected' source not selected. Also verify T1 Route Advertisement has 'All Connected Segments' enabled.

North-south ping fails but east-west works

Fix: T0 SR to physical router path issue. Check: (1) T0 uplink interface status, (2) Edge node connectivity to VLAN segment, (3) physical router has return route to NSX overlay subnets (should be via BGP).

Asymmetric routing in Active-Active T0

Fix: Both Edge nodes advertise same routes. Physical router must support ECMP for symmetric forwarding. If not, use Active-Standby or configure BGP communities/AS-path prepending to prefer one Edge.

Final Validation

Complete NSX routing hierarchy operational with BGP peering and verified traffic flows

✓ T0 BGP peering → Established state on all BGP neighbors with routes exchanged

✓ T1 connected segments → 3 segments connected with gateway IPs, route advertisement enabled

✓ Route redistribution → T1 subnets visible in physical router's BGP table

✓ North-south traffic → VM on overlay can reach external networks via T0 → physical router

✓ East-west traffic → Inter-segment traffic routed by T1 DR on ESXi host (no Edge hairpin)

✓ Distributed routing verification → T1 DR route table on ESXi shows connected routes and default via T0

Cleanup / Restore

• Disconnect test VMs from segments

• Delete segments (seg-web, seg-app, seg-db)

• Delete T1 gateway (T1-App-Prod) — must remove segments first

• Delete T0 gateway (T0-Lab) — must remove T1 links first

• Verify routes withdrawn from physical router BGP table

Design Reflection (VCDX)

T0 HA mode selection is a classic VCDX design decision. Be ready to justify Active-Active (ECMP bandwidth, no single Edge bottleneck) vs Active-Standby (stateful services support, simpler failover). In VCF 9.0, the pattern is: management domain T0 = Active-Standby (needs NAT/firewall), workload domain T0 = Active-Active if stateful services live on T1 gateways. Discuss Edge cluster sizing — each Edge node must handle full T0 bandwidth during partner failure.

Requirements

BGP peering to physical network for dynamic route exchange
Support for 3-tier application with separate broadcast domains
East-west traffic must not hairpin through Edge nodes

Constraints

Active-Active T0 cannot host stateful services (NAT, FW, LB, VPN)
Edge cluster requires dedicated hosts or resource reservation
Maximum 1 T0 per Edge cluster in VCF-managed deployments

Assumptions

Physical router supports BGP and ECMP
Edge uplink VLAN is routable to the physical infrastructure
DNS and NTP are reachable from overlay segments via the T0 path

Risks

Edge node failure in Active-Standby causes brief north-south outage during failover (10-15s without BFD)
BGP reconvergence time during Edge failure depends on timers — production should use BFD (sub-second detection)
Over-provisioning T1 gateways without Edge cluster capacity planning leads to resource contention

Self-Assessment Discussion Prompts

When would you choose static routing over BGP for T0 uplinks?
How do you size Edge nodes for a workload domain expecting 40 Gbps north-south throughput?
What happens to stateful NAT sessions during Active-Standby T0 failover?
How does the Distributed Router (DR) on ESXi improve east-west performance compared to centralized routing?

Extensions

Switch T0 to Active-Standby and configure NAT (SNAT for overlay → external, DNAT for inbound services)

Add a second T1 gateway for a different application and verify route isolation

Configure OSPF instead of BGP on the T0 uplinks and compare convergence behavior

Enable Gateway Firewall on T0 for north-south inspection and test policy enforcement

⚠ Known Pitfalls (from Community KB)

Selecting Active-Active for T0 that needs NAT — stateful services are silently ignored; traffic will pass un-NATted

Forgetting route redistribution — T1 subnets exist in NSX routing table but are never advertised to the physical network, causing asymmetric routing or black-holing return traffic

Not enabling BFD on BGP neighbors — without BFD, BGP hold-down timer (default 180s) means 3 minutes of black-holing during Edge failure

Creating T1 without Edge cluster binding — T1 services (NAT, LB, GW FW) require an Edge cluster; without it, only distributed routing works

References

NSX 4.2 Administration Guide — Tier-0 Gateways: techdocs.broadcom.com
VCF 9.0 Networking Reference Architecture — Edge Cluster Design: techdocs.broadcom.com
KB 91532 — BGP Troubleshooting on NSX Edge Nodes

Mark complete