Academy/NSX 4.x Network Virtualization Professional (2V0-41.24)/Lab N2: NSX Federation Across Two VCF 9.0 Instances
This lab targets VCF 9.0

Lab N2: NSX Federation Across Two VCF 9.0 Instances

VCF 9.0Advancedvcp-foundation⏱ 150 min

NSX 9.0.x (feature set inherited from the NSX 4.2 line; NSX 4.2 docs remain a valid technical reference) Federation — Global Manager, cross-site policy, stretched segments, location-aware networking

Objectives

  • Understand NSX Federation architecture — Global Manager vs Local Manager
  • Configure federation peering between two VCF instances
  • Create stretched segments and gateways spanning both sites
  • Implement location-aware DFW policies for cross-site security
  • Test site failure scenarios and understand split-brain handling
  • Compare federated vs non-federated multi-site designs

Prerequisites

Two VCF 9.0 instances (or simulated with two NSX Manager clusters in the same lab), each with Edge clusters and overlay transport zones operational. Inter-site L3 connectivity established between the two environments, including a routed RTEP VLAN/subnet per site for the Edge nodes.

Prior labs: net-virt-01, net-virt-02, net-virt-05

Required skills:

  • NSX Manager administration
  • Multi-site networking concepts
  • BGP/routing between sites

Lab Environment

Two VCF sites: Site-A (NSX Manager cluster-A, Edge cluster-A) and Site-B (NSX Manager cluster-B, Edge cluster-B). Both sites connected via L3 routed link (WAN or simulated inter-site VLAN). Global Manager appliance deployed on management network accessible from both sites.

Tasks

Task 1 Configure NSX Federation with Stretched Segments and Cross-Site Policy

NSX Federation uses a hierarchical model: Global Manager (GM) defines global objects (stretched T0/T1, segments, DFW policies) that are synchronized to Local Managers (LM) at each site. Each LM maintains an independent control plane — if inter-site connectivity is lost, each site continues to operate locally. Federation is not a replacement for disaster recovery — it enables policy consistency and workload mobility across sites. In VCF 9.0, Federation is supported across VCF instances in the same or different Private Clouds.

Build a federated NSX topology spanning two VCF instances, demonstrating how NSX Federation enables consistent policy across sites while maintaining local control plane independence. This is critical for disaster recovery, multi-site workload mobility, and consistent security posture across geographically distributed infrastructure.

Step 1

Deploy Global Manager. Deploy the NSX Manager OVA with the role 'NSX Federation Global Manager' on the management network — it is separate from both sites' Local Manager clusters. Size: Medium (6 vCPU, 24 GB RAM) minimum — Medium, Large or Extra Large are the supported GM sizes; Small (4 vCPU, 16 GB) must not be used for scale (VCF 9.0.1 docs p.2300, p.2757). Configure: FQDN='gm.lab.local'. In production, form a 3-node GM cluster with a VIP and add a standby GM cluster at the other location (latency 500 ms or less), made standby from System > Location Manager (VCF 9.0.1 docs p.2300); a single GM node is acceptable only in the lab. The GM must be reachable from both Site-A and Site-B NSX Manager clusters.

Step 2

Register Local Managers with Global Manager. Log into the Global Manager UI → System → Location Manager → Add On-Prem Location. For Site-A: Location Name='Site-A', FQDN/IP=<Site-A NSX Manager cluster VIP> (not an individual NSX Manager FQDN or IP), the admin credentials (no other account), and the SHA-256 thumbprint from 'get certificate cluster thumbprint' run on any Site-A NSX Manager node; click Check Compatibility (VCF 9.0.1 docs p.2301). Repeat for Site-B. After registration, the GM discovers each site's fabric: Edge clusters, transport zones, segments. Verify: Location Manager shows both sites with status 'Active'. Then configure Remote Tunnel Endpoints (RTEP) on each location's Edge cluster — stretched gateways and segments are carried Edge-to-Edge over RTEP, not host TEP to host TEP: on each Local Manager (or via the GM Location Manager site selector) System → Quick Start → Configure Remote Tunnel Endpoint → Getting Started; select the Edge nodes, Edge switch, an RTEP VLAN and IP pool separate from the TEP network; set the RTEP MTU under System → Fabric → Settings → Remote Tunnel Endpoint (default 1500; raise it as high as the inter-site path supports). Confirm L3 reachability between the RTEP subnets of both sites.

Step 3

Create stretched Tier-0 gateway. On the Global Manager → Networking → Tier-0 Gateways → Add. Name='GM-T0-Stretched', HA Mode=Active-Standby (recommended for stretched — stateful services needed for cross-site NAT). Assign Edge clusters: Site-A Edge cluster as Primary, Site-B Edge cluster as Secondary. Configure uplink interfaces at each site to peer with local physical routers. The GM creates the T0 on both LMs simultaneously.

Step 4

Create stretched Tier-1 and overlay segment. On the GM → Networking → Tier-1 Gateways → Add. Name='GM-T1-App', Linked T0='GM-T0-Stretched'. Create a stretched segment: Name='seg-app-stretched', Connected Gateway='GM-T1-App', Subnet=10.20.1.1/24, Span=Both sites. This segment is realized on transport nodes at BOTH sites — VMs at Site-A and Site-B can communicate on the same L2 domain.

Step 5

Verify stretched segment realization. On each site's Local Manager (NOT the GM), navigate to Networking → Segments — the stretched segment should appear as a 'Read-Only' object (managed by GM, cannot be edited locally). Deploy a VM at each site on the stretched segment: vm-site-a (10.20.1.11) at Site-A, vm-site-b (10.20.1.12) at Site-B. Ping between them — within a site the traffic uses host TEPs; across sites it is forwarded via that site's Edge nodes and carried over the RTEP tunnel to the Edge nodes of the other location.

Step 6

Configure location-aware DFW policies. On the GM → Security → Distributed Firewall → Add Policy. Name='Cross-Site-Policy'. Create rules with location awareness: (a) Rule 1: Source=SG-Web (Site-A only), Dest=SG-App (any site), Service=HTTPS, Action=Allow, Applied-To=Site-A only — this rule is pushed ONLY to Site-A hosts; (b) Rule 2: Source=SG-DB (Site-B only), Dest=<backup-server>, Service=NFS, Action=Allow, Applied-To=Site-B only. Location-scoped rules enable site-specific policies within a global policy framework.

Step 7

Verify policy distribution. On Site-A's Local Manager → Security → DFW — the global policies appear as read-only entries above local policies. Local admins can still create site-local DFW policies below the global ones. On Site-B, verify the same global policy appears but with the Site-B-specific rules. This demonstrates the hierarchy: Global policies (from GM) → Local policies (from LM).

Step 8

Examine the ingress path (north-south). GM-T0-Stretched is Active-Standby with Site-A as the primary location, so all traffic for the stretched subnet enters and leaves through the active Edge node at Site-A (VCF 9.0.1 docs p.2316); traffic for vm-site-b therefore crosses to Site-B over RTEP (tromboning). Compare the BGP tables at both sites' physical routers to see where 10.20.1.0/24 is advertised and preferred. Then discuss the options for steering ingress — AS-path prepending on the BGP peers in the secondary locations (p.2315), a Tier-1 gateway whose span is a single location, whose segments therefore stay local (p.2290, p.2314), or GSLB in front of the application (an Avi Load Balancer feature, p.235) — rather than expecting a single toggle (a T0 setting named 'Ingress Routing Optimization' did not surface in the VCF 9.0.1 NSX Federation chapters, p.2289 onward).

Step 9

Simulate site failure. Disconnect inter-site connectivity (or shut down Site-B's NSX Manager). Observe: (a) GM shows Site-B as 'Disconnected'; (b) Site-A continues operating normally — all local VMs maintain connectivity and DFW enforcement; (c) Site-B also continues independently — its LM takes autonomous control of local fabric. This is the key Federation design principle: independent failure domains. Cross-site stretched segment traffic is disrupted, but local operations are unaffected.

Step 10

Recovery and reconciliation. Restore inter-site connectivity. Observe: (a) GM reconnects to Site-B and shows 'Active' status; (b) Any local changes made during the partition are flagged — GM reconciliation process merges non-conflicting changes and flags conflicts for admin review; (c) Stretched segment connectivity resumes — VMs at both sites can communicate again. Monitor the GM Events log for reconciliation details.

Validation Gate

Check: NSX Federation operational with stretched resources and cross-site policy

Expected: Global Manager managing both sites, stretched T0/T1/segments operational with cross-site VM connectivity, location-aware DFW policies distributed correctly, site failure handled gracefully with independent local operations, recovery reconciliation successful

Common Errors

Local Manager registration fails with 'Certificate error'
Fix: GM and LM must trust each other's certificates. If using self-signed certs (lab), import the LM cert into GM trust store and vice versa. In production, use CA-signed certificates for all NSX Manager appliances.
Stretched segment shows 'Partial Success' — one site not realized
Fix: The overlay transport zone at the failing site may not have the same name or configuration. Ensure both sites have compatible overlay TZs. Also verify: RTEP is configured on an Edge cluster at the failing site and the RTEP subnets of both sites are reachable across the WAN/inter-site link.
Cross-site VM traffic fails despite segment being stretched
Fix: The inter-site tunnel runs between Edge RTEPs: (a) RTEP configured on the Edge nodes of each location, on a VLAN/subnet separate from TEP, with L3 reachability between the RTEP networks; (b) RTEP MTU (System → Fabric → Settings; default 1500) consistent with what the inter-site path carries end to end — jumbo MTU is desirable but not a requirement; (c) Geneve (UDP 6081) permitted between the RTEP subnets by WAN firewalls.
Global policy not appearing on Local Manager
Fix: Check GM-to-LM synchronization status: GM → System → Location Manager → site status. If 'Sync Pending', the LM may be under heavy load or network latency between GM and LM is high. Allow time for sync (up to 5 minutes for large policy sets).

Final Validation

NSX Federation operational with cross-site networking and security policy

✓ GM location status → Both sites show 'Active' in Location Manager

✓ Stretched segment connectivity → VMs at Site-A and Site-B communicate on the stretched segment

✓ Location-aware DFW → Site-specific rules only appear on hosts at the target site

✓ Ingress path → BGP tables at both physical routers show where the stretched subnet is advertised and preferred; the ingress steering option chosen (primary location, AS-path prepend, per-location Tier-1 or GSLB) is documented

✓ Site failure independence → Surviving site operates normally during partition

✓ Recovery reconciliation → GM reconnects and reconciles after partition healing

Cleanup / Restore

• Delete stretched segments and VMs from GM

• Remove stretched T1 and T0 gateways from GM

• Deregister Local Managers from Global Manager (Location Manager → Remove)

• Power off Global Manager appliance

Design Reflection (VCDX)

Federation is a premium VCDX topic — it addresses multi-site design, disaster recovery, and operational complexity. Key design decisions: (1) GM placement — an active GM cluster (3 nodes, VIP) at one location plus a standby GM cluster at another location within 500 ms latency, both on management networks reachable from all Local Managers; (2) Stretched vs local objects — only stretch what needs cross-site L2 mobility; keep everything else local to minimize blast radius; (3) Location-aware DFW — enables global security baseline with site-specific exceptions; (4) RTEP design — cross-location traffic hairpins through the Edge nodes (RTEP), so Edge cluster placement, RTEP VLAN/IP pools and RTEP MTU on the WAN are design decisions, not host-TEP settings.

Requirements

  • Consistent security policy across 2+ VCF sites
  • Workload mobility between sites without IP change
  • Independent site operations during WAN failure

Constraints

  • Global Manager is a single point of policy management — its failure prevents global changes (but doesn't affect local operations)
  • Stretched segments require RTEPs on the Edge clusters of every participating location, with RTEP-to-RTEP reachability and a consistent RTEP MTU (default 1500) across the inter-site path
  • Location count per the VMware Configuration Maximums tool — a Medium GM is sized for up to four locations and 128 hypervisors; Large or Extra Large GMs cover higher scale (VCF 9.0.1 docs p.2757)

Assumptions

  • Inter-site WAN provides reliable L3 connectivity with < 150ms RTT
  • Both sites have compatible NSX versions (federation requires same major version)
  • GM is deployed highly available: a 3-node active GM cluster with a VIP plus a standby GM cluster at the other location

Risks

  • GM failure during a partition leaves both sites operating independently — any changes diverge and require manual reconciliation
  • Stretched segment traffic trombone through remote site Edge if ingress steering (primary location choice, AS-path prepend, per-location Tier-1 or GSLB) is not designed
  • Over-stretching resources (every segment, every policy) increases inter-site dependency and blast radius

Self-Assessment Discussion Prompts

  1. When would you choose NSX Federation over independent NSX domains with API-driven policy sync?
  2. How do you handle MTU constraints on a 1500-byte WAN for GENEVE stretched segments?
  3. What is the GM recovery procedure if the Global Manager cluster is permanently lost?
  4. How does Federation interact with VCF's Private Cloud → Fleet → Instance hierarchy?

Extensions

Deploy a standby GM cluster at Site-B, make it standby from the active GM (System > Location Manager > Add Standby) and test GM failover

Configure Avi (NSX ALB) GSLB to steer traffic to the closest site for a stretched application

Implement a DR runbook: planned failover of all workloads from Site-A to Site-B using stretched segments

Test the maximum number of stretched segments and DFW rules to understand federation scale limits

⚠ Known Pitfalls

Placing the Global Manager at one site only — if that site is lost, global changes are impossible until GM recovery; run the active GM as a 3-node cluster with a VIP at one location and add a standby GM cluster at another location (VCF 9.0.1 docs p.2300)
Stretching all segments by default — creates unnecessary inter-site dependency; only stretch segments that truly need cross-site L2 mobility
Ignoring WAN MTU for GENEVE overhead — stretched segment traffic uses GENEVE encapsulation across the WAN; if WAN MTU is 1500, either enable fragmentation or reduce inner MTU to ~1400
Not testing site partition behavior before production — split-brain handling is complex; validate reconciliation procedures in the lab first

References

  • VCF 9.0 documentation — Multisite and NSX Federation (VCF 9.0.1 docs p.2274 onward; NSX Federation p.2289): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html
  • VCF 9.0 documentation — Install the Active and Standby Global Manager (VCF 9.0.1 docs p.2299): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html
  • VCF 9.0 documentation — Add a Location (VCF 9.0.1 docs p.2301): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html
  • VCF 9.0 documentation — Configure Edge Nodes for Stretched Networking (RTEP) (VCF 9.0.1 docs p.2304): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.