Lab N2: NSX Federation Across Two VCF 9.0 Instances
Objectives
- Understand NSX Federation architecture — Global Manager vs Local Manager
- Configure federation peering between two VCF instances
- Create stretched segments and gateways spanning both sites
- Implement location-aware DFW policies for cross-site security
- Test site failure scenarios and understand split-brain handling
- Compare federated vs non-federated multi-site designs
Prerequisites
Two VCF 9.0 instances (or simulated with two NSX Manager clusters in the same lab), each with Edge clusters and overlay transport zones operational. Inter-site L3 connectivity established between the two environments, including a routed RTEP VLAN/subnet per site for the Edge nodes.
Prior labs: net-virt-01, net-virt-02, net-virt-05
Required skills:
- NSX Manager administration
- Multi-site networking concepts
- BGP/routing between sites
Lab Environment
Two VCF sites: Site-A (NSX Manager cluster-A, Edge cluster-A) and Site-B (NSX Manager cluster-B, Edge cluster-B). Both sites connected via L3 routed link (WAN or simulated inter-site VLAN). Global Manager appliance deployed on management network accessible from both sites.
Tasks
Task 1 Configure NSX Federation with Stretched Segments and Cross-Site Policy
Build a federated NSX topology spanning two VCF instances, demonstrating how NSX Federation enables consistent policy across sites while maintaining local control plane independence. This is critical for disaster recovery, multi-site workload mobility, and consistent security posture across geographically distributed infrastructure.
Deploy Global Manager. Deploy the NSX Manager OVA with the role 'NSX Federation Global Manager' on the management network — it is separate from both sites' Local Manager clusters. Size: Medium (6 vCPU, 24 GB RAM) minimum — Medium, Large or Extra Large are the supported GM sizes; Small (4 vCPU, 16 GB) must not be used for scale (VCF 9.0.1 docs p.2300, p.2757). Configure: FQDN='gm.lab.local'. In production, form a 3-node GM cluster with a VIP and add a standby GM cluster at the other location (latency 500 ms or less), made standby from System > Location Manager (VCF 9.0.1 docs p.2300); a single GM node is acceptable only in the lab. The GM must be reachable from both Site-A and Site-B NSX Manager clusters.
Register Local Managers with Global Manager. Log into the Global Manager UI → System → Location Manager → Add On-Prem Location. For Site-A: Location Name='Site-A', FQDN/IP=<Site-A NSX Manager cluster VIP> (not an individual NSX Manager FQDN or IP), the admin credentials (no other account), and the SHA-256 thumbprint from 'get certificate cluster thumbprint' run on any Site-A NSX Manager node; click Check Compatibility (VCF 9.0.1 docs p.2301). Repeat for Site-B. After registration, the GM discovers each site's fabric: Edge clusters, transport zones, segments. Verify: Location Manager shows both sites with status 'Active'. Then configure Remote Tunnel Endpoints (RTEP) on each location's Edge cluster — stretched gateways and segments are carried Edge-to-Edge over RTEP, not host TEP to host TEP: on each Local Manager (or via the GM Location Manager site selector) System → Quick Start → Configure Remote Tunnel Endpoint → Getting Started; select the Edge nodes, Edge switch, an RTEP VLAN and IP pool separate from the TEP network; set the RTEP MTU under System → Fabric → Settings → Remote Tunnel Endpoint (default 1500; raise it as high as the inter-site path supports). Confirm L3 reachability between the RTEP subnets of both sites.
Create stretched Tier-0 gateway. On the Global Manager → Networking → Tier-0 Gateways → Add. Name='GM-T0-Stretched', HA Mode=Active-Standby (recommended for stretched — stateful services needed for cross-site NAT). Assign Edge clusters: Site-A Edge cluster as Primary, Site-B Edge cluster as Secondary. Configure uplink interfaces at each site to peer with local physical routers. The GM creates the T0 on both LMs simultaneously.
Create stretched Tier-1 and overlay segment. On the GM → Networking → Tier-1 Gateways → Add. Name='GM-T1-App', Linked T0='GM-T0-Stretched'. Create a stretched segment: Name='seg-app-stretched', Connected Gateway='GM-T1-App', Subnet=10.20.1.1/24, Span=Both sites. This segment is realized on transport nodes at BOTH sites — VMs at Site-A and Site-B can communicate on the same L2 domain.
Verify stretched segment realization. On each site's Local Manager (NOT the GM), navigate to Networking → Segments — the stretched segment should appear as a 'Read-Only' object (managed by GM, cannot be edited locally). Deploy a VM at each site on the stretched segment: vm-site-a (10.20.1.11) at Site-A, vm-site-b (10.20.1.12) at Site-B. Ping between them — within a site the traffic uses host TEPs; across sites it is forwarded via that site's Edge nodes and carried over the RTEP tunnel to the Edge nodes of the other location.
Configure location-aware DFW policies. On the GM → Security → Distributed Firewall → Add Policy. Name='Cross-Site-Policy'. Create rules with location awareness: (a) Rule 1: Source=SG-Web (Site-A only), Dest=SG-App (any site), Service=HTTPS, Action=Allow, Applied-To=Site-A only — this rule is pushed ONLY to Site-A hosts; (b) Rule 2: Source=SG-DB (Site-B only), Dest=<backup-server>, Service=NFS, Action=Allow, Applied-To=Site-B only. Location-scoped rules enable site-specific policies within a global policy framework.
Verify policy distribution. On Site-A's Local Manager → Security → DFW — the global policies appear as read-only entries above local policies. Local admins can still create site-local DFW policies below the global ones. On Site-B, verify the same global policy appears but with the Site-B-specific rules. This demonstrates the hierarchy: Global policies (from GM) → Local policies (from LM).
Examine the ingress path (north-south). GM-T0-Stretched is Active-Standby with Site-A as the primary location, so all traffic for the stretched subnet enters and leaves through the active Edge node at Site-A (VCF 9.0.1 docs p.2316); traffic for vm-site-b therefore crosses to Site-B over RTEP (tromboning). Compare the BGP tables at both sites' physical routers to see where 10.20.1.0/24 is advertised and preferred. Then discuss the options for steering ingress — AS-path prepending on the BGP peers in the secondary locations (p.2315), a Tier-1 gateway whose span is a single location, whose segments therefore stay local (p.2290, p.2314), or GSLB in front of the application (an Avi Load Balancer feature, p.235) — rather than expecting a single toggle (a T0 setting named 'Ingress Routing Optimization' did not surface in the VCF 9.0.1 NSX Federation chapters, p.2289 onward).
Simulate site failure. Disconnect inter-site connectivity (or shut down Site-B's NSX Manager). Observe: (a) GM shows Site-B as 'Disconnected'; (b) Site-A continues operating normally — all local VMs maintain connectivity and DFW enforcement; (c) Site-B also continues independently — its LM takes autonomous control of local fabric. This is the key Federation design principle: independent failure domains. Cross-site stretched segment traffic is disrupted, but local operations are unaffected.
Recovery and reconciliation. Restore inter-site connectivity. Observe: (a) GM reconnects to Site-B and shows 'Active' status; (b) Any local changes made during the partition are flagged — GM reconciliation process merges non-conflicting changes and flags conflicts for admin review; (c) Stretched segment connectivity resumes — VMs at both sites can communicate again. Monitor the GM Events log for reconciliation details.
Validation Gate
Check: NSX Federation operational with stretched resources and cross-site policy
Expected: Global Manager managing both sites, stretched T0/T1/segments operational with cross-site VM connectivity, location-aware DFW policies distributed correctly, site failure handled gracefully with independent local operations, recovery reconciliation successful
Common Errors
Final Validation
NSX Federation operational with cross-site networking and security policy
✓ GM location status → Both sites show 'Active' in Location Manager
✓ Stretched segment connectivity → VMs at Site-A and Site-B communicate on the stretched segment
✓ Location-aware DFW → Site-specific rules only appear on hosts at the target site
✓ Ingress path → BGP tables at both physical routers show where the stretched subnet is advertised and preferred; the ingress steering option chosen (primary location, AS-path prepend, per-location Tier-1 or GSLB) is documented
✓ Site failure independence → Surviving site operates normally during partition
✓ Recovery reconciliation → GM reconnects and reconciles after partition healing
Cleanup / Restore
• Delete stretched segments and VMs from GM
• Remove stretched T1 and T0 gateways from GM
• Deregister Local Managers from Global Manager (Location Manager → Remove)
• Power off Global Manager appliance
Design Reflection (VCDX)
Federation is a premium VCDX topic — it addresses multi-site design, disaster recovery, and operational complexity. Key design decisions: (1) GM placement — an active GM cluster (3 nodes, VIP) at one location plus a standby GM cluster at another location within 500 ms latency, both on management networks reachable from all Local Managers; (2) Stretched vs local objects — only stretch what needs cross-site L2 mobility; keep everything else local to minimize blast radius; (3) Location-aware DFW — enables global security baseline with site-specific exceptions; (4) RTEP design — cross-location traffic hairpins through the Edge nodes (RTEP), so Edge cluster placement, RTEP VLAN/IP pools and RTEP MTU on the WAN are design decisions, not host-TEP settings.
Requirements
- Consistent security policy across 2+ VCF sites
- Workload mobility between sites without IP change
- Independent site operations during WAN failure
Constraints
- Global Manager is a single point of policy management — its failure prevents global changes (but doesn't affect local operations)
- Stretched segments require RTEPs on the Edge clusters of every participating location, with RTEP-to-RTEP reachability and a consistent RTEP MTU (default 1500) across the inter-site path
- Location count per the VMware Configuration Maximums tool — a Medium GM is sized for up to four locations and 128 hypervisors; Large or Extra Large GMs cover higher scale (VCF 9.0.1 docs p.2757)
Assumptions
- Inter-site WAN provides reliable L3 connectivity with < 150ms RTT
- Both sites have compatible NSX versions (federation requires same major version)
- GM is deployed highly available: a 3-node active GM cluster with a VIP plus a standby GM cluster at the other location
Risks
- GM failure during a partition leaves both sites operating independently — any changes diverge and require manual reconciliation
- Stretched segment traffic trombone through remote site Edge if ingress steering (primary location choice, AS-path prepend, per-location Tier-1 or GSLB) is not designed
- Over-stretching resources (every segment, every policy) increases inter-site dependency and blast radius
Self-Assessment Discussion Prompts
- When would you choose NSX Federation over independent NSX domains with API-driven policy sync?
- How do you handle MTU constraints on a 1500-byte WAN for GENEVE stretched segments?
- What is the GM recovery procedure if the Global Manager cluster is permanently lost?
- How does Federation interact with VCF's Private Cloud → Fleet → Instance hierarchy?
Extensions
Deploy a standby GM cluster at Site-B, make it standby from the active GM (System > Location Manager > Add Standby) and test GM failover
Configure Avi (NSX ALB) GSLB to steer traffic to the closest site for a stretched application
Implement a DR runbook: planned failover of all workloads from Site-A to Site-B using stretched segments
Test the maximum number of stretched segments and DFW rules to understand federation scale limits
⚠ Known Pitfalls
References
- VCF 9.0 documentation — Multisite and NSX Federation (VCF 9.0.1 docs p.2274 onward; NSX Federation p.2289): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html
- VCF 9.0 documentation — Install the Active and Standby Global Manager (VCF 9.0.1 docs p.2299): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html
- VCF 9.0 documentation — Add a Location (VCF 9.0.1 docs p.2301): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html
- VCF 9.0 documentation — Configure Edge Nodes for Stretched Networking (RTEP) (VCF 9.0.1 docs p.2304): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0.html