NSX 4.x Network Virtualization Professional (2V0-41.24)

NSX Architecture Deep Dive#

Three-Plane Architecture

NSX 4.x separates control and management into distinct planes for scalability and resilience:

Management Plane: NSX Manager Cluster (3-node HA)

Cluster config:

3 NSX Manager nodes (6vCPU/24GB each) with floating VIP. All nodes serve API traffic concurrently (Active-Active).

Functions:

Policy management (segments, DFW rules, AD integration), inventory sync with vCenter, certificate management

Failure tolerance:

1 node down = cluster quorum maintained (2-of-3); 2 nodes down = cluster isolated (no config changes)

Database:

Shared PostgreSQL cluster (3-node, synchronous replication); all data updates replicate within <100ms

Control Plane: Central Control Plane (CCP) & Local Control Plane (LCP)

Central Control Plane (CCP):

NSX Controller cluster (3 nodes recommended). Responsible for:

BUM (Broadcast/Unknown-Unicast/Multicast) traffic handling delegation

Segment topology distribution (which host hosts which VMs in overlay)

VTEP (VXLAN Tunnel Endpoint) mapping for Geneve encapsulation

Local Control Plane (LCP):

• Runs on every ESXi host as kernel module (nsx_exec.o). Responsible for:
• Fast local switching decisions (VXLAN encap/decap in kernel)
• Distributed firewall rule evaluation (stateful inspection at kernel)
• Redundancy: If CCP lost, LCP maintains connectivity using cached topology

Data Plane: Host switch (VDS 7.0+ on ESXi; N-VDS only on Edge nodes) & TEP Tunneling

┌──────────────────────────────────────────────────────┐
│                    ESXi Host 1                       │
│  ┌────────────────────────────────────────────────┐ │
│  │ VMs: Web VM, App VM (on Overlay Segment)       │ │
│  └─────────┬──────────────────────────────────────┘ │
│            │ vNIC traffic                           │
│  ┌────────▼───────────────────────────────────┐    │
│  │  VDS 7.0+ with NSX (ESXi host switch)      │    │
│  │  • DFW rules applied (kernel)              │    │
│  │  • VXLAN encapsulation (Geneve proto)      │    │
│  └─────────┬────────────────────────────────┬─┘    │
│            │                                │       │
│  ┌────────▼────┐                  ┌───────▼──┐   │
│  │ TEP NIC 1    │                  │ TEP NIC 2 │   │
│  │ 192.168.1.11 │                  │ 192.168.2│    │
│  └──────────────┘                  │.11       │    │
│                                    └──────────┘    │
│                                                    │
│         TEP (VXLAN Tunnel Endpoint) uplinks       │
│         to physical leaf switches                 │
└──────────────────────────────────────────────────────┘

N-VDS:

Kernel-based virtual switch (replaces traditional vDS for NSX). NOTE: on ESXi the data plane is the VDS — N-VDS was deprecated in NSX-T 3.1.2 and removed for ESXi hosts in NSX 4.0.0.1; it remains only for Edge transport nodes. Multiple host switches per host support complex topologies (zone-isolated segments)

VXLAN encapsulation:

Geneve protocol (overlay) encapsulates logical packets in physical underlay. Header: IP(src=TEP_A, dst=TEP_B)/UDP(6081)/Geneve

TEP redundancy:

Minimum 2 TEP NICs per host (redundancy); can be same VLAN (convergence) or separate VLANs (isolation)

MTU:

Physical NICs must support MTU 1600+ (1500 original + 50 bytes Geneve + 50 bytes IP header)

NSX Edge Nodes

NSX Edge provides gateway services (Tier-0, Tier-1), VPN, load balancing:

Bare metal edge:

Dell/HP/Lenovo server running NSX Edge OS (Linux-based, minimal GUI). Typical: 16vCPU, 64GB RAM, 2x25G NICs. Throughput: >100 Gbps with crypto acceleration

VM-based edge:

NSX Edge as a large vSphere VM (8vCPU/16GB) for smaller deployments. Throughput capped: ~10-20 Gbps

Cluster topology:

2+ edges in Active-Active (HA) or Active-Standby. Active-Active requires BGP/ECMP load balancing across edges

Datapath:

Edge nodes also run LCP; maintain local routing table cache for sub-millisecond convergence on CCP loss
Exam insight: since NSX-T 2.4 the central control plane (CCP) runs INSIDE the NSX Manager appliance — there is no separate NSX Controller appliance to deploy. Production uses a 3-node NSX Manager cluster (quorum 2 of 3; a 2-node cluster is never supported), and VCF 9.0 also supports a single NSX Manager for reduced-footprint deployments.

Understand Management/Control/Data plane separation

Know TEP tunneling and the VDS data path (N-VDS was removed on ESXi hosts in NSX 4.0.0.1; it survives only inside NSX Edge nodes, public-cloud agents and bare-metal workloads)

Key Takeaways

• Exam insight: since NSX-T 2.4 the central control plane (CCP) runs INSIDE the NSX Manager appliance — there is no separate NSX Controller appliance to deploy. Production uses a 3-node NSX Manager cluster (quorum 2 of 3; a 2-node cluster is never supported), and VCF 9.0 also supports a single NSX Manager for reduced-footprint deployments.
• Exam question: "Your cluster has 200 hosts and broadcast storms causing network saturation. How do you optimize?" Answer: Switch to hierarchical replication + enable BUM rate-limiting on segment profile.
• T0/T1 Routing Model: T0 connects to physical network (edge nodes), T1 connects to segments (logical switches). ECMP design: use multiple T0s in active-active, BGP for load balancing. Edge node sizing: bare metal for >50 Gbps; VMs capped at 10-20 Gbps. Failure modes: T0 loss = no external connectivity; T1 loss = isolated segment but overlay unaffected.
• Federation Design Caveats: Multi-site NSX requires stretched L2 or L3 reachability to all sites (no WAN-only). Manager cluster must be co-located; stretched deployments increase RTT and failover latency (5-15 sec controller convergence). Recommendation: test stretched segments and failover scenarios with production-like latency before rollout.
• Network Evolution: NSX-V→NSX-T→NSX 4.x transition eliminated legacy modes. NSX 4.x requires native VXLAN/Geneve (no vLANs for underlay). MTU sizing critical: physical fabric must support 1600+ MTU. Holodeck lab constraint: edge node VMs limited to modest sizing; plan accordingly for resource-constrained environments.

NSX Logical Switching & Routing — Segments, Gateways, Transport Zones#

NSX logical switching and routing form the backbone of VCF networking. This topic covers segment creation, transport zone design, Tier-0/Tier-1 gateway architecture, and routing protocols.

Logical Segments (Overlay & VLAN)

Overlay Segments: GENEVE-encapsulated Layer 2 broadcast domains. VMs attached to the same overlay segment can communicate at L2 regardless of which ESXi host they reside on. Traffic between hosts is tunneled via TEP-to-TEP GENEVE encapsulation (UDP port 6081).

VLAN Segments: Direct VLAN-tagged segments for connecting to physical infrastructure (bare-metal servers, physical appliances, external networks). No encapsulation — traffic uses standard 802.1Q VLAN tags on the physical switch fabric.

Segment Profiles: Define QoS, IP Discovery, MAC Discovery, and Segment Security settings per segment. Key profiles:

• IP Discovery: ARP snooping and DHCP snooping to build IP-MAC mapping tables. Prevents ARP spoofing.
• MAC Discovery: Controls MAC learning behavior. MAC limit per port prevents MAC flooding attacks.
• Segment Security: BPDU filter, DHCP server/client blocking, rate limiting for broadcast/unknown-unicast/multicast (BUM) traffic.

BUM Traffic Handling: Three replication modes for overlay segments:

• Head Replication (Unicast): Source TEP replicates BUM frame to each remote TEP individually. Simple but doesn't scale beyond 100 hosts (N copies generated by source).
• Hierarchical Replication (MTEP): Source TEP sends one copy to a designated MTEP per VTEP group. MTEP replicates to local TEPs. Scales to 200+ hosts.
• Hybrid: Mix of head and hierarchical per segment. Use hierarchical for large segments, head for small.

Transport Zones

Transport Zone (TZ) defines the span of a logical switch — which hosts can participate in a given set of segments.

Overlay Transport Zone: Hosts that participate in GENEVE overlay networking. Each host can belong to one overlay TZ. TZ scope determines which hosts can host VMs on overlay segments.

VLAN Transport Zone: Hosts or Edge nodes that participate in VLAN-backed segments. A host/edge can belong to multiple VLAN TZs. Used for uplink connectivity to physical network.

Design Patterns:

• Single Overlay TZ (simple): All hosts in one overlay TZ. All segments reachable from all hosts. Suitable for single-domain deployments.
• Multi TZ (isolation): Separate overlay TZs per workload domain or tenant. Segments in TZ-A are invisible to hosts in TZ-B. Provides network-level isolation.
• Edge TZ: Separate VLAN TZ for Edge node uplinks. Isolates Edge traffic from host traffic.

Tier-0 Gateway — External Connectivity

Tier-0 (T0) gateway provides north-south routing between the NSX overlay and the physical network. T0 runs as a service router on NSX Edge nodes.

T0 HA Modes:

• Active-Standby: One active Edge, one standby. Stateful failover (sessions preserved). Use when stateful services (NAT, firewall, VPN) are required.
• Active-Active (ECMP): Multiple active Edges, each handling traffic. Traffic distributed via ECMP across Edges. Higher aggregate throughput but no stateful failover — sessions must be re-established on failure. Use for high-bandwidth, stateless routing.

Routing Protocols:

• BGP: Preferred for enterprise deployments. T0 peers with physical ToR/spine switches via eBGP. Supports route redistribution (connected, static, NAT, LB VIPs, T1 connected subnets). Multi-hop BGP for non-directly-connected peers.
• OSPF: Alternative to BGP. T0 participates in OSPF area. Suitable for environments where BGP is not permitted or physical switches only support OSPF.
• Static Routes: Manual route entries. Simplest but no dynamic failover. Use for lab/small environments.

Route Redistribution: Controls which routes T0 advertises to the physical network. Options: Connected subnets (T0 interfaces), T1 connected (subnets behind T1 gateways), NAT IPs, LB VIPs, static routes, IPSec VPN subnets.

Tier-1 Gateway — Tenant/Workload Routing

Tier-1 (T1) gateway provides routing for tenant/workload segments. T1 connects upstream to T0 for external access.

Design Patterns:

• One T1 per workload domain: Simple, clean isolation between domains.
• One T1 per application: Granular control. Each app (payroll, CRM, HR) gets its own T1 with dedicated NAT, LB, and firewall rules.
• One T1 per tenant: Multi-tenancy. Each tenant gets isolated routing and services.

T1 Services:

• NAT: SNAT (outbound masquerade) and DNAT (inbound service publishing). Configured per-T1.
• DHCP: Local DHCP server on T1 or DHCP relay to external server.
• DNS Forwarding: Conditional DNS forwarding rules on T1.
• Load Balancing: L4/L7 load balancing on T1 (or Avi ALB integration for advanced).
• Gateway Firewall: Stateful firewall rules on T1 for north-south traffic inspection.

Route Advertisement: T1 advertises its connected subnets, NAT IPs, and LB VIPs to T0. T0 then redistributes to physical network. This chain enables end-to-end reachability: Physical → T0 → T1 → Overlay Segment → VM.

NSX Edge Cluster Design

Edge Cluster: Logical grouping of Edge nodes that host T0 and T1 service routers.

Sizing Guidelines:

• Small/Lab: 2 Edge VMs (Medium: 4 vCPU, 8GB). Throughput: ~5 Gbps per Edge.
• Medium/Production: 2 Edge VMs (Large: 8 vCPU, 32GB). Throughput: ~10-20 Gbps per Edge.
• Large/High-Performance: 2+ Bare-Metal Edge nodes. Throughput: 100+ Gbps with DPDK/crypto acceleration.

Edge Placement Anti-Affinity: Edge VMs must run on separate ESXi hosts (anti-affinity rules). Edge VMs should NOT run on the same hosts as workload VMs if possible (dedicated Edge cluster recommended for production).

Edge Uplink Design: Each Edge node has uplink interfaces connected to VLAN TZ segments that connect to physical ToR switches. Redundant uplinks to two ToR switches for HA. BGP peering from both uplinks for ECMP load distribution.

Key Takeaways

• Overlay segments use GENEVE encapsulation (UDP 6081). VLAN segments use 802.1Q tagging. Transport zones control segment visibility across hosts.
• T0 gateway: north-south routing on Edge nodes. Active-Standby (stateful) or Active-Active ECMP (high throughput). BGP preferred for enterprise, OSPF as alternative.
• T1 gateway: per-tenant/workload routing with NAT, DHCP, DNS, LB, Gateway FW services. One T1 per domain/app/tenant depending on isolation requirements.
• BUM handling: Head replication for <100 hosts, Hierarchical (MTEP) for 200+ hosts. Critical for overlay performance at scale.

NSX Security Services, VPC, & Advanced Features in VCF 9.0#

NSX 4.x in VCF 9.0 introduces Virtual Private Cloud (VPC) constructs and enhanced security services beyond basic DFW and Gateway FW.

Virtual Private Cloud (VPC) in NSX

VPC is a new construct in VCF 9.0 / NSX 4.2+ that provides self-service network isolation for tenants within a VCF environment.

VPC Architecture: A VPC is created under an NSX Project. Each VPC gets its own isolated network space with: private IP subnets (RFC 1918), NAT gateway for external access, security policies (DFW rules scoped to VPC), and load balancer instances.

VPC vs Traditional Segments:

• Traditional: Admin creates segments, T1 gateways, DFW rules manually. Requires NSX expertise.
• VPC: Tenant self-service. Tenant creates subnets within their VPC allocation. NSX handles underlying segment, T1, and NAT configuration automatically.
• Use Case: Multi-tenant environments where each team needs isolated networking without requiring NSX skills.

NSX Projects: Organizational boundary for VPCs. A Project maps to a team or business unit. Projects use the system default overlay transport zone. Project admin can create VPCs, subnets, and security groups within their project scope.

NSX Intelligence & VCF Operations for Networks

NSX Intelligence: Analytics platform for network visibility (included with VCF Enterprise or vDefend license):

• Flow Visualization: Real-time and historical flow graphs showing VM-to-VM communication.
• Recommendation Engine: Suggests DFW rules based on observed traffic patterns.
• Security Posture: Dashboard showing unprotected VMs, open ports, and recommended segmentation.
• Micro-Segmentation Planning: Automated discovery of application boundaries from flow data.

VCF Operations for Networks (formerly Aria Operations for Networks / vRealize Network Insight):

• Cross-Platform Visibility: Network topology across NSX, physical switches, AWS VPC, Azure VNet.
• Path Analysis: End-to-end path from VM to external endpoint showing every hop, rule, and latency.
• Change Tracking: Audit trail of all NSX configuration changes with before/after comparison.
• Troubleshooting: Correlate network events with VM performance metrics from VCF Operations.

NSX Federation

NSX Federation enables consistent network and security policy across multiple NSX Manager instances (typically across datacenters).

Architecture:

• Global Manager (GM): Central policy manager for federation. Deploys as 3-node cluster (separate from local NSX Managers).
• Local Manager (LM): Per-site NSX Manager cluster. Executes policies received from GM.
• Stretched Segments: L2 segments spanning multiple sites via GM coordination. VMs on same segment at different sites communicate at L2.
• Stretched T0 Gateway: T0 spanning multiple sites for consistent north-south routing.

Federation Use Cases:

• DR with consistent network policy: Same DFW rules at both primary and DR site.
• Active-active datacenters: Workloads at both sites share network identity (same IP/MAC).
• Workload mobility: vMotion or HCX migration between federated sites without network reconfiguration.

Limitations: GM-LM communication requires low-latency connectivity (< 150ms RTT recommended). GM failure doesn't affect local LM operations (LMs continue independently). Stretched segments add complexity — plan BUM traffic handling and failure domain isolation carefully.

NSX Service Insertion

Service Insertion: Allows third-party network services (Palo Alto, Fortinet, Check Point) to be inserted into the NSX data path without changing the network topology.

Architecture: NSX redirects selected traffic to a partner service appliance for inspection, then returns it to the original path. Configuration: define service chain (which appliances, in what order), attach to DFW rules via service profiles.

Use Case: Organizations that require specific vendor firewalls for compliance but want to leverage NSX for transport and micro-segmentation.

NSX Load Balancing

Native NSX Load Balancer: L4 (TCP/UDP) and L7 (HTTP/HTTPS) load balancing on T1 gateways.

• Virtual Server: VIP + port + pool + health monitor.
• Pool: Group of backend servers with load distribution method (round-robin, least-connection, IP-hash).
• Health Monitor: HTTP/HTTPS/TCP/UDP/ICMP checks against pool members.
• Persistence: Source IP, cookie-based session persistence.

Avi ALB (Advanced Load Balancer): For advanced L7 requirements: WAF (Web Application Firewall), SSL offload, content-based routing, real-time analytics, autoscaling. Deployed as Avi Controller + Avi Service Engine (SE) VMs. Integrated with NSX and VCF Automation.

Key Takeaways

• VPC in VCF 9.0: Self-service network isolation for tenants. VPCs are created under NSX Projects. Each VPC gets private subnets, NAT, security policies, and LB — without requiring NSX expertise.
• NSX Federation: Global Manager (cross-site policy) + Local Managers (per-site execution). Enables stretched segments and consistent DFW policies across datacenters.
• NSX Intelligence provides flow-based micro-segmentation recommendations. VCF Operations for Networks provides cross-platform path analysis and change tracking.
• NSX LB: Native L4/L7 on T1 gateways for basic needs. Avi ALB for advanced (WAF, analytics, autoscaling).

DPU-Based Acceleration (NSX 4.x scope)#

DPU-based acceleration — vSphere Distributed Services Engine — offloads NSX datapath work from the host CPU onto a SmartNIC/DPU. SCOPE THIS TO NSX 4.x: see the EOL note at the end.

WHAT IS OFFLOADED (verbatim from the NSX 4.0.1.1 release notes)

├── Networking: overlay and VLAN segments, distributed IPv4/IPv6 routing, NIC teaming across DPU ports
├── Security (Tech Preview at 4.0.1.1): Distributed Firewall, Distributed IDS/IPS
└── Visibility & Ops: Traceflow, IPFIX, packet capture, port mirroring, statistics — plus UPT v2

VERSION PROGRESSION

├── NSX 4.1.0: DFW-on-DPU and UPT v2 promoted to GA
├── Distributed IDS/IPS on DPU: NEVER left Tech Preview
└── NSX 4.2.0: dual-DPU support (active/standby and active/active), superseding the earlier one-DPU-per-host limit

REQUIREMENTS

vLCM-managed cluster; vSphere 8.0+; NSX Enterprise Plus per-core licensing; Enhanced Data Path (EDP) host-switch mode; a supported NVIDIA BlueField-2 (25G, 100G from 4.1.1) or AMD Pensando (25G/100G) DPU.

END OF LIFE — DO NOT CARRY THIS INTO VCF 9.x
vDefend 9.0 announces End of Life for the vDefend Distributed Firewall for Data Processing Units (SmartNICs/DPU), foreshadowed in NSX 4.2.3. VCF 9.0's "What's New — NSX" contains zero mentions of DPU/SmartNIC/DSE/UPT, and the VCF 9.0 NSX admin guide has no vLCM/DSE chapter. Be precise about the distinction: Broadcom explicitly EOL'd the DFW engine running in the DPU, while the networking-offload half is simply ABSENT from 9.0 documentation rather than formally killed. The only DPU feature documented in VCF 9.0 is Uniform Passthrough (UPT) on NSX Edge datapath interfaces (Edge HW version 20+, ESX 8.0+).

Key Takeaways

• DPU offload (vSphere Distributed Services Engine) covers overlay/VLAN segments, distributed routing, NIC teaming, Traceflow, IPFIX, packet capture, port mirroring and statistics; DFW-on-DPU went GA in NSX 4.1.0 while Distributed IDS/IPS on DPU never left Tech Preview.
• Requires a vLCM cluster, vSphere 8.0+, EDP host-switch mode, NSX Enterprise Plus per-core licensing, and a supported NVIDIA BlueField-2 or AMD Pensando DPU. NSX 4.2.0 added dual-DPU (A/S and A/A).
• Scope DPU content to NSX 4.x: vDefend 9.0 declares EOL for the DFW-in-DPU engine, and VCF 9.0's NSX docs carry no DPU host-offload content at all. The only DPU feature in VCF 9.0 is UPT on Edge datapath interfaces.