Academy/VCF 9.0 Support (2V0-15.25)/NSX Connectivity Troubleshooting
This lab targets VCF 9.0

NSX Connectivity Troubleshooting

VCF 9.0Advancedvcp-foundationvcp-supportnsx-advanced⏱ 90 min

Advanced NSX data plane and control plane diagnostics.

Objectives

  • Map complete NSX data plane path (VM, segment, T1 router, T0 router, external gateway)
  • Understand NSX control plane dependencies (Manager cluster, CCP, LCP on transport nodes)
  • Use NSX Traceflow tool to trace packet path and identify drop reasons
  • Interpret nsxcli commands for router, firewall, and logical switch state validation
  • Diagnose transport endpoint (TEP) connectivity failures and BFD session health
  • Correlate control plane (Manager, CCP/LCP) issues with data plane connectivity loss
  • Document RCA (root cause analysis) and create troubleshooting runbooks for common NSX issues

Prerequisites

VCF 9.0 lab with NSX 9.0 deployed and operational (VCF 9.0 BOM ships NSX 9.0.0.0) (at minimum, 2 T1 routers, 1 T0 router, 2+ transport nodes)

Prior labs: NSX Fundamentals (basic segment creation, static routing)

Required skills:

  • NSX Manager UI navigation
  • SSH/CLI access to transport nodes (ESXi hosts or Edge nodes)
  • Basic network troubleshooting (IP routing, packet flow concepts)
  • Firewall rule logic and stateful vs. stateless packet handling

Lab Environment

Multi-tier NSX topology: VM-A on Tier-1-app, VM-B on Tier-1-db, both connected to Tier-0 router. Transport nodes: 2+ ESXi hosts or 2 Edge VMs. NSX Manager cluster (at least 1 node, 3+ recommended for HA).

  • Transport nodes needed: 2
  • Tier-0 routers needed: 1
  • Tier-1 routers needed: 2
  • Segments needed: 3

Tasks

Task 1 Map the NSX Network Path and Identify Control Plane Dependencies

Task requires multi-layer system thinking. Panelists will ask: (1) What happens to control plane if NSX Manager becomes unavailable? (2) How does CCP (Central Control Plane) differ from LCP (Local Control Plane)? (3) Where are the Service Routers in this east-west path, and when would one be used at all? (4) Can you articulate the difference between a T0 data plane issue vs. a control plane issue?

Develop a complete understanding of the data plane (packet forwarding) and control plane (configuration sync, state replication) paths. Demonstrate the ability to trace a packet from source VM through all routing, switching, and firewall layers to destination VM.

Step 1

Draw a detailed network topology diagram showing: VM-A on logical segment app-segment (connected to T1-app), VM-B on logical segment db-segment (connected to T1-db), both T1 routers connected to T0 router. Include physical transport nodes (ESXi hosts or Edge VMs) where each logical component resides.

Diagram shows: [VM-A on ESX Host 1] → [DFW on VM-A's vNIC] → [app-segment] → [T1-app DR] → [T0 DR] → [T1-db DR] (all three DR hops in the Host 1 kernel) → Geneve tunnel to Host 2 → [db-segment] → [DFW on VM-B's vNIC] → [VM-B on ESX Host 2]. Label each arrow with component function (switching, routing, firewall).
Use a tool like draw.io or Visio to create a professional diagram. Include physical host locations to show which components run on which transport node.
Step 2

Identify all components in the data plane path: (1) Distributed Firewall (DFW) at source and destination segments, (2) Logical switches (app-segment, db-segment), (3) T1 Service Router (T1-SR) and Distributed Router (T1-DR), (4) T0 Service Router (T0-SR) and Distributed Router (T0-DR), (5) Tunnel between transport nodes (if applicable).

Path documented: Packet from VM-A ingresses app-segment → hits DFW at ingress (security policy applied) → logical switch forwards to T1-app DR → T1-app DR routes to T0 DR (local path if on same host, or tunneled path if cross-host) → T0 DR routes to T1-db DR → T1-db DR routes to logical switch db-segment → DFW at egress (security policy applied) → delivers to VM-B. Total components in path: 7 (2 DFWs, 2 logical switches, 2 T1 routers, 1 T0 router).
Not all components are guaranteed to be in the path. If VM-A and VM-B are on the same T1 (not common in multi-tier), traffic may bypass T0 entirely.
Step 3

Identify control plane components: (1) NSX Manager Cluster (controls policy distribution), (2) Central Control Plane (CCP; since NSX 2.4 it runs inside the NSX Manager appliances of the three-node cluster), (3) Local Control Plane (LCP, runs on each transport node), (4) Edge Control Plane (if using Edge VMs). Document the communication path: Manager → CCP → LCP on each transport node.

Control plane architecture documented: NSX Manager cluster (3 nodes) publishes DFW rules and routing tables to CCP → CCP replicates to LCP on ESXi-1 and ESXi-2 → Each LCP (the local control plane daemons on the transport node) caches the state locally and programs the data plane: the DFW filters (vsip) on each vNIC and the distributed router in the forwarding engine; nsx-mpa is the management-plane agent, not a datapath module. If control plane link is broken, data plane uses cached rules but cannot accept new policies.
Rule realisation is normally quick but not instant, and there is no documented fixed figure: measure it in the lab (realisation status in NSX Manager, then vsipioctl getrules on the host). Until a new DFW rule is realised on a host, packets there still match the old rule set. Document this for panel discussion.
Step 4

For each layer in the path (DFW, logical switch, T1 DR, T1 SR, T0 DR, T0 SR), document: (a) Where it runs (which transport node), (b) What it does (function), (c) What can fail there (common failure modes).

Detailed layer analysis: (1) DFW at source: runs on ESXi-1, filters egress traffic, can drop packets if rule denies. (2) app-segment: logical switch on ESXi-1, learns MAC addresses, can lose MAC entries if control plane disconnects. (3) T1-app DR: distributed on ESXi-1, handles local routing, can have stale routing table if CCP link is down. (4) T1-app SR: centralized on Edge VM, used for NAT/services, can be unavailable if Edge VM crashes. (5) T0 components: similar failure modes at cluster interconnect layer.
Most NSX failures occur at control plane boundaries: rule caching inconsistency, routing table divergence, tunnel down.
Step 5

In NSX Manager UI, navigate to Networking > Connectivity > Segments. Verify that both app-segment and db-segment exist and are connected to their respective T1 routers. Verify T1 and T0 routers are connected. Document the state (READY, DOWN, DEGRADED).

UI shows: app-segment (READY, connected to T1-app, DFW attached), db-segment (READY, connected to T1-db, DFW attached), T1-app (READY, connected to T0), T1-db (READY, connected to T0), T0 (READY, uplink connected). All components green = healthy baseline.
If any component shows DEGRADED or DOWN, do not proceed with subsequent tasks. Resolve first.
Step 6

From NSX Manager, navigate to System > Fabric > Transport Nodes. Verify both ESXi hosts (or Edge VMs) show 'READY' state. Click each node and check: (a) CCP connectivity (should be READY), (b) VNI pool status (should have available VNIs), (c) TEP (tunnel endpoint) IP address and reachability.

Transport nodes show: ESXi-1 (READY, CCP: READY, TEP: 192.168.100.10/24), ESXi-2 (READY, CCP: READY, TEP: 192.168.100.11/24). Both TEPs should be on the same subnet and reachable via vmkping.
TEP reachability is prerequisite for NSX tunneling. If TEP is unreachable, all NSX traffic fails. Note TEP IP and VLAN for later CLI diagnostics.
Step 7

Document your complete topology map and control plane architecture in a written summary. Include: (1) ASCII diagram or reference to your saved diagram, (2) Data plane component list and function, (3) Control plane component list and communication path, (4) Baseline state of all components (READY), (5) Potential failure points and impact (if DFW is DOWN, which VMs are affected? If T0 SR fails, which traffic is rerouted?).

Written summary (500+ words) describing the complete NSX topology. Example: 'If NSX Manager CCP link is lost, the LCP on each transport node continues operating with cached state. New DFW rules cannot be pushed, but existing rules remain active. This is a data plane resilience feature: control plane is HA, but data plane degrades gracefully if control plane is unavailable (read-only mode).'
This summary demonstrates to VCDX panelists that you understand NSX as a system, not just a checklist of components.

Validation Gate

Check: Complete topology map created with identified data plane and control plane components

Expected: Lab notes contain detailed topology diagram, component-by-component analysis, documented baseline state, and written summary of NSX architecture

Common Errors

Treating all NSX routers as equivalent. T1 routers are per-tenant or per-app-tier; T0 routers are per-cluster backbone. They have different roles and failure implications.
Fix: Remember: T1 is local (VPC-like), T0 is global (core router-like). A failed T1 affects one app; a failed T0 affects all traffic.
Ignoring the control plane entirely. Focus solely on data plane 'packet flow' without understanding how rules are distributed and cached.
Fix: Control plane issues often manifest as data plane symptoms. Before troubleshooting packet drops, always confirm control plane is healthy (Manager UP, CCP READY, LCP synchronized).
Assuming east-west traffic traverses an Edge. T1-to-T1 traffic under the same T0 is routed by the DR instances on the source host (T1 DR → T0 DR → T1 DR) and never touches an Edge node; SRs enter the path only for centralised services (NAT, gateway firewall, LB) or north-south traffic. ('East-West Gateway' is not an NSX construct.)
Fix: Verify T1 router configuration: Does it have a T0 uplink? Does the T1 have an SR (an Edge cluster plus centralised services such as NAT, gateway firewall or load balancer)? Document the actual path, not the assumed path.

Task 2 Troubleshoot with NSX Traceflow

Traceflow is the most powerful NSX troubleshooting tool for data plane issues. Panelists will ask: (1) How does Traceflow simulate a real packet? (2) How do you read a Dropped observation — which component dropped the packet and why? (3) If Traceflow succeeds but real traffic fails, what could be the issue? (4) Can Traceflow detect transient connectivity issues or only sustained failures?

Master the Traceflow tool to diagnose connectivity issues and identify the exact point where packets are dropped (DFW, T1 router, T0 router, TEP failure, etc.). Learn to interpret Traceflow output to correlate drops with configuration errors.

Step 1

In NSX Manager select Plan & Troubleshoot > Traffic Analysis > Traceflow > Get Started (VCF 9.0.1 docs p.2499) to create a new trace from VM-A to VM-B.

Traceflow UI opens with source/destination VM selector. Both your test VMs (VM-A, VM-B) should be available in the dropdown.
Traceflow requires VMs to be registered with NSX. If VMs don't appear, they may not be on NSX-managed segments. Verify VM placement first.
Step 2

Configure the trace: Source = VM-A (on app-segment), Destination = VM-B (on db-segment). Protocol = TCP. Source Port = any, Destination Port = 443 (or any port where VM-B is listening). Action = 'Run Trace'.

Traceflow injects a trace packet and lists observations per hop: the table starts with an Injected observation, then Delivered or Dropped observations. Filter them with All / Delivered / Dropped (the Dropped filter is applied by default when drops exist). A DFW deny shows as a Dropped observation naming the firewall rule — there is no separate 'BLOCKED' state.
Delivered means every component forwarded the packet to the destination port. Dropped names the component and reason — a firewall rule, no route, a missing ARP entry — so read the reason, not just the state.
Step 3

If trace is DELIVERED, examine the detailed hop-by-hop results: (1) Packet enters DFW at source (rule applied, allowed). (2) Packet switches through app-segment (MAC learned, forwarded to T1 DR port). (3) T1-app DR routes to T0 (route lookup succeeded, next-hop is T0 DR). (4) T0 DR routes to T1-db (return route lookup succeeded). (5) T1-db DR routes to db-segment (MAC lookup, forwarded to VM-B port). (6) DFW at destination checks rule (allowed). (7) Packet delivered to VM-B.

Traceflow hops document: [DFW@app: ALLOW] → [Switch@app: FORWARD] → [T1-app DR: ROUTE to T0] → [T0 DR: ROUTE to T1-db] → [T1-db DR: ROUTE to db-segment] → [Switch@db: FORWARD] → [DFW@db: ALLOW] → [DELIVERED]. The graphical map shows the backplane and router links; no latency estimate is part of the output.
Each hop is logged with rule ID, route entry, MAC address. If a hop fails, note the exact layer (e.g., 'DFW rejected at ingress due to rule <security-policy-ID>').
Step 4

Simulate a failure scenario: In NSX Manager, navigate to Security > Distributed Firewall (p.2612). Find the DFW rule allowing app-to-db traffic. Create a NEW rule ABOVE it that explicitly DENIES the same traffic (to simulate misconfiguration). Save the rule.

DFW rule added in priority order: Rule-1 (DENY app-to-db), Rule-2 (ALLOW app-to-db). Rules are evaluated top-to-bottom; Rule-1 matches first and drops traffic.
DFW rule order matters. This is a common misconfiguration: admin adds a blanket DENY rule and forgets to adjust priorities.
Step 5

Re-run Traceflow (VM-A to VM-B, TCP port 443). The trace now shows a Dropped observation at the DFW on the source port. Examine the detailed output: 'Dropped — Firewall, rule <ID>' with the rule's policy name.

Traceflow output: [Injected] → [Dropped: Firewall, rule vcp-support-07.deny-app-db, policy app-deny-policy]. The trace identifies the exact rule dropping the traffic. Rule ID can be used to locate the misconfigured rule in NSX Manager.
Real troubleshooting: When a user reports 'can't reach VM-B from VM-A,' run Traceflow. It immediately identifies the blocking layer and rule. Fix the rule (delete or reprioritize) and re-run Traceflow to confirm.
Step 6

Fix the DFW rule by deleting the deny rule or moving the allow rule above it. Re-run Traceflow to confirm the trace now shows DELIVERED.

Trace reverts to DELIVERED. This demonstrates the fix and validates the remediation.
In a real incident, you'd communicate with the app team: 'The DFW rule policy-XYZ was blocking traffic. I've corrected the rule priority. Traffic should now flow. Please test from your application and confirm.'
Step 7

Simulate a routing failure: in NSX Manager edit T1-db > Route Advertisement and clear 'All Connected Segments & Service Ports' (or detach db-segment from T1-db). Segments reach the T0 through the auto-plumbed router link and T1 route advertisement, not through T0 static routes or BGP/OSPF (those are north-south protocols), so there is normally no T0 static route to delete. Apply the change temporarily to simulate routing failure.

Route removed. The next Traceflow attempt from VM-A to VM-B will show a Dropped observation at the Tier-1 DR routing hop (no route).
East-west routing failures inside NSX are usually configuration errors — a segment detached from its T1, route advertisement switched off, or an overlapping subnet — rather than protocol failures; BGP/OSPF matter only for the T0's north-south peering.
Step 8

Re-run Traceflow. Observe the result: [T1-app DR: DROP (no route to <db-segment-subnet>)] → [DROPPED]. This identifies the routing issue. Restore the route, re-run Traceflow to confirm recovery.

Traceflow shows DROP at routing layer. Pinpoints exact layer (T1 vs. T0 router). After restoring route, trace shows DELIVERED.
Traceflow injects a real trace packet that traverses the overlay but is not delivered to the recipient VM; it shows exactly where the network drops it. Use it to build confidence in configuration changes before deploying to production.
Step 9

Document your Traceflow methodology: (1) Always run Traceflow bidirectionally (A→B and B→A) if traffic is bi-directional. (2) Test common protocols (TCP/UDP, common ports 80, 443, 3306). (3) If Traceflow succeeds but real traffic fails, suspect MTU issues, TCP option stripping, or asymmetric routing. (4) Set Advanced Settings > Frame Size (default 128) to the overlay-sized value you want to test; a trace with a 1600-byte frame exercises the TEP MTU, while the default 128-byte trace will not.

Documented methodology: 'Traceflow Test Procedure: (1) Bidirectional traces (A→B, B→A). (2) Common protocols: TCP 80, 443; UDP 53, 123. (3) Success criteria: all traces DELIVERED and rules are expected. (4) If real traffic fails despite DELIVERED trace, run: vmkping (TEP connectivity), tcpdump on VTEP (packet capture), guest OS routing table (ensure guest has return route).'
This methodology is your Traceflow troubleshooting SOP (Standard Operating Procedure). Document it for your lab reference and panel discussion.

Validation Gate

Check: Traceflow used to diagnose multiple connectivity scenarios (baseline, DFW block, routing failure)

Expected: Lab notes document Traceflow output for each scenario, interpreted drop reasons, and remediation steps

Common Errors

Relying solely on Traceflow for diagnosis. Traceflow injects a real trace packet but it never reaches the guest; it will not show guest-OS, application or return-path problems, and it tests only the frame size you set in Advanced Settings.
Fix: If Traceflow succeeds but real traffic fails, use tcpdump/Wireshark to capture actual packets. Compare packet size, TCP flags, and return path in real network.
Assuming DELIVERED means the application is working. Traceflow validates L3-L7 policy and routing, but doesn't test application-level protocols (is port 443 actually listening? Is TLS handshake working?).
Fix: DELIVERED trace is necessary but not sufficient. Follow up with: SSH to VM-B, verify listening ports (netstat -an), test connectivity (telnet <VM-B-IP> 443, curl, nc).
Not using Traceflow bidirectionally. Traffic A→B succeeds but B→A fails due to asymmetric routing or reverse DFW rule misconfiguration.
Fix: Always test both directions. Document which direction fails. Asymmetric failures often indicate return path issues or missing stateful DFW rules.

Task 3 CLI-Based Troubleshooting and Transport Node Diagnostics

CLI troubleshooting is the gold standard for deep diagnostics. Panelists will ask: (1) How do you validate that LCP has synchronized the DFW rules from Manager? (2) What does a healthy BFD session look like? (3) How would you diagnose a TEP network partition? (4) Can you correlate an nsxcli 'get logical-switch' output with a Traceflow drop?

Master nsxcli commands and lower-layer diagnostics to validate control plane sync, routing tables, firewall state, and TEP connectivity when UI-based tools are unavailable or slow.

Step 1

SSH to one of your transport nodes (ESXi host or Edge VM). Log in as root or a user with admin privileges.

SSH session established. Prompt shows: '[root@<host>]# ' or similar, ready for commands.
Ensure you have nsxcli available. On ESXi, nsxcli is in the default PATH. On Edge VMs, it may be in /opt/vmware/nsx/bin/ or bundled with the system.
Step 2

Run 'get managers' to verify the host can reach NSX Manager cluster. Output should show Manager node(s) with status CONNECTED.

Output: 'Manager IP: 192.168.100.200, Status: CONNECTED, Last Update: 30s ago'. If status is DISCONNECTED or UNKNOWN, the LCP lost connection to Manager. This is a critical control plane issue.
If Manager is DISCONNECTED, check network connectivity: vmkping <Manager-IP>. If vmkping fails, network path to Manager is down. If vmkping succeeds, suspect Manager process issue (NSX Manager app may be down or overloaded).
Step 3

Run 'get logical-switches' to list logical switches on this host (NSX CLI Guide 9.0.0, ESXi commands). Output should show app-segment and db-segment (or whichever segments are connected to this host's VMs).

Output: '1. app-segment (UUID: <UUID>, VLAN: 0, VNI: 5000, State: UP, MAC Entries: 3). 2. db-segment (UUID: <UUID>, VLAN: 0, VNI: 5001, State: UP, MAC Entries: 2)'. Each segment shows VNI (virtual network ID assigned by NSX), state (UP=healthy, DOWN=control plane issue), and MAC table size.
MAC Entries = number of MAC addresses learned on this segment. If you have 3 VMs on the segment, expect ~3 MAC entries. If 0, segment is not forwarding traffic.
Step 4

Run 'get logical-routers' to verify T1 and T0 routers are visible to this host (NSX CLI Guide 9.0.0, ESXi commands). Output should list all T1 routers and the T0 router.

Output: '1. T1-app (UUID: <UUID>, Type: TIER1, Status: UP). 2. T1-db (UUID: <UUID>, Type: TIER1, Status: UP). 3. T0 (UUID: <UUID>, Type: TIER0, Status: UP)'. Each router shows its type and operational status.
If a T1 router shows status DOWN, its LCP is not operational on this host. This would indicate control plane sync failure or a LCP process crash.
Step 5

Take the T1-app router's UUID from 'get logical-routers', then run 'get logical-router <uuid> forwarding' to examine its forwarding-table entries ('get logical-router <uuid> interfaces' lists its interfaces; NSX CLI Guide 9.0.0, ESXi commands). Look for routes to db-segment subnet and upstream routes to T0.

Output: the 'Logical Routers Forwarding Table' of the T1-app DR (selected by its UUID), with columns Network, Gateway, Type (flags such as U Up, G Gateway, C Connected, I Interface) and Interface UUID. The connected app-segment appears as '192.168.1.0/24 0.0.0.0 UCI <interface UUID>'; routes towards the T0 point at a 100.64.x.x transit next hop with the G flag, and that is the path traffic to db-segment (10.2.0.0/24) must match (NSX CLI Guide 9.0.0, HostShowLogicalRouterRoutes). Each entry shows a destination prefix and next hop, validating the routing configuration.
If an expected entry is missing (e.g., nothing covers the db-segment subnet or points towards the T0), it indicates a control plane sync failure. The LCP cache is stale or incomplete.
Step 6

Inspect the realised DFW rules on this host. Find the VM's filter with 'summarize-dvfilter' (look for the vNIC's nic-… name), then 'vsipioctl getrules -f <filter-name>' lists the rules realised for that vNIC; 'nsxcli -c "get firewall <vif-uuid> ruleset rules"' is the nsxcli form. There is no host-wide 'get firewall-rules' command and no PENDING/INSTALLED rule state on the host — realisation status is read in NSX Manager (policy > realisation status). Command forms: 'get firewall <vif-uuid> ruleset rules' per the NSX CLI Guide 9.0.0 (ESXi commands); summarize-dvfilter and vsipioctl getrules per Broadcom KB 379438 (Troubleshooting Distributed Firewall).

Output lists the rules realised on VM-A's vNIC filter with their rule IDs, e.g. 'rule 1023 at 1 inout protocol tcp from addrset … to addrset … port 443 accept' and the default rule; a rule missing from the filter means it was not realised on this host.
If a published rule is missing from the vNIC filter, check the policy's realisation status in NSX Manager (Success / In Progress / Failed) and 'get managers' on the host; a Failed realisation names the host and the error.
Step 7

List the TEP interfaces on this host: 'esxcli network ip interface ipv4 get' (TEP vmkernel ports sit in the vxlan netstack) and 'esxcli network ip interface list' for the MTU; count the remote TEPs with 'nsxdp-cli bfd sessions list'. ('get tunnel-interfaces' / 'get vtep-interfaces' did not surface in the NSX documentation.)

Output shows the TEP vmkernel port (e.g. vmk10, 192.168.100.10/24, netstack vxlan, MTU 1700) and one BFD session per remote TEP. The TEP MTU must be at least 1700 on VCF TEP VLANs (VCF-NET-REQD-MTU-006; Geneve itself needs 1600); a TEP at 1500 drops large encapsulated frames.
If Active Tunnels = 0, this host is isolated from other hosts in the NSX cluster. Data plane traffic between VMs on different hosts will fail. Check TEP network connectivity from the TEP netstack: vmkping ++netstack=vxlan -I vmk10 192.168.100.11 (the other host's TEP).
Step 8

Check the TEP-to-TEP BFD sessions: on an ESX host run 'nsxdp-cli bfd sessions list' (append '| grep down' to filter failures); 'get bfd-sessions' is the Edge transport-node command (VCF 9.0.1 docs p.2042). BFD is NSX's keep-alive mechanism for tunnel health. Output should show all BFD sessions as UP.

Output lists one BFD session per remote TEP, e.g. 'local 192.168.100.10 → remote 192.168.100.11 state up' and 'local 192.168.100.10 → remote 192.168.100.12 state up' (sessions are per remote TEP, not per T0/T1 tunnel). State up = the Geneve path is forwarding and keep-alives are exchanged.
If BFD shows DOWN, the tunnel to that peer is broken. This is a critical data plane failure. Correlate with vmkping to TEP and network diagnostics (network partition, firewall blocking UDP 3784).
Step 9

Test TEP-to-TEP reachability from the TEP netstack (the TEP vmknics live on the vxlan netstack, so a plain vmkping leaves by the default stack and does not test the TEP path): 'vmkping ++netstack=vxlan -I vmk10 192.168.100.11'. Then prove the overlay MTU with a large don't-fragment packet: 'vmkping ++netstack=vxlan -I vmk10 -d -s 1600 192.168.100.11' (VCF 9.0.1 docs p.2780). Both should show 0% loss.

Output: 'PING 192.168.100.11 (192.168.100.11) 56(84) bytes of data. 64 bytes from 192.168.100.11: icmp_seq=1 ttl=64 time=2.34ms. 0% loss, avg latency 2.3ms'. A successful netstack ping confirms the TEP path; a successful -d -s 1600 ping confirms the path carries Geneve-sized frames.
If vmkping fails (100% loss), the TEP network is partitioned. Check the ESX network stack: esxcli network ip interface list (verify vmk10 is enabled and on the vxlan netstack), esxcli network ip route ipv4 list -N vxlan (verify the route to the TEP subnet). If the plain ping works but the -d -s 1600 ping fails, the physical path MTU is below 1600. If network is healthy but vmkping fails, suspect firewall policy on network switch blocking traffic.
Step 10

For Edge VMs (if used for T0 or T1 services), run 'get controllers' to verify the Edge VM's connection to NSX Manager's Control Plane. Output should show CONNECTED.

Output: 'Controller: 192.168.100.200 (nsxmanager-node-1), Status: CONNECTED, Role: central-control-plane'. If DISCONNECTED, the Edge VM cannot receive configuration updates from Manager.
Edge VMs are mission-critical. If the Edge loses its Manager/control-plane connection, existing T0/T1 services keep forwarding on the last realised configuration (headless operation) but no changes can be pushed — the lab's own risk mitigation says the same. Keep at least 2 Edge nodes in the Edge cluster for HA.
Step 11

Document your nsxcli diagnostics workflow: (1) Check Manager connectivity (get managers). (2) List logical switches/routers (get logical-switches, get logical-routers). (3) Inspect forwarding tables (get logical-router <uuid> forwarding). (4) Verify firewall rules (summarize-dvfilter + vsipioctl getrules -f <filter>, realisation status in NSX Manager). (5) Validate TEP connectivity (esxcli network ip interface list, vmkping ++netstack=vxlan). (6) Confirm BFD health (nsxdp-cli bfd sessions list).

Documented workflow in lab notes. Example: 'NSX CLI Troubleshooting SOP: (1) Manager UP? (2) Switches/Routers UP? (3) Routes valid? (4) Firewall rules realised on the vNIC filter? (5) TEP reachable (vmkping)? (6) BFD sessions UP? If any step fails, escalate to that layer's issue.'
This workflow is your CLI troubleshooting checklist. Use it during incidents to systematically validate control and data plane health.

Validation Gate

Check: CLI commands executed on transport node(s) with documented output and interpretation

Expected: Lab notes contain nsxcli command output for logical switches, routers, firewall rules, TEP interfaces, BFD sessions, and documented health assessment

Common Errors

Reading a rule's presence in the vNIC filter as proof that it is current. The filter shows the rules realised on this host, but a stale realisation looks the same as a fresh one.
Fix: Compare the rule IDs from vsipioctl getrules with the rule IDs shown in NSX Manager and check the policy's realisation status (Success / In Progress / Failed) and the host's 'get managers' state; In Progress or Failed means the host has not received the latest version.
Assuming TEP network is OK if vmkping to one peer succeeds. TEP might have asymmetric paths (A→B works, B→A fails).
Fix: vmkping is ICMP-based. NSX overlay tunnels use Geneve (UDP 6081, RFC 8926), and the TEP-to-TEP BFD sessions ride the same path; a firewall may allow ICMP but block Geneve. Use 'nsxdp-cli bfd sessions list' on the host (or 'get bfd-sessions' on an Edge) to verify tunnel health is actually up (the fabric BFD status reflects the network's ability to forward Geneve packets, VCF 9.0.1 docs p.2042), not just assumed.
Not distinguishing between 'Manager disconnected' (control plane failure) and 'Rule sync failure' (data plane stale but operational). A disconnected Manager is critical; stale rules are recoverable.
Fix: If Manager is disconnected, data plane uses cached rules (read-only mode). If Manager is connected but the policy's realisation status stays In Progress or Failed, suspect the Manager → host control-plane channel or an overloaded Manager.

Task 4 Remediation, Validation, and RCA Documentation

This task emphasizes the full troubleshooting cycle: diagnose → fix → validate → document. Panelists will ask: (1) How do you ensure a fix doesn't break something else? (2) How long do you test the fix before declaring the incident closed? (3) What is the difference between a temporary workaround and a permanent fix? (4) How would you automate this troubleshooting in a CI/CD pipeline?

Apply fixes to identified issues, validate remediation with Traceflow, and document root cause analysis in a structured format suitable for runbooks and incident post-mortems.

Step 1

Based on your diagnostics from Tasks 2-3, identify the root cause of the connectivity issue (e.g., DFW rule misconfiguration, routing table incomplete, TEP network unreachable). Document the exact issue in a brief RCA statement.

RCA statement: 'Traffic from VM-A to VM-B fails due to missing DFW allow rule in policy app-to-db. Traceflow showed Dropped (Firewall) at the source DFW, rule ID <XXXX>. NSX Manager showed realisation Failed for the policy (invalid CIDR in the rule's destination group) and the rule was absent from the vNIC filter on the host. Root cause: DFW rule syntax error during initial deployment.'
Distinguish root cause from symptom. Symptom: 'User says app is slow.' Root cause: 'DFW rule drop rate is 50% due to misconfigured range.'
Step 2

Apply the fix. Example: If DFW rule is misconfigured, correct the rule in NSX Manager > Security > Distributed Firewall. Edit rule, fix CIDR block, save and publish.

DFW rule updated and published. NSX Manager confirms rule is ACTIVE (published to Manager cluster). LCP on transport nodes will receive updated rule within seconds.
Always test rule changes in a staging environment first (if available). In a VCDX interview, emphasize change control: 'I would create a change request, have peer review the rule, test in staging, then deploy to production during maintenance window.'
Step 3

Monitor realisation: in NSX Manager the policy's realisation status moves from In Progress to Success, then on the ESX host confirm the corrected rule appears in the VM's vNIC filter (vsipioctl getrules -f <filter>). Allow for realisation time (measure it; no fixed figure is documented) before testing.

NSX Manager shows realisation Success and the host's vNIC filter lists the corrected rule. Publication-to-realisation is typically seconds.
If realisation does not reach Success within a minute, suspect the Manager-to-host control-plane channel. Check 'get managers' on transport node; if status is DISCONNECTED, diagnose Manager connectivity first.
Step 4

Re-run Traceflow from VM-A to VM-B with the same parameters as before (TCP, port 443). Trace should now show DELIVERED (all hops pass, no blocks).

Traceflow output: [DFW@app: ALLOW] → [... (hops)] → [DELIVERED]. Compare this output to the pre-fix trace to confirm the fix resolved the specific drop point.
If trace still shows BLOCKED at a different layer (e.g., T1 DR routing), there are multiple issues. Continue troubleshooting to resolve all identified problems before considering the incident closed.
Step 5

Validate the fix at the application layer. SSH to VM-B and verify the service is listening on port 443. From VM-A, attempt to connect to VM-B: 'telnet <VM-B-IP> 443' or 'curl https://<VM-B-IP>' (if web service). Connection should succeed.

Connection succeeds. Example: 'Connected to 192.168.2.10:443. HTTP 200 OK' or similar application-level success. This confirms not just network connectivity (Traceflow), but end-to-end functionality.
Some teams rely solely on Traceflow validation. Always do application-level testing to confirm the user-reported issue is actually resolved.
Step 6

Document the RCA in a structured format: (1) Incident Date/Time. (2) Symptom (user report or automated alert). (3) Timeline (when incident was detected, when troubleshooting started, when fix was applied). (4) Root Cause (technical explanation). (5) Impact (how many VMs/users affected, downtime duration). (6) Fix Applied (specific configuration change). (7) Validation (Traceflow + application test). (8) Prevention (how to avoid this in the future).

RCA document (500+ words): 'Incident: VM-A → VM-B connectivity loss on 2024-12-15 10:30 UTC. User Report: 'Application server cannot reach database.' Symptom: Traceflow VM-A → VM-B showed Dropped — Firewall (rule XYZ). Root Cause: DFW rule deployed with typo in destination CIDR (192.168.2.0/24 instead of 192.168.20.0/24). Impact: 1 application server, 5000 users affected, ~45-minute downtime. Fix: Corrected CIDR in DFW rule, re-published. Validation: Traceflow DELIVERED, application connectivity restored. Prevention: Implement DFW rule deployment pipeline with syntax validation and staging environment testing.'
This RCA format is industry-standard (used by Google SRE, AWS, etc.). Panelists will recognize and appreciate the structure.
Step 7

Create a troubleshooting runbook entry based on this incident. Runbook should be a step-by-step guide that a junior engineer or NOC (Network Operations Center) tech can follow to diagnose the same issue in the future.

Runbook entry: 'NSX Connectivity Troubleshooting Runbook > DFW Rule Misconfiguration. Symptom: Ping fails between VMs on different T1s. Step 1: Run Traceflow. If Dropped at the source DFW, proceed to Step 2. Step 2: Get rule ID from Traceflow. Step 3: check the policy's realisation status in NSX Manager, then SSH to the host and run vsipioctl getrules -f <filter> | grep <rule-ID> to confirm the rule and its CIDR as realised. Step 4: If CIDR is wrong, escalate to Network Security team to correct. Step 5: After fix, re-run Traceflow to confirm DELIVERED. Step 6: Test application connectivity (ping, telnet, curl). Success = incident closed.'
Runbooks are VCDX-gold. Panelists want to see evidence that you're building organizational knowledge from incidents, not just fixing today's problem.
Step 8

If the issue was a design flaw (not a one-off misconfiguration), recommend architectural changes. Example: If T1 routers lack redundancy, design and propose a HA T1 setup. If TEP network is single-home, design a dual-homed TEP network.

Architectural recommendation: 'A Tier-1 gateway is one logical router: its DR runs on every host and its SR, if it has centralised services, runs active/standby across two Edge nodes in the Edge cluster — so the fix is not a second T1 but an Edge cluster of at least two nodes, a T1 HA mode choice (preemptive/non-preemptive) and BFD on the Edge uplinks. A T1 with no services has no SR and nothing to make HA. Cost and benefit: state the Edge node footprint and the measured failover time from your test, tied to the application RTO — not a generic uptime percentage.'
This is where you distinguish yourself as a senior engineer. VCDX panelists want to see you think beyond immediate fixes to systemic improvements.

Validation Gate

Check: Issue fixed, remediation validated with Traceflow, RCA documented, runbook created

Expected: Lab notes contain complete RCA document (symptom, root cause, fix, validation, prevention) and troubleshooting runbook entry suitable for team reference

Common Errors

Fixing the symptom without addressing the root cause. Example: Adding a static ARP entry to work around a network discovery issue, instead of fixing the underlying DHCP or VLAN misconfiguration.
Fix: Always ask 'Why?' three times before applying a fix. Why is the DFW rule misconfigured? (Typo in rule creation.) Why did the typo make it through QA? (No rule syntax validation in deployment pipeline.) True fix: Implement validation automation.
Not testing the fix thoroughly. A DFW rule fix may unblock the intended traffic but inadvertently open a security hole. Always test both positive (intended traffic flows) and negative (unintended traffic is still blocked) cases.
Fix: For DFW rules, test: (1) Correct traffic flows (bidirectional). (2) Incorrect traffic is blocked (e.g., wrong IP range still blocked). (3) No unintended side effects (other rules still working).
Declaring the incident closed too early. Fix is applied, but validation is incomplete (only Traceflow tested, not actual user traffic). Weeks later, user reports issue persists.
Fix: Validation checklist: (1) Traceflow DELIVERED. (2) Application-level test successful (telnet, curl, etc.). (3) Full workload test (run user workflow end-to-end). (4) Monitor for 24 hours (verify fix is stable, no anomalies). Only then close incident.

Final Validation

Lab completed when all 4 tasks are executed and fully documented with topology maps, Traceflow tests, CLI diagnostics, RCA, and runbooks.

✓ Complete NSX topology mapped with data plane and control plane identified → Task 1 complete: diagram shows all components (VM, DFW, logical switches, T1/T0 routers, transport nodes, Manager, CCP, LCP)

✓ Traceflow used to diagnose baseline, failure, and remediation scenarios → Task 2 complete: Traceflow traces documented showing Delivered (healthy), Dropped — Firewall rule (DFW misconfiguration), Dropped — no route (routing failure), and remediated Delivered

✓ nsxcli commands executed to validate control and data plane state → Task 3 complete: Output from get managers, get logical-switches, get logical-routers, vsipioctl getrules, esxcli network ip interface list, nsxdp-cli bfd sessions list, vmkping documented and interpreted

✓ Issue remediated and validated with Traceflow + application testing → Task 4 complete: RCA document (symptom, root cause, fix, impact, prevention), troubleshooting runbook entry, and architectural recommendations for future prevention

Cleanup / Restore

• Remove any temporary DFW rules created during testing (deny rules added for failure simulation)

• Restore any deleted logical switches, routers, or routes (use NSX Manager snapshots if available)

• Verify VM-A and VM-B are back to baseline state (connectivity DELIVERED, no policy blocks)

• Document any permanent architectural changes recommended (for infrastructure team to implement)

Design Reflection (VCDX)

This lab emphasizes NSX troubleshooting methodology and operational excellence. VCDX panelists will probe: (1) How do you debug control plane issues when UI is slow or unavailable? (2) What is your mental model of NSX data flow and where failures occur? (3) Can you prioritize between control plane (Manager, CCP/LCP) failures and data plane (DFW, routing, TEP) failures? (4) How do you design NSX monitoring to detect these issues proactively? The RCAR framework emphasizes root cause analysis and architectural thinking, not just tactical fixes.

Requirements

  • NSX 9.0 Manager cluster (VCF 9.0 BOM; 1 node in a lab, 3 for HA testing)
  • 2+ transport nodes (ESXi hosts or Edge VMs) for tunnel/TEP testing
  • 2 logical segments (app-segment, db-segment)
  • 2 tier-1 routers (T1-app, T1-db) and 1 tier-0 router (T0)
  • Test VMs: VM-A on app-segment, VM-B on db-segment
  • Firewall rules defining app-to-db policy
  • SSH/CLI access to transport nodes for nsxcli commands

Constraints

  • NSX Traceflow only works for VMs registered with NSX; bare-metal or external devices require packet capture (tcpdump) instead
  • DFW rule changes take time to realise from Manager to transport nodes; tests must wait for realisation status Success (and confirm on the host) before judging a rule
  • TEP network must be routable; if TEP network is not L2-adjacent, tunneling requires L3 routing setup (may not be available in lab)
  • BFD/tunnel health is binary (UP/DOWN); subtle issues like high latency or packet loss may not show as DOWN immediately
  • nsxcli commands vary by host type (ESXi vs. Edge VM); command syntax differs between NSX versions

Assumptions

  • NSX Manager and transport nodes have network connectivity (at baseline)
  • Lab operator has admin-level access to NSX Manager and transport node CLIs
  • VMs can reach each other if no security policies interfere (i.e., baseline connectivity works before introducing deliberate failures)
  • Firewall policies are deployed and synchronized before the lab starts (Manager → LCP sync is operational)
  • RCA and runbook documentation are stored in a centralized repository (Confluence, GitHub, etc.) for team access

Risks

  • Risk: Breaking actual connectivity during failure simulation. If you delete a real route or firewall rule, production traffic could fail. — Mitigation: Use a dedicated lab environment, not shared infrastructure. Snapshot the NSX Manager config before destructive tests. Have a revert plan (snapshot or config backup).
  • Risk: Misinterpreting Traceflow results. A DELIVERED trace does not guarantee real traffic will succeed (MTU, TCP options, asymmetric paths). — Mitigation: Always follow Traceflow with application-level testing (telnet, curl, packet capture). Document limitations of Traceflow in your runbook.
  • Risk: Control plane issues (Manager down, LCP disconnected) cascading to data plane. If Manager is unavailable, LCP goes read-only; new policies cannot be deployed. — Mitigation: Test with Manager deliberately unavailable (simulate network partition). Understand graceful degradation: data plane uses cached rules, services degrade but do not fail completely.
  • Risk: TEP network partition causing split-brain. Host A can reach Host B, but Host B cannot reach Host A due to asymmetric routing. Traceflow from A→B succeeds, but real traffic fails both ways. — Mitigation: Test bidirectional paths from both hosts with 'vmkping ++netstack=vxlan -I vmk10' and confirm BFD status with 'nsxdp-cli bfd sessions list' on each host ('get bfd-sessions' on an Edge); one-way ICMP success is not tunnel health.

Self-Assessment Discussion Prompts

  1. In the DFW misconfiguration scenario, you identified the issue via Traceflow. But in a real incident, what would prevent you from using Traceflow (e.g., VMs not NSX-registered, NSX Manager down)? What is your backup diagnostic method?
  2. You validated the fix with Traceflow DELIVERED + application test. But a week later, user reports intermittent failures (traffic works sometimes, fails sometimes). Traceflow always shows DELIVERED. What could cause transient failures that Traceflow doesn't catch?
  3. In your RCA, you recommended adding a second T1 SR Edge VM for HA. What is the cost/benefit? What are the design trade-offs? How would you validate that HA failover actually works under load?
  4. Your lab uses static routing (T1 → T0). In production the T0 peers with the physical network over BGP (OSPF is also supported on the T0); the T1 → T0 link stays auto-plumbed. What are the pros and cons of BGP versus static default routes on the T0, and how does that change your north-south troubleshooting?
  5. You ran Traceflow and got DELIVERED, but NSX Manager still shows the policy's realisation as In Progress on that host. Does Traceflow evaluate the realised ruleset on the host or the intended policy? What does that tell you about trusting Traceflow during a publish?
  6. If NSX Manager becomes unavailable (network partition), you said data plane goes read-only. What happens to new traffic if a new VM is spun up? Is that VM reachable without Manager? When does Manager become critical again?
  7. You used vmkping to validate the TEP network. But vmkping is ICMP; NSX uses Geneve (UDP 6081). Can network policies block Geneve while allowing ICMP? How would you detect that scenario?

Extensions

NSX Manager Cluster Failover Testing

Extend the lab to test NSX Manager HA and control plane resilience. Deliberately shut down the primary Manager node (or simulate network partition). Monitor: (1) When does LCP detect Manager disconnection? (2) What is the failover time to secondary Manager? (3) Can you deploy new policies during failover? (4) What happens to existing rules—do they remain active? Create a 'NSX Manager High Availability' runbook documenting expected behavior and recovery steps.

MTU and Fragmentation Troubleshooting

Extend the lab to test packet size and MTU issues. Configure a large packet test: 'ping -s 1400 VM-B' from VM-A. Verify MTU is sufficient end-to-end (VM networks typically 1500; TEP VLANs at least 1600, 1700 recommended for Geneve; jumbo frames are 9000). Run a test with packet size > MTU to observe fragmentation handling. Document how NSX handles fragmentation (Geneve encapsulation adds roughly 50-60 bytes of headers; the TEP MTU must be at least 1600, 1700 recommended). This extension shows that the default Traceflow frame (128 bytes) does not exercise the overlay MTU: set Advanced Settings > Frame Size to an overlay-sized value, or test with vmkping -d -s, to catch MTU faults — a hidden cause of real-world failures.

NSX Monitoring and Alerting Design

Design a monitoring strategy based on your diagnostics. Define KPIs and alerts: (1) Alert if BFD sessions DOWN for > 30 seconds. (2) Alert if LCP Manager connectivity is DISCONNECTED. (3) Alert if DFW rule push latency > 30 seconds. (4) Alert if TEP vmkping latency > 10ms. Implement these alerts in VCF Operations or a custom monitoring tool. This extension demonstrates proactive incident prevention (monitoring) vs. reactive troubleshooting (response).

⚠ Known Pitfalls

Assuming Traceflow is 100% accurate for predicting real-world traffic. Traceflow simulates an exact packet and returns deterministic results. But real traffic may encounter conditions Traceflow doesn't test: large packet sizes, TCP options, asymmetric paths, transient network congestion, or edge-case firewall rules.
Resolution: Use Traceflow as a diagnostic starting point, not ground truth. Always validate with real traffic and packet capture (tcpdump). If Traceflow succeeds but real traffic fails, investigate: packet size (MTU), TCP options (TCP MSS), network asymmetry, and firewall edge cases.
Treating control plane and data plane failures as independent. Control plane issues (Manager down, LCP disconnected) cause data plane symptoms (rules not synced, traffic drops). A true diagnosis requires understanding their coupling.
Resolution: When diagnosing, always check both: (1) Control plane: Is Manager reachable? Is LCP synchronized? (2) Data plane: Are rules installed? Are tunnels UP? If control plane is down, data plane operates in read-only mode with cached state.
Over-relying on the host-side rule dump (vsipioctl getrules) to diagnose DFW issues. The rule may be realised but be incorrect (wrong CIDR, wrong direction, wrong protocol). nsxcli does not validate rule logic, only deployment status.
Resolution: Always verify rule logic in NSX Manager UI or via Traceflow. nsxcli shows what's deployed, not whether it's correct. Traceflow shows the actual effect of rules on traffic.
Assuming TEP network failures are rare. In large-scale NSX deployments, TEP network issues (VLAN misconfiguration, MTU problems, network switch failures) are common and cause widespread outages.
Resolution: Treat TEP network as critical infrastructure. Monitor vmkping and BFD status proactively. Test TEP network failover scenarios (e.g., primary VLAN down, secondary VLAN active). Design TEP network for high availability (dual-home, LACP, etc.).

References

Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.