Lab: Run Traceflow & Capture IPFIX

VCF 9.0Intermediatevcp-foundation⏱ 90 min

NSX 9.0.x (feature set inherited from the NSX 4.2 line; NSX 4.2 docs remain a valid technical reference) troubleshooting — Traceflow for path analysis, IPFIX for flow telemetry, port mirroring for deep inspection

Objectives

Use NSX Traceflow to trace the complete forwarding path between VMs
Interpret Traceflow results across overlay segments, DFW, and routing hops
Configure IPFIX flow export to an external collector
Analyze IPFIX flow records for traffic pattern visibility
Understand port mirroring options for deep packet inspection
Diagnose common connectivity issues using Traceflow observations

Prerequisites

VCF 9.0 with NSX Manager, T0/T1 routing topology deployed (Lab net-virt-02), DFW policies active (Lab net-virt-03), minimum 2 VMs on different segments for tracing

Prior labs: net-virt-01, net-virt-02, net-virt-03

Required skills:

TCP/IP packet flow fundamentals
Syslog/IPFIX concepts
Basic network troubleshooting

Lab Environment

VMs on different overlay segments connected via T1 gateway. DFW policies active. Optional: IPFIX collector VM (e.g., ntopng, Splunk with IPFIX add-on, or vRealize Log Insight / Aria Operations for Logs).

Tasks

Task 1 Traceflow Analysis and IPFIX Flow Monitoring

Traceflow is non-intrusive — it injects a tagged packet that is traced through the data plane without affecting production traffic. IPFIX exports sampled flow records (not full packets) to external collectors, providing traffic visibility with minimal overhead. In production, enable IPFIX on critical segments only to manage collector storage and processing load.

Master NSX's built-in troubleshooting tools — Traceflow for hop-by-hop path analysis and IPFIX for continuous flow telemetry — to diagnose connectivity issues, validate DFW rule enforcement, and gain visibility into east-west traffic patterns.

Step 1

Run basic Traceflow. NSX Manager → Plan & Troubleshoot → Traceflow → New Traceflow. Source: select test-web-01 (VIF/vNIC), Destination: select test-app-01 by IP (10.10.2.11). Packet Type=ICMP (default). Click Trace. The injected packet traverses the actual data path and reports observations at each hop.

Step 2

Interpret Traceflow results. The output shows a table of observations. Each row is a hop with: Component Type (e.g., DFW, Router, Switch), Component Name, Observation Type (Forwarded, Delivered, Dropped, Received). For cross-segment traffic, expect: (1) Source host DFW → Forwarded; (2) Source host T1 DR → Routed; (3) Transport tunnel (GENEVE) → Forwarded; (4) Dest host T1 DR → Received → Routed to segment; (5) Dest host DFW → Forwarded; (6) Dest vNIC → Delivered. If any hop shows 'Dropped', the Component Name identifies exactly where (e.g., 'DFW rule 1045' or 'T0-SR no route').

Step 3

Traceflow for blocked traffic. Run a Traceflow from test-web-01 to test-db-01:3306 (which should be blocked by DFW from Lab net-virt-03). The result should show: Observation='Dropped' at Component='DFW', with the specific rule ID that blocked the traffic. This is invaluable for troubleshooting — it tells you exactly which rule is causing a block, not just that traffic is blocked.

Step 4

Traceflow with custom packet headers. Create a new Traceflow with: Packet Type=TCP, Source Port=12345, Destination Port=443. This simulates a specific application flow. You can also set DSCP values to test QoS path behavior. For L2 Traceflow (same segment), select Traceflow Type='Unicast' with source and dest on the same segment — useful for verifying MAC learning and segment realization.

Step 5

Configure IPFIX flow export. NSX Manager → System → IPFIX → Firewall IPFIX tab → Add Collector Profile. Configure: Name='Lab-Collector', Collector IP=<collector-VM-IP>, Collector Port=4739 (default IPFIX), Protocol=TCP (reliable) or UDP (lightweight). Then: Add IPFIX DFW Profile → Name='DFW-IPFIX', Collector Profile='Lab-Collector', Observation Domain ID=1, Active Timeout=60s, Idle Timeout=300s. Apply to all DFW sections or specific policies.

Step 6

Configure Switch IPFIX (for overlay segment flow visibility). IPFIX → Switch IPFIX Profiles tab → Add Profile. Name='Switch-IPFIX', Collector Profile='Lab-Collector', Sampling Rate=1 (every packet for lab; use 512 or 1024 in production). Apply to specific segments: Edit segment → IPFIX Profile='Switch-IPFIX'. Switch IPFIX captures ALL flows traversing the segment, not just firewall events.

Step 7

Generate test traffic and verify IPFIX collection. From test-web-01, run: for i in $(seq 1 100); do curl -s -o /dev/null http://10.10.2.11; done. This generates 100 HTTP flows. On the IPFIX collector, verify flow records appear. Key fields: sourceIPv4Address, destinationIPv4Address, protocolIdentifier, sourceTransportPort, destinationTransportPort, flowStartMilliseconds, flowEndMilliseconds, octetDeltaCount, packetDeltaCount.

Step 8

Analyze flow patterns. On the collector (or NSX Manager if using VCF Operations for Networks), look for: (a) Top talkers by byte count — identifies high-bandwidth flows; (b) Unique source-destination pairs — maps actual communication patterns; (c) Denied flows — from DFW IPFIX, shows blocked attempts (potential lateral movement or misconfiguration); (d) Long-duration flows — identifies persistent connections (database connections, monitoring agents) vs. ephemeral traffic.

Step 9

Port mirroring for deep inspection. NSX Manager → Networking → Switching → Port Mirroring → Add Session. Types available: (a) LogicalPortMirrorSource — mirrors traffic from specific vNICs to a destination port (span VM); (b) LogicalPortMirrorDest — receives mirrored traffic; (c) RemoteMirrorSource — sends encapsulated (ERSPAN-like) mirrored traffic to a remote collector. Configure a session: Source=test-web-01 vNIC, Direction=Bidirectional, Dest=analyzer VM. Run tcpdump on the analyzer to see full packets.

Step 10

Traceflow API for automation. For bulk troubleshooting, use the NSX API: POST /api/v1/traceflow with JSON body: {"source": {"field": "LOGICALPORT", "value": "<port-id>"}, "destination": {"field": "IPADDRESS", "value": "10.10.2.11"}, "packet": {"resource_type": "FieldsPacketData", "frame_size": 128, "transport_type": "UNICAST"}}. Then GET /api/v1/traceflow/<id>/observations to retrieve results. This enables scripted path validation after infrastructure changes.

Validation Gate

Check: Traceflow and IPFIX operational with verified results

Expected: Traceflow shows complete hop-by-hop path for allowed traffic, identifies exact DFW rule for blocked traffic. IPFIX collector receiving flow records with correct 5-tuple data. Port mirroring captures full packets for deep inspection.

Common Errors

Traceflow shows 'Timeout' with no observations

Fix: The source VM's vNIC port may not be on an NSX-managed segment, or the VM is powered off. Verify: the source must be a VIF on an NSX segment. Also check: NSX Controller cluster health — Traceflow uses the control plane for packet injection.

IPFIX collector receives no data

Fix: Check: (1) Collector IP and port are correct and reachable from ESXi hosts, (2) no firewall blocking IPFIX port (4739 TCP/UDP), (3) IPFIX profile is applied to the correct segments/DFW policies. On ESXi, verify: esxcli network ipfix list shows the collector configured.

Traceflow shows 'Dropped' at DFW but rule should allow

Fix: Traceflow uses the exact packet headers you specify. If you set Dest Port=80 but the allow rule is for port 443, the trace will show a drop. Match the trace packet to the exact rule you want to validate. Also check rule order — a higher-priority deny may be matching first.

Port mirroring causes performance degradation

Fix: Mirroring doubles traffic on the source host. In production, use targeted mirroring (specific vNICs, not entire segments) and set a capture duration. For continuous monitoring, IPFIX is more efficient than port mirroring.

Final Validation

NSX troubleshooting toolkit validated — Traceflow, IPFIX, and port mirroring operational

✓ Traceflow allowed path → Shows complete path: DFW→DR→Tunnel→DR→DFW→Delivered for cross-segment traffic

✓ Traceflow blocked path → Shows Dropped observation with specific DFW rule ID

✓ IPFIX flow collection → Collector receiving flow records with 5-tuple, byte/packet counts, and timestamps

✓ Port mirroring → Analyzer VM captures full packet data from mirrored source port

Cleanup / Restore

• Delete port mirroring session (Networking → Switching → Port Mirroring → Delete)

• Remove IPFIX profiles from segments (Edit segment → IPFIX Profile=None)

• Delete IPFIX collector and DFW profiles (System → IPFIX → Delete)

• Close any open Traceflow sessions

Design Reflection (VCDX)

Troubleshooting tools are essential for Day 2 operations and VCDX defense questions about operational procedures. Traceflow is unique to NSX — no equivalent exists in physical networking. It eliminates guesswork by showing the exact forwarding decision at every hop. IPFIX provides the continuous visibility needed for security compliance (flow audit trails) and capacity planning. Design your monitoring architecture with dedicated collector infrastructure and retention policies.

Requirements

Hop-by-hop path visibility for overlay and routed traffic
Continuous flow telemetry for security and compliance audit
Deep packet inspection capability for incident response

Constraints

Traceflow is a point-in-time diagnostic — not continuous monitoring
IPFIX sampling rate impacts collector storage (1:1 ratio generates massive data)
Port mirroring doubles traffic on source host — cannot run continuously on production workloads

Assumptions

IPFIX collector infrastructure is sized for expected flow volume
Traceflow control plane path is healthy (NSX Manager → CCP → transport node)
Collector VMs are on a management segment not subject to DFW deny rules

Risks

IPFIX collector overload causes flow record loss — size for peak traffic
Over-reliance on Traceflow without understanding its limitations (doesn't test actual VM OS stack)
Port mirroring without access controls could expose sensitive traffic to unauthorized analyzer VMs

Self-Assessment Discussion Prompts

How would you design an IPFIX collection architecture for a 500-host VCF deployment?
What are the limitations of Traceflow for diagnosing intermittent connectivity issues?
How do you use Traceflow observations to validate DFW rule changes before production deployment?
Compare NSX Traceflow with physical network traceroute — what can each do that the other cannot?

Extensions

Build a Traceflow automation script that tests all critical application paths after every DFW policy change

Configure IPFIX with vRealize Log Insight / Aria Operations for Logs and build dashboards for top talkers and denied flows

Use Live Traffic Analysis (LTA) for real-time packet capture without port mirroring overhead

Integrate Traceflow results with a change management workflow to validate post-change connectivity

⚠ Known Pitfalls (from Community KB)

Assuming Traceflow tests the full application path — it tests NSX data plane only, not the guest OS network stack or application layer

Setting IPFIX sampling rate to 1:1 in production — generates enormous data volume; use 1:512 or 1:1024 for production workloads

Forgetting that Traceflow requires NSX Controller connectivity — if the CCP is down, Traceflow will not function

Running port mirroring to a VM on the same segment being mirrored — creates a loop; use a dedicated analyzer segment

References

NSX 4.2 Troubleshooting Guide — Traceflow: techdocs.broadcom.com
NSX 4.2 Administration Guide — IPFIX Configuration: techdocs.broadcom.com
KB 91785 — Traceflow Troubleshooting and Interpretation Guide

Mark complete