Lab: Run Traceflow & Capture IPFIX
Objectives
- Use NSX Traceflow to trace the complete forwarding path between VMs
- Interpret Traceflow results across overlay segments, DFW, and routing hops
- Configure IPFIX flow export to an external collector
- Analyze IPFIX flow records for traffic pattern visibility
- Understand port mirroring options for deep packet inspection
- Diagnose common connectivity issues using Traceflow observations
Prerequisites
VCF 9.0 with NSX Manager, T0/T1 routing topology deployed (Lab net-virt-02), DFW policies active (Lab net-virt-03), minimum 2 VMs on different segments for tracing
Prior labs: net-virt-01, net-virt-02, net-virt-03
Required skills:
- TCP/IP packet flow fundamentals
- Syslog/IPFIX concepts
- Basic network troubleshooting
Lab Environment
VMs on different overlay segments connected via T1 gateway. DFW policies active. Optional: IPFIX collector VM (e.g., ntopng, Splunk with IPFIX add-on, or vRealize Log Insight / Aria Operations for Logs).
Tasks
Task 1 Traceflow Analysis and IPFIX Flow Monitoring
Master NSX's built-in troubleshooting tools — Traceflow for hop-by-hop path analysis and IPFIX for continuous flow telemetry — to diagnose connectivity issues, validate DFW rule enforcement, and gain visibility into east-west traffic patterns.
Run basic Traceflow. NSX Manager → Plan & Troubleshoot → Traceflow → New Traceflow. Source: select test-web-01 (VIF/vNIC), Destination: select test-app-01 by IP (10.10.2.11). Packet Type=ICMP (default). Click Trace. The injected packet traverses the actual data path and reports observations at each hop.
Interpret Traceflow results. The output shows a table of observations. Each row is a hop with: Component Type (e.g., DFW, Router, Switch), Component Name, Observation Type (Forwarded, Delivered, Dropped, Received). For cross-segment traffic, expect: (1) Source host DFW → Forwarded; (2) Source host T1 DR → Routed; (3) Transport tunnel (GENEVE) → Forwarded; (4) Dest host T1 DR → Received → Routed to segment; (5) Dest host DFW → Forwarded; (6) Dest vNIC → Delivered. If any hop shows 'Dropped', the Component Name identifies exactly where (e.g., 'DFW rule 1045' or 'T0-SR no route').
Traceflow for blocked traffic. Run a Traceflow from test-web-01 to test-db-01:3306 (which should be blocked by DFW from Lab net-virt-03). The result should show: Observation='Dropped' at Component='DFW', with the specific rule ID that blocked the traffic. This is invaluable for troubleshooting — it tells you exactly which rule is causing a block, not just that traffic is blocked.
Traceflow with custom packet headers. Create a new Traceflow with: Packet Type=TCP, Source Port=12345, Destination Port=443. This simulates a specific application flow. You can also set DSCP values to test QoS path behavior. For L2 Traceflow (same segment), select Traceflow Type='Unicast' with source and dest on the same segment — useful for verifying MAC learning and segment realization.
Configure IPFIX flow export. NSX Manager → Plan & Troubleshoot → IPFIX → Collectors tab → Add New Collector → IPFIX Firewall. Configure: Name='Lab-Collector', IP address=<collector-VM-IP>, Port=4739 (the IANA IPFIX port; up to four collectors can be listed). Then: Firewall IPFIX Profiles tab → Add Firewall IPFIX Profile → Name='DFW-IPFIX', Active Flow Export Timeout=1 minute (default), Observation Domain ID=0 (default), Collector Configuration='Lab-Collector', Applied To=<group containing the lab VMs>, Priority=0 (VCF 9.0.1 docs p.2492-2493).
Configure Switch IPFIX (for overlay segment flow visibility). Plan & Troubleshoot → IPFIX → Switch IPFIX Profiles tab → Add Switch IPFIX Profile. Name='Switch-IPFIX', Collector Configuration='Lab-Collector', Packet Sampling Probability=100% for this lab only (the default is 0.1%; keep it low in production), Applied To=Segment seg-web and seg-app (the profile is bound from its own Applied To setting: Segment, Segment Port or Groups). Switch IPFIX samples all traffic on the bound segments or ports, not just firewall events; exported packet counts are extrapolated from the sample (VCF 9.0.1 docs p.2494-2495).
Generate test traffic and verify IPFIX collection. From test-web-01, run: for i in $(seq 1 100); do curl -s -o /dev/null http://10.10.2.11; done. This generates 100 HTTP flows. On the IPFIX collector, verify flow records appear. Key fields: sourceIPv4Address, destinationIPv4Address, protocolIdentifier, sourceTransportPort, destinationTransportPort, flowStartMilliseconds, flowEndMilliseconds, octetDeltaCount, packetDeltaCount.
Analyze flow patterns. On the collector (or NSX Manager if using VCF Operations for Networks), look for: (a) Top talkers by byte count — identifies high-bandwidth flows; (b) Unique source-destination pairs — maps actual communication patterns; (c) Denied flows — from DFW IPFIX, shows blocked attempts (potential lateral movement or misconfiguration); (d) Long-duration flows — identifies persistent connections (database connections, monitoring agents) vs. ephemeral traffic.
Port mirroring for deep inspection. NSX Manager → Plan & Troubleshoot → Port Mirroring → Add. Session types: Local Span; Remote Span; Remote L3 Span (GRE, ERSPAN TWO or ERSPAN THREE encapsulation to a remote collector); and Logical Span (overlay segments only). Configure a Logical Span session: Source=test-web-01 vNIC, Direction=Bidirectional, Destination=analyzer VM port. Run tcpdump on the analyzer to see full packets. Port mirroring is not recommended for long-running monitoring because it affects performance (VCF 9.0.1 docs p.2495-2496).
Traceflow API for automation. For bulk troubleshooting, use the NSX Policy API — the Manager API paths /api/v1/traceflows are listed under Deleted Operations in the NSX Manager API changelog of VCF 9.0:
PUT /policy/api/v1/infra/traceflows/<traceflow-id>
GET /policy/api/v1/infra/traceflows/<traceflow-id>/observationsThe PUT body names the source segment port and the packet to inject (for example destination 10.10.2.11, frame size 128, unicast); take the exact schema from the NSX API Guide. The GET call retrieves the hop-by-hop observations. This enables scripted path validation after infrastructure changes.
Validation Gate
Check: Traceflow and IPFIX operational with verified results
Expected: Traceflow shows complete hop-by-hop path for allowed traffic, identifies exact DFW rule for blocked traffic. IPFIX collector receiving flow records with correct 5-tuple data. Port mirroring captures full packets for deep inspection.
Common Errors
Final Validation
NSX troubleshooting toolkit validated — Traceflow, IPFIX, and port mirroring operational
✓ Traceflow allowed path → Shows complete path: DFW→DR→Tunnel→DR→DFW→Delivered for cross-segment traffic
✓ Traceflow blocked path → Shows Dropped observation with specific DFW rule ID
✓ IPFIX flow collection → Collector receiving flow records with 5-tuple, byte/packet counts, and timestamps
✓ Port mirroring → Analyzer VM captures full packet data from mirrored source port
Cleanup / Restore
• Delete port mirroring session (Plan & Troubleshoot → Port Mirroring → Delete)
• Remove the segments from the Switch IPFIX profile's Applied To, then delete the profile (Plan & Troubleshoot → IPFIX → Switch IPFIX Profiles)
• Delete the Firewall IPFIX profile, then the collector (Plan & Troubleshoot → IPFIX → Firewall IPFIX Profiles / Collectors → Delete)
• Close any open Traceflow sessions
Design Reflection (VCDX)
Troubleshooting tools are essential for Day 2 operations and VCDX defense questions about operational procedures. Traceflow is unique to NSX — no equivalent exists in physical networking. It eliminates guesswork by showing the exact forwarding decision at every hop. IPFIX provides the continuous visibility needed for security compliance (flow audit trails) and capacity planning. Design your monitoring architecture with dedicated collector infrastructure and retention policies.
Requirements
- Hop-by-hop path visibility for overlay and routed traffic
- Continuous flow telemetry for security and compliance audit
- Deep packet inspection capability for incident response
Constraints
- Traceflow is a point-in-time diagnostic — not continuous monitoring
- IPFIX sampling probability impacts collector storage (100% sampling generates massive data)
- Port mirroring doubles traffic on source host — cannot run continuously on production workloads
Assumptions
- IPFIX collector infrastructure is sized for expected flow volume
- Traceflow control plane path is healthy (NSX Manager → CCP → transport node)
- Collector VMs are on a management segment not subject to DFW deny rules
Risks
- IPFIX collector overload causes flow record loss — size for peak traffic
- Over-reliance on Traceflow without understanding its limitations (doesn't test actual VM OS stack)
- Port mirroring without access controls could expose sensitive traffic to unauthorized analyzer VMs
Self-Assessment Discussion Prompts
- How would you design an IPFIX collection architecture for a 500-host VCF deployment?
- What are the limitations of Traceflow for diagnosing intermittent connectivity issues?
- How do you use Traceflow observations to validate DFW rule changes before production deployment?
- Compare NSX Traceflow with physical network traceroute — what can each do that the other cannot?
Extensions
Build a Traceflow automation script that tests all critical application paths after every DFW policy change
Configure IPFIX with vRealize Log Insight / Aria Operations for Logs and build dashboards for top talkers and denied flows
Use Live Traffic Analysis (LTA) for real-time packet capture without port mirroring overhead
Integrate Traceflow results with a change management workflow to validate post-change connectivity
⚠ Known Pitfalls
References
- Perform a Traceflow (VCF 9.0 NSX docs): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/advanced-network-management/administration-guide/network-monitoring/perform-a-traceflow.html
- Add an IPFIX Collector (VCF 9.0 NSX docs): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/advanced-network-management/administration-guide/network-monitoring/add-a-new-collector-profile.html
- Add a Firewall IPFIX Profile (VCF 9.0 NSX docs): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/advanced-network-management/administration-guide/network-monitoring/add-a-firewall-ipfix-profile.html
- Add a Switch IPFIX Profile (VCF 9.0 NSX docs): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/advanced-network-management/administration-guide/network-monitoring/add-a-switch-ipfix-profile.html
- Add a Port Mirroring Session (VCF 9.0 NSX docs): https://techdocs.broadcom.com/us/en/vmware-cis/vcf/vcf-9-0-and-later/9-0/advanced-network-management/administration-guide/network-monitoring/add-a-port-mirroring-session.html