Academy/VMware vDefend Security for VCF 5.x Administrator (6V0-21.25)/Lab: IDS/IPS Deployment, False-Positive Tuning & Prevention Mode
This lab targets VCF 9.0

Lab: IDS/IPS Deployment, False-Positive Tuning & Prevention Mode

VCF 9.0Advancedspecialistvcdx⏱ 120 min

NSX 4.2 Distributed IDS/IPS — signature profiles, detection vs prevention modes, false-positive tuning, per-rule IDS/IPS binding, CVSS-based severity filtering, signature exclusions

Objectives

  • Enable NSX Distributed IDS/IPS and configure signature bundle updates from Broadcom Threat Intelligence Cloud
  • Create signature profiles with severity-based filtering (Critical/High/Medium/Low) for different security zones
  • Deploy IDS in detection-only mode and analyze alert quality before switching to prevention
  • Identify and tune false positives using signature exclusions and per-VM/per-flow exceptions
  • Switch to IPS prevention mode with validated confidence and verify active blocking
  • Analyze IDS/IPS events with CVSS scoring, MITRE ATT&CK mapping, and affected VM context

Prerequisites

VCF 9.0 with NSX 9.0.x and vDefend license enabled. IDS/IPS feature requires NSX Advanced or vDefend add-on license. 3+ VMs with network connectivity for attack simulation. NSX Manager with outbound HTTPS access to Broadcom signature update service.

Prior labs: vdefend-01

Required skills:

  • NSX DFW rule creation (from Lab vdefend-01)
  • Basic security concepts (IDS vs IPS, signatures, false positives)
  • HTTP request structure (for simulating web attacks)
  • CVSS scoring fundamentals

Lab Environment

VCF 9.0 with NSX overlay networking. 3 VMs from Lab vdefend-01 (web-01, app-01, db-01). Web-01 running a simple HTTP service (e.g., nginx or python http.server on port 8080). App-01 running an application that responds to HTTP requests.

Tasks

Task 1 IDS/IPS Deployment, False-Positive Tuning & Prevention Mode

IDS/IPS deployment is a phased process, not a single action. The detection phase is where you build confidence in signature accuracy for YOUR specific traffic patterns. Every environment generates different false positives depending on application behavior, API patterns, and internal scanning tools. Skipping the tuning phase and going straight to IPS prevention mode is the most common cause of self-inflicted outages in security deployments.

Deploy the full IDS-to-IPS lifecycle: enable detection, generate and analyze alerts, identify and suppress false positives, then switch to prevention mode with confidence that legitimate traffic won't be blocked. This mirrors the real-world deployment pattern — organizations that skip the detection phase and go straight to prevention invariably break production applications.

Step 1
Enable IDS/IPS globally. NSX Manager → Security → IDS/IPS & Malware Prevention → Settings → toggle IDS/IPS to 'Enabled'. For Distributed IDS/IPS (host-based), ensure 'Distributed IDS/IPS' is enabled. For Gateway IDS/IPS (Edge-based), enable on the T1 gateway if needed for north-south inspection. Click 'Update Signatures' → NSX Manager downloads the latest signature bundle from Broadcom Threat Intelligence Cloud. Wait for status: 'Signatures up to date' (shows timestamp of latest update). Verify signature count: typically 15,000-20,000+ signatures in the full bundle. Configure auto-update schedule: Settings → Auto Update → enable daily signature updates to ensure new threat coverage.
Step 2
Create IDS/IPS Signature Profiles for different security zones. NSX Manager → Security → IDS/IPS & Malware Prevention → Profiles → Add Profile. (a) Profile 'IDS-DMZ-Strict': Severity = Critical + High + Medium. This catches most threats including medium-confidence signatures — appropriate for internet-facing DMZ where attack surface is highest. (b) Profile 'IDS-Internal-Balanced': Severity = Critical + High only. Excludes Medium/Low to reduce noise on internal segments where attack probability is lower. (c) Profile 'IDS-DB-Critical-Only': Severity = Critical only. Minimal alerting for database tier — focused on high-confidence, severe threats to avoid alert fatigue in low-attack-surface zones. Each profile filters the full signature set by CVSS severity — Critical (CVSS 9.0-10.0), High (7.0-8.9), Medium (4.0-6.9), Low (0.1-3.9).
Step 3
Bind IDS profiles to DFW rules. NSX Manager → Security → Distributed Firewall → edit the Web-to-App allow rule (from Lab vdefend-01). In the rule row, click the IDS column → Enable IDS → select profile 'IDS-DMZ-Strict' → Mode: 'Detect Only'. This binds IDS inspection to traffic matching this specific DFW rule — only web→app traffic on TCP 8080 is inspected, not all traffic on the host. Repeat for App-to-DB rule: bind 'IDS-DB-Critical-Only' profile in Detect Only mode. Publish. This per-rule binding is a key NSX advantage — IDS inspection only runs on traffic that matches the DFW rule, reducing CPU overhead compared to inspecting all traffic.
Step 4

Generate test traffic including simulated attacks. From web-01, send a mix of legitimate and malicious-pattern requests: (a) Legitimate traffic (should NOT trigger alerts): curl http://app-01:8080/api/health — health check endpoint. curl http://app-01:8080/api/users?page=1&limit=10 — normal API query. for i in $(seq 1 10); do curl http://app-01:8080/api/data; done — repeated normal requests. (b) Simulated attack traffic (SHOULD trigger alerts): curl 'http://app-01:8080/search?q=1%27%20OR%201%3D1--' — SQL injection pattern. curl 'http://app-01:8080/../../etc/passwd' — directory traversal. curl -A 'nikto/2.1.6' http://app-01:8080/ — scanner user-agent. curl 'http://app-01:8080/<script>alert(1)</script>' — XSS attempt. Wait 2-3 minutes for IDS event processing and correlation.

Step 5
Analyze IDS events and identify false positives. NSX Manager → Security → IDS/IPS & Malware Prevention → Events. Review the events dashboard: (a) Sort by severity (Critical/High first). (b) For each event, examine: Signature ID (e.g., sid:2024897), Signature Name (e.g., 'ET WEB_SERVER SQL Injection Attempt'), CVSS Score, Affected VMs (source + destination), Timestamp, Classification (e.g., Web Application Attack). (c) Expected TRUE positives: SQL injection, directory traversal, XSS, scanner detection. (d) Check for FALSE positives: did any legitimate requests (health check, normal API) trigger alerts? Common false positives: API requests with special characters in query strings, health monitoring tools that match scanner signatures, internal vulnerability scanning triggering IDS. Document: for each false positive, note the Signature ID, the traffic that triggered it, and why it's benign.
Step 6
Tune false positives using signature exclusions. For each identified false positive, choose a tuning approach: (a) Global signature disable: NSX Manager → Profiles → edit 'IDS-DMZ-Strict' → find signature by ID → toggle to 'Disabled'. Use when the signature is fundamentally irrelevant to your environment (e.g., a Windows-specific exploit on a Linux-only infrastructure). (b) Per-flow exclusion: Security → IDS/IPS → Exclusions → Add. Specify: Signature ID + Source VM + Destination VM. This suppresses the signature only for this specific traffic flow — more surgical than global disable. Use when the signature is valid generally but false-positive for one specific application. (c) Profile adjustment: switch from 'IDS-DMZ-Strict' (Crit+High+Med) to a custom profile that excludes the noisy category. After tuning, re-run the SAME test traffic from Step 4 and verify: (1) attack traffic still generates alerts, (2) legitimate traffic no longer triggers false positives.
Step 7

Establish detection-mode baseline period. Before switching to IPS prevention, run in detection mode for a baseline period to build confidence: (a) Lab environment: 24-48 hours of normal application operation in IDS mode. (b) Production recommendation: 7-14 days minimum, covering all business cycle patterns (weekday/weekend, month-end processing, batch jobs). During baseline: monitor false positive rate daily. Target: zero false positives on legitimate traffic for 3+ consecutive days before switching to prevention. Document the false-positive tuning log: date, signature ID, action taken (disabled/excluded), reason.

Step 8
Switch to IPS Prevention Mode. After achieving zero false positives during baseline: Edit the DFW rules → change IDS/IPS mode from 'Detect Only' to 'Detect & Prevent'. Publish. IMPORTANT: switch one rule at a time, starting with the least critical traffic flow. Monitor for 30 minutes after each switch before proceeding to the next rule. After switching Web-to-App rule to IPS: (a) Re-run SQL injection test: curl 'http://app-01:8080/search?q=1%27%20OR%201%3D1--' → connection should be RESET or timed out (IPS blocks the packet). (b) Verify legitimate traffic: curl http://app-01:8080/api/health → should succeed (not matching any IDS signature). (c) Check events: NSX Manager → Events → filter by Action='Dropped' → confirm attack traffic shows as blocked.
Step 9

Verify IPS enforcement at the host level. SSH to the ESXi host running web-01. Run: nsxcli -c 'get ids-ips statistics' — shows packet inspection counts, alerts generated, and packets dropped by IPS. Check: dropped count should be > 0 after running attack traffic with IPS enabled. Run: nsxcli -c 'get ids-ips profile' — shows which IDS/IPS profiles are applied to which vNICs. Verify the IPS profile binding matches your DFW rule configuration. Also check VCF Operations (if configured) for IDS/IPS metrics: alerts per signature, dropped traffic volume, false positive trend over time.

Step 10

Document IDS/IPS operational procedures. Create a runbook for ongoing IDS/IPS management: (a) Signature updates: verify daily auto-update is functioning, check for failed updates weekly. (b) False positive review: review new alerts weekly, tune as needed, document all exclusions. (c) Mode switching: document the process for switching new rules from IDS to IPS (detection period, baseline criteria, change management). (d) Incident response: when a true positive is detected in IPS mode, document the investigation workflow — event review, VM isolation, forensic data collection. (e) Signature profile review: quarterly review of profile configurations to ensure severity levels match current threat landscape.

Validation Gate

Check: IDS/IPS deployed with tuned profiles and prevention mode active

Expected: IDS/IPS enabled with current signatures, 3 severity-based profiles created and bound to DFW rules, false positives identified and tuned via signature exclusions, IPS prevention mode actively blocking attack traffic while allowing legitimate, host-level IPS statistics confirming enforcement, operational runbook documented

Common Errors

IDS/IPS toggle is grayed out — cannot enable
Fix: IDS/IPS requires the vDefend or NSX Advanced license. Check: NSX Manager → System → Licenses → verify 'Threat Prevention' or 'Advanced' license is applied. Also verify: the NSX Manager nodes must have outbound HTTPS connectivity to the Broadcom signature update service — air-gapped environments require offline signature bundle import.
Signature update fails with 'Connection Error'
Fix: NSX Manager needs outbound HTTPS (443) connectivity to the Broadcom Threat Intelligence Cloud. Check: (1) DNS resolution from NSX Manager to the update endpoint, (2) firewall rules allowing HTTPS outbound from NSX Manager management IP, (3) proxy configuration if the environment uses an HTTP proxy. For air-gapped environments: download the signature bundle manually and import via NSX Manager → IDS/IPS → Settings → Manual Upload.
IPS mode blocks legitimate traffic — production impact
Fix: Immediate response: switch the affected DFW rule back to 'Detect Only' mode and publish — this restores traffic flow within seconds. Then: identify the signature that caused the block (Events → filter by Action='Dropped', timestamp matching the incident), create a per-flow exclusion for that signature + affected VMs, re-run test traffic to verify the exclusion works, then switch back to IPS. Root cause: insufficient detection-mode baseline period. Extend the baseline for this traffic flow.
IDS events show high volume of Medium/Low severity alerts causing alert fatigue
Fix: Medium and Low severity signatures have higher false-positive rates. For internal segments: use profiles with only Critical+High severity. For DMZ: start with Critical+High, add Medium only after tuning Medium-severity false positives. Never enable Low severity in production — the signal-to-noise ratio is too low for operational use. Use Low severity only during threat hunting exercises.
IDS/IPS not inspecting traffic despite being enabled on the DFW rule
Fix: Verify: (1) The DFW rule action is 'Allow' — IDS/IPS only inspects traffic that the DFW rule allows. A 'Drop' rule never reaches IDS inspection. (2) The IDS/IPS mode column shows 'Detect Only' or 'Detect & Prevent', not blank. (3) The rule is published (not in draft state). (4) The signature profile assigned to the rule is not empty — verify the profile has signatures enabled.

Final Validation

IDS/IPS lifecycle complete from detection through tuning to prevention

✓ IDS/IPS enabled → Feature enabled with current signature bundle (15,000+ signatures)

✓ Signature profiles → 3 profiles created with severity-appropriate filtering per zone

✓ Detection mode alerts → True positive alerts for simulated attacks, no false positives after tuning

✓ False-positive tuning → Exclusions documented with signature IDs, affected flows, and tuning rationale

✓ IPS prevention → Attack traffic actively blocked, legitimate traffic unaffected

✓ Host-level verification → nsxcli IDS/IPS statistics showing inspection counts and drops

Cleanup / Restore

• Switch DFW rules back from IPS to IDS mode (or disable IDS/IPS on rules)

• Delete IDS/IPS signature profiles

• Remove signature exclusions

• Optionally disable IDS/IPS globally if not needed for subsequent labs

Design Reflection (VCDX)

IDS/IPS deployment strategy demonstrates operational maturity in a VCDX design. Key defense points: (1) Phased deployment (detect → tune → prevent) shows you understand that security controls must be validated before enforcement — skipping this phase causes outages. (2) Per-rule IDS binding (not global) shows efficiency awareness — inspecting only DFW-allowed traffic reduces CPU overhead by 60-80% vs inspecting all traffic. (3) Severity-based profiles per zone show defense-in-depth thinking — DMZ needs aggressive detection, internal segments need balanced coverage. VCDX panelists will ask: 'How do you handle a new signature update that introduces false positives in IPS mode?'

Requirements

  • Detect and prevent known attack patterns (SQL injection, XSS, directory traversal) at the vNIC level
  • Zero false-positive rate on legitimate traffic before enabling prevention mode
  • Automated daily signature updates from Broadcom Threat Intelligence Cloud

Constraints

  • IDS/IPS requires vDefend or NSX Advanced license — additional cost
  • IDS inspection adds CPU overhead on ESXi hosts (~5-10% for inspected traffic flows)
  • Air-gapped environments cannot auto-update signatures — require manual bundle import process

Assumptions

  • Application traffic patterns are stable enough for a 7-14 day detection baseline to be representative
  • Broadcom signature updates are vetted and don't introduce mass false positives (historically reliable)
  • SOC team has capacity to review IDS alerts during the detection phase

Risks

  • New signature update introduces false positive in IPS mode — blocks legitimate production traffic. Mitigation: staged signature rollout, test in detection mode for 24h before auto-applying to prevention rules.
  • IDS alert fatigue — SOC team ignores alerts due to volume. Mitigation: severity-based profiles, false-positive tuning, alert routing with priority filtering.
  • Zero-day attacks bypass signature-based detection. Mitigation: layer with NDR behavioral analytics (Lab vdefend-03) for anomaly-based detection.

Self-Assessment Discussion Prompts

  1. How do you handle IDS/IPS in an environment with 500+ applications — is per-application tuning feasible?
  2. When should you use Distributed IDS/IPS (host-based) vs Gateway IDS/IPS (Edge-based)?
  3. How do you measure the CPU overhead of IDS/IPS inspection and when does it become a performance concern?
  4. What is the appropriate detection-mode baseline period for a healthcare environment with monthly billing cycles?

Extensions

Create a custom IDS signature for a specific application vulnerability unique to your environment

Automate false-positive tuning reports using the NSX API: GET /policy/api/v1/infra/settings/firewall/security/intrusion-services/ids-events

Design an IDS/IPS deployment plan for a 500-VM environment with 5 security zones and zone-specific profiles

Integrate IDS/IPS alerts with SIEM (Splunk/QRadar) via syslog forwarding and create correlation rules

⚠ Known Pitfalls (from Community KB)

Deploying IPS prevention mode without a detection-mode baseline period — the #1 cause of self-inflicted outages. Always run detection mode for 7-14 days minimum before switching to prevention.
Enabling all severity levels (including Low) on internal segments — Low-severity signatures generate massive alert volume with poor signal-to-noise ratio. Use Low only for DMZ or during active threat hunting.
Binding IDS/IPS globally instead of per-DFW-rule — global IDS inspects ALL traffic including management plane, vMotion, and vSAN. This wastes CPU and generates irrelevant alerts. Bind IDS only to workload DFW rules.
Not documenting signature exclusions — when you disable a signature for false-positive tuning, document WHY. Future signature reviews need this context to determine if the exclusion is still valid or if the application changed.
Treating IDS/IPS as set-and-forget — signatures need regular review: new application deployments may trigger new false positives, retired applications may leave unnecessary exclusions, new threat patterns require profile updates

References

  • NSX 4.2 IDS/IPS Administration Guide: techdocs.broadcom.com
  • vDefend Distributed IDS/IPS Design Guide: techdocs.broadcom.com
  • NSX IDS/IPS Signature Management: techdocs.broadcom.com
  • KB 88205 — NSX IDS/IPS Troubleshooting: https://knowledge.broadcom.com/external/article?legacyId=88205
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.