Academy/VMware vDefend Security for VCF 5.x Administrator (6V0-21.25)/Lab: IDS/IPS Deployment, False-Positive Tuning & Prevention Mode
This lab targets VCF 9.0

Lab: IDS/IPS Deployment, False-Positive Tuning & Prevention Mode

VCF 9.0Advancedadd-on exams (6V0)vcdx⏱ 120 min

NSX 9.0 Distributed IDS/IPS — signature profiles, detection vs prevention modes, false-positive tuning, Distributed IDS/IPS rules (separate from DFW rules), Intrusion Severity filtering (separate from the CVSS filter), signature exclusions. Exam note: 6V0-21.25 is written for VCF 5.x (NSX 4.2); IDS/IPS concepts (profiles, detect vs prevent, signature exclusions) are the same, but UI paths may differ from the VCF 9.0 UI used here.

Objectives

  • Enable NSX Distributed IDS/IPS and configure signature bundle updates from Broadcom Threat Intelligence Cloud
  • Create signature profiles with Intrusion Severity filtering (Critical/High/Medium/Low/Suspicious), optionally narrowed by CVSS, attack type and target, for different security zones
  • Deploy IDS in detection-only mode and analyze alert quality before switching to prevention
  • Identify and tune false positives using signature exclusions in profiles and flow-scoped IDS/IPS rules (tuned profile or Exempt action)
  • Switch to IPS prevention mode with validated confidence and verify active blocking
  • Analyze IDS/IPS events with CVSS scoring, MITRE ATT&CK mapping, and affected VM context

Prerequisites

VCF 9.0 with NSX 9.0.x and vDefend license enabled. IDS/IPS feature requires NSX Advanced or vDefend add-on license. 3+ VMs with network connectivity for attack simulation. NSX Manager with outbound HTTPS access to Broadcom signature update service.

Prior labs: vdefend-01

Required skills:

  • NSX DFW rule creation (from Lab vdefend-01)
  • Basic security concepts (IDS vs IPS, signatures, false positives)
  • HTTP request structure (for simulating web attacks)
  • CVSS scoring fundamentals

Lab Environment

VCF 9.0 with NSX overlay networking. 3 VMs from Lab vdefend-01 (web-01, app-01, db-01). Web-01 running a simple HTTP service (e.g., nginx or python http.server on port 8080). App-01 running an application that responds to HTTP requests.

Tasks

Task 1 IDS/IPS Deployment, False-Positive Tuning & Prevention Mode

IDS/IPS deployment is a phased process, not a single action. The detection phase is where you build confidence in signature accuracy for YOUR specific traffic patterns. Every environment generates different false positives depending on application behavior, API patterns, and internal scanning tools. Skipping the tuning phase and going straight to IPS prevention mode is the most common cause of self-inflicted outages in security deployments.

Deploy the full IDS-to-IPS lifecycle: enable detection, generate and analyze alerts, identify and suppress false positives, then switch to prevention mode with confidence that legitimate traffic won't be blocked. This mirrors the real-world deployment pattern — organizations that skip the detection phase and go straight to prevention invariably break production applications.

Step 1

Enable IDS/IPS globally. NSX Manager → Security → IDS/IPS & Malware Prevention → Settings → toggle IDS/IPS to 'Enabled'. For Distributed IDS/IPS (host-based), ensure 'Distributed IDS/IPS' is enabled. For Gateway IDS/IPS (Edge-based), enable on the T1 gateway if needed for north-south inspection. Click 'Update Signatures' → NSX Manager downloads the latest signature bundle from Broadcom Threat Intelligence Cloud. Wait for status: 'Signatures up to date' (shows timestamp of latest update). Verify signature count: typically 15,000-20,000+ signatures in the full bundle. Configure auto-update schedule: Settings → Auto Update → enable daily signature updates to ensure new threat coverage.

Step 2

Create IDS/IPS Signature Profiles for different security zones. NSX Manager → Security → IDS/IPS & Malware Prevention → Profiles → Add Profile. (a) Profile 'IDS-DMZ-Strict': Severity = Critical + High + Medium. This catches most threats including medium-confidence signatures — appropriate for internet-facing DMZ where attack surface is highest. (b) Profile 'IDS-Internal-Balanced': Severity = Critical + High only. Excludes Medium/Low to reduce noise on internal segments where attack probability is lower. (c) Profile 'IDS-DB-Critical-Only': Severity = Critical only. Minimal alerting for database tier — focused on high-confidence, severe threats to avoid alert fatigue in low-attack-surface zones. Each profile selects signatures by Intrusion Severity (Critical, High, Medium, Low, Suspicious) and can be narrowed further under Additional Options by Attack Types, CVSS score range, CVEs and Attack Targets. Intrusion Severity comes from the signature's threat classification and is a separate filter from CVSS.

Step 3

Create Distributed IDS/IPS rules. Distributed IDS/IPS has its own rule table, separate from the DFW: NSX Manager → Security → IDS/IPS & Malware Prevention → Distributed Rules → Add Policy 'Lab-IDS' → Add Rule 'Web-to-App-IDS': Sources=SG-Web-Tier, Destinations=SG-App-Tier, Services=TCP 8080, Security Profiles='IDS-DMZ-Strict', Applied To=SG-App-Tier, Mode='Detect Only'. Add 'App-to-DB-IDS': Sources=SG-App-Tier, Destinations=SG-DB-Tier, Services=TCP 5432, Security Profiles='IDS-DB-Critical-Only', Applied To=SG-DB-Tier, Mode='Detect Only'. Publish. The DFW must be enabled and IDS/IPS only sees traffic the DFW allows (the Web-to-App and App-to-DB allow rules from Lab vdefend-01); scoping the IDS rules to those flows keeps inspection overhead down.

Step 4

Generate test traffic including simulated attacks. From web-01, send a mix of legitimate and malicious-pattern requests: (a) Legitimate traffic (should NOT trigger alerts): curl http://app-01:8080/api/health — health check endpoint. curl http://app-01:8080/api/users?page=1&limit=10 — normal API query. for i in $(seq 1 10); do curl http://app-01:8080/api/data; done — repeated normal requests. (b) Simulated attack traffic (SHOULD trigger alerts): curl 'http://app-01:8080/search?q=1%27%20OR%201%3D1--' — SQL injection pattern. curl 'http://app-01:8080/../../etc/passwd' — directory traversal. curl -A 'nikto/2.1.6' http://app-01:8080/ — scanner user-agent. curl 'http://app-01:8080/<script>alert(1)</script>' — XSS attempt. Wait 2-3 minutes for IDS event processing and correlation.

Step 5

Analyze IDS events and identify false positives. NSX Manager → Security → IDS/IPS & Malware Prevention → Events. Review the events dashboard: (a) Sort by severity (Critical/High first). (b) For each event, examine: Signature ID (e.g., sid:2024897), Signature Name (e.g., 'ET WEB_SERVER SQL Injection Attempt'), CVSS Score, Affected VMs (source + destination), Timestamp, Classification (e.g., Web Application Attack). (c) Expected TRUE positives: SQL injection, directory traversal, XSS, scanner detection. (d) Check for FALSE positives: did any legitimate requests (health check, normal API) trigger alerts? Common false positives: API requests with special characters in query strings, health monitoring tools that match scanner signatures, internal vulnerability scanning triggering IDS. Document: for each false positive, note the Signature ID, the traffic that triggered it, and why it's benign.

Step 6

Tune false positives using signature exclusions. For each identified false positive, choose a tuning approach: (a) Global signature disable: NSX Manager → Profiles → edit 'IDS-DMZ-Strict' → find signature by ID → toggle to 'Disabled'. Use when the signature is fundamentally irrelevant to your environment (e.g., a Windows-specific exploit on a Linux-only infrastructure). (b) Flow-scoped tuning: create a dedicated Distributed IDS/IPS rule for the affected source/destination/service that uses a copy of the profile with the noisy signature excluded (Manage signatures for this profile → Exclude), or an Exempt rule if that flow should not be inspected at all. Place it above the broader rule. This is more surgical than excluding the signature globally. Use when the signature is valid generally but false-positive for one specific application. (c) Profile adjustment: switch from 'IDS-DMZ-Strict' (Crit+High+Med) to a custom profile that excludes the noisy category. After tuning, re-run the SAME test traffic from Step 4 and verify: (1) attack traffic still generates alerts, (2) legitimate traffic no longer triggers false positives.

Step 7

Establish detection-mode baseline period. Before switching to IPS prevention, run in detection mode for a baseline period to build confidence: (a) Lab environment: 24-48 hours of normal application operation in IDS mode. (b) Production recommendation: 7-14 days minimum, covering all business cycle patterns (weekday/weekend, month-end processing, batch jobs). During baseline: monitor false positive rate daily. Target: zero false positives on legitimate traffic for 3+ consecutive days before switching to prevention. Document the false-positive tuning log: date, signature ID, action taken (disabled/excluded), reason.

Step 8

Switch to IPS Prevention Mode. After achieving zero false positives during baseline: Security → IDS/IPS & Malware Prevention → Distributed Rules → edit the IDS rule → Mode from 'Detect Only' to 'Detect & Prevent'. Publish. IMPORTANT: switch one rule at a time, starting with the least critical traffic flow. Monitor for 30 minutes after each switch before proceeding to the next rule. After switching the Web-to-App-IDS rule to Detect & Prevent: (a) Re-run SQL injection test: curl 'http://app-01:8080/search?q=1%27%20OR%201%3D1--' → connection should be RESET or timed out (IPS blocks the packet). (b) Verify legitimate traffic: curl http://app-01:8080/api/health → should succeed (not matching any IDS signature). (c) Check events: NSX Manager → Events → filter by Action='Dropped' → confirm attack traffic shows as blocked.

Step 9

Verify IPS enforcement at the host level. SSH to the ESXi host running web-01. Run: nsxcli -c 'get ids status' — confirms IDS/IPS is enabled and licensed on the host. Run: nsxcli -c 'get ids profiles' — lists the IDS/IPS profile IDs pushed to this host (per host, not per vNIC). Run: nsxcli -c 'get ids events stats' and nsxcli -c 'get ids engine global stats' — event counters and engine statistics; the drop counters should rise after the attack traffic in Detect & Prevent mode. Verify the profile IDs match the profiles referenced by your Distributed IDS/IPS rules. Also check VCF Operations (if configured) for IDS/IPS metrics: alerts per signature, dropped traffic volume, false positive trend over time.

Step 10

Document IDS/IPS operational procedures. Create a runbook for ongoing IDS/IPS management: (a) Signature updates: verify daily auto-update is functioning, check for failed updates weekly. (b) False positive review: review new alerts weekly, tune as needed, document all exclusions. (c) Mode switching: document the process for switching new rules from IDS to IPS (detection period, baseline criteria, change management). (d) Incident response: when a true positive is detected in IPS mode, document the investigation workflow — event review, VM isolation, forensic data collection. (e) Signature profile review: quarterly review of profile configurations to ensure severity levels match current threat landscape.

Validation Gate

Check: IDS/IPS deployed with tuned profiles and prevention mode active

Expected: IDS/IPS enabled with current signatures, 3 severity-based profiles created and referenced by Distributed IDS/IPS rules, false positives identified and tuned via signature exclusions, IPS prevention mode actively blocking attack traffic while allowing legitimate, host-level IPS statistics confirming enforcement, operational runbook documented

Common Errors

IDS/IPS toggle is grayed out — cannot enable
Fix: IDS/IPS requires the vDefend or NSX Advanced license. Check: NSX Manager → System → Licenses → verify 'Threat Prevention' or 'Advanced' license is applied. Also verify: the NSX Manager nodes must have outbound HTTPS connectivity to the Broadcom signature update service — air-gapped environments require offline signature bundle import.
Signature update fails with 'Connection Error'
Fix: NSX Manager needs outbound HTTPS (443) connectivity to the Broadcom Threat Intelligence Cloud. Check: (1) DNS resolution from NSX Manager to the update endpoint, (2) firewall rules allowing HTTPS outbound from NSX Manager management IP, (3) proxy configuration if the environment uses an HTTP proxy. For air-gapped environments: download the signature bundle manually and import via NSX Manager → IDS/IPS → Settings → Manual Upload.
IPS mode blocks legitimate traffic — production impact
Fix: Immediate response: switch the affected Distributed IDS/IPS rule back to 'Detect Only' mode and publish — this restores traffic flow within seconds. Then: identify the signature that caused the block (Events → filter by Action='Dropped', timestamp matching the incident), exclude that signature in a copy of the profile and apply it through a dedicated IDS/IPS rule for the affected flow placed above the broader rule (or use an Exempt rule if that flow should not be inspected), re-run test traffic to verify the tuning works, then switch back to IPS. Root cause: insufficient detection-mode baseline period. Extend the baseline for this traffic flow.
IDS events show high volume of Medium/Low severity alerts causing alert fatigue
Fix: Medium and Low severity signatures have higher false-positive rates. For internal segments: use profiles with only Critical+High severity. For DMZ: start with Critical+High, add Medium only after tuning Medium-severity false positives. Never enable Low severity in production — the signal-to-noise ratio is too low for operational use. Use Low severity only during threat hunting exercises.
IDS/IPS not inspecting traffic despite a published Distributed IDS/IPS rule
Fix: Verify: (1) The DFW is enabled and the matching DFW rule action is 'Allow' — IDS/IPS only inspects traffic the DFW allows; a 'Drop' rule never reaches IDS inspection. (2) IDS/IPS is enabled on the cluster/hosts (Settings) and the IDS rule's Sources/Destinations/Services/Applied To actually match the flow. (3) The IDS rule is published (not in draft state). (4) The security profile on the rule is not empty — verify the profile has signatures enabled.

Final Validation

IDS/IPS lifecycle complete from detection through tuning to prevention

✓ IDS/IPS enabled → Feature enabled with current signature bundle (15,000+ signatures)

✓ Signature profiles → 3 profiles created with severity-appropriate filtering per zone

✓ Detection mode alerts → True positive alerts for simulated attacks, no false positives after tuning

✓ False-positive tuning → Exclusions documented with signature IDs, affected flows, and tuning rationale

✓ IPS prevention → Attack traffic actively blocked, legitimate traffic unaffected

✓ Host-level verification → nsxcli get ids status / get ids profiles / get ids events stats showing enabled state, pushed profiles and rising event and drop counters

Cleanup / Restore

• Switch the Distributed IDS/IPS rules back to Detect Only (or delete the Lab-IDS policy)

• Delete IDS/IPS signature profiles

• Remove signature exclusions

• Optionally disable IDS/IPS globally if not needed for subsequent labs

Design Reflection (VCDX)

IDS/IPS deployment strategy demonstrates operational maturity in a VCDX design. Key defense points: (1) Phased deployment (detect → tune → prevent) shows you understand that security controls must be validated before enforcement — skipping this phase causes outages. (2) Per-rule IDS binding (not global) shows efficiency awareness — inspecting only DFW-allowed traffic reduces CPU overhead by 60-80% vs inspecting all traffic. (3) Severity-based profiles per zone show defense-in-depth thinking — DMZ needs aggressive detection, internal segments need balanced coverage. VCDX panelists will ask: 'How do you handle a new signature update that introduces false positives in IPS mode?'

Requirements

  • Detect and prevent known attack patterns (SQL injection, XSS, directory traversal) at the vNIC level
  • Zero false-positive rate on legitimate traffic before enabling prevention mode
  • Automated daily signature updates from Broadcom Threat Intelligence Cloud

Constraints

  • IDS/IPS requires vDefend or NSX Advanced license — additional cost
  • IDS inspection adds CPU overhead on ESXi hosts (~5-10% for inspected traffic flows)
  • Air-gapped environments cannot auto-update signatures — require manual bundle import process

Assumptions

  • Application traffic patterns are stable enough for a 7-14 day detection baseline to be representative
  • Broadcom signature updates are vetted and don't introduce mass false positives (historically reliable)
  • SOC team has capacity to review IDS alerts during the detection phase

Risks

  • New signature update introduces false positive in IPS mode — blocks legitimate production traffic. Mitigation: staged signature rollout, test in detection mode for 24h before auto-applying to prevention rules.
  • IDS alert fatigue — SOC team ignores alerts due to volume. Mitigation: severity-based profiles, false-positive tuning, alert routing with priority filtering.
  • Zero-day attacks bypass signature-based detection. Mitigation: layer with NDR behavioral analytics (Lab vdefend-03) for anomaly-based detection.

Self-Assessment Discussion Prompts

  1. How do you handle IDS/IPS in an environment with 500+ applications — is per-application tuning feasible?
  2. When should you use Distributed IDS/IPS (host-based) vs Gateway IDS/IPS (Edge-based)?
  3. How do you measure the CPU overhead of IDS/IPS inspection and when does it become a performance concern?
  4. What is the appropriate detection-mode baseline period for a healthcare environment with monthly billing cycles?

Extensions

Create a custom IDS signature for a specific application vulnerability unique to your environment

Automate false-positive tuning reports using the NSX API: GET /policy/api/v1/infra/settings/firewall/security/intrusion-services/ids-events

Design an IDS/IPS deployment plan for a 500-VM environment with 5 security zones and zone-specific profiles

Integrate IDS/IPS alerts with SIEM (Splunk/QRadar) via syslog forwarding and create correlation rules

⚠ Known Pitfalls

Deploying IPS prevention mode without a detection-mode baseline period — the #1 cause of self-inflicted outages. Always run detection mode for 7-14 days minimum before switching to prevention.
Enabling all severity levels (including Low) on internal segments — Low-severity signatures generate massive alert volume with poor signal-to-noise ratio. Use Low only for DMZ or during active threat hunting.
Writing an any/any Distributed IDS/IPS rule with a broad profile — every allowed workload flow is then inspected against every signature, wasting CPU and generating irrelevant alerts. Scope IDS rules to the application flows that need inspection and pick the profile per flow (vmkernel traffic such as vMotion and vSAN is not in the vNIC data path and is never inspected).
Not documenting signature exclusions — when you disable a signature for false-positive tuning, document WHY. Future signature reviews need this context to determine if the exclusion is still valid or if the application changed.
Treating IDS/IPS as set-and-forget — signatures need regular review: new application deployments may trigger new false positives, retired applications may leave unnecessary exclusions, new threat patterns require profile updates

References

  • vDefend Advanced Threat Prevention 9.x documentation — Distributed IDS/IPS: techdocs.broadcom.com
  • vDefend Distributed IDS/IPS Design Guide: techdocs.broadcom.com
  • NSX IDS/IPS Signature Management: techdocs.broadcom.com
  • vDefend Advanced Threat Prevention 9.1 — Distributed IDS/IPS and Gateway IDS/IPS for VMware vDefend (Managing IDS/IPS Events, Managing System Signatures): https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/vdefend/vdefend-atp/9-1/vdefend-ids-ips.html
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.