Lab: IDS/IPS Deployment, False-Positive Tuning & Prevention Mode
Objectives
- Enable NSX Distributed IDS/IPS and configure signature bundle updates from Broadcom Threat Intelligence Cloud
- Create signature profiles with severity-based filtering (Critical/High/Medium/Low) for different security zones
- Deploy IDS in detection-only mode and analyze alert quality before switching to prevention
- Identify and tune false positives using signature exclusions and per-VM/per-flow exceptions
- Switch to IPS prevention mode with validated confidence and verify active blocking
- Analyze IDS/IPS events with CVSS scoring, MITRE ATT&CK mapping, and affected VM context
Prerequisites
VCF 9.0 with NSX 9.0.x and vDefend license enabled. IDS/IPS feature requires NSX Advanced or vDefend add-on license. 3+ VMs with network connectivity for attack simulation. NSX Manager with outbound HTTPS access to Broadcom signature update service.
Prior labs: vdefend-01
Required skills:
- NSX DFW rule creation (from Lab vdefend-01)
- Basic security concepts (IDS vs IPS, signatures, false positives)
- HTTP request structure (for simulating web attacks)
- CVSS scoring fundamentals
Lab Environment
VCF 9.0 with NSX overlay networking. 3 VMs from Lab vdefend-01 (web-01, app-01, db-01). Web-01 running a simple HTTP service (e.g., nginx or python http.server on port 8080). App-01 running an application that responds to HTTP requests.
Tasks
Task 1 IDS/IPS Deployment, False-Positive Tuning & Prevention Mode
Deploy the full IDS-to-IPS lifecycle: enable detection, generate and analyze alerts, identify and suppress false positives, then switch to prevention mode with confidence that legitimate traffic won't be blocked. This mirrors the real-world deployment pattern — organizations that skip the detection phase and go straight to prevention invariably break production applications.
Enable IDS/IPS globally. NSX Manager → Security → IDS/IPS & Malware Prevention → Settings → toggle IDS/IPS to 'Enabled'. For Distributed IDS/IPS (host-based), ensure 'Distributed IDS/IPS' is enabled. For Gateway IDS/IPS (Edge-based), enable on the T1 gateway if needed for north-south inspection. Click 'Update Signatures' → NSX Manager downloads the latest signature bundle from Broadcom Threat Intelligence Cloud. Wait for status: 'Signatures up to date' (shows timestamp of latest update). Verify signature count: typically 15,000-20,000+ signatures in the full bundle. Configure auto-update schedule: Settings → Auto Update → enable daily signature updates to ensure new threat coverage.
Create IDS/IPS Signature Profiles for different security zones. NSX Manager → Security → IDS/IPS & Malware Prevention → Profiles → Add Profile. (a) Profile 'IDS-DMZ-Strict': Severity = Critical + High + Medium. This catches most threats including medium-confidence signatures — appropriate for internet-facing DMZ where attack surface is highest. (b) Profile 'IDS-Internal-Balanced': Severity = Critical + High only. Excludes Medium/Low to reduce noise on internal segments where attack probability is lower. (c) Profile 'IDS-DB-Critical-Only': Severity = Critical only. Minimal alerting for database tier — focused on high-confidence, severe threats to avoid alert fatigue in low-attack-surface zones. Each profile filters the full signature set by CVSS severity — Critical (CVSS 9.0-10.0), High (7.0-8.9), Medium (4.0-6.9), Low (0.1-3.9).
Bind IDS profiles to DFW rules. NSX Manager → Security → Distributed Firewall → edit the Web-to-App allow rule (from Lab vdefend-01). In the rule row, click the IDS column → Enable IDS → select profile 'IDS-DMZ-Strict' → Mode: 'Detect Only'. This binds IDS inspection to traffic matching this specific DFW rule — only web→app traffic on TCP 8080 is inspected, not all traffic on the host. Repeat for App-to-DB rule: bind 'IDS-DB-Critical-Only' profile in Detect Only mode. Publish. This per-rule binding is a key NSX advantage — IDS inspection only runs on traffic that matches the DFW rule, reducing CPU overhead compared to inspecting all traffic.
Generate test traffic including simulated attacks. From web-01, send a mix of legitimate and malicious-pattern requests: (a) Legitimate traffic (should NOT trigger alerts): curl http://app-01:8080/api/health — health check endpoint. curl http://app-01:8080/api/users?page=1&limit=10 — normal API query. for i in $(seq 1 10); do curl http://app-01:8080/api/data; done — repeated normal requests. (b) Simulated attack traffic (SHOULD trigger alerts): curl 'http://app-01:8080/search?q=1%27%20OR%201%3D1--' — SQL injection pattern. curl 'http://app-01:8080/../../etc/passwd' — directory traversal. curl -A 'nikto/2.1.6' http://app-01:8080/ — scanner user-agent. curl 'http://app-01:8080/<script>alert(1)</script>' — XSS attempt. Wait 2-3 minutes for IDS event processing and correlation.
Analyze IDS events and identify false positives. NSX Manager → Security → IDS/IPS & Malware Prevention → Events. Review the events dashboard: (a) Sort by severity (Critical/High first). (b) For each event, examine: Signature ID (e.g., sid:2024897), Signature Name (e.g., 'ET WEB_SERVER SQL Injection Attempt'), CVSS Score, Affected VMs (source + destination), Timestamp, Classification (e.g., Web Application Attack). (c) Expected TRUE positives: SQL injection, directory traversal, XSS, scanner detection. (d) Check for FALSE positives: did any legitimate requests (health check, normal API) trigger alerts? Common false positives: API requests with special characters in query strings, health monitoring tools that match scanner signatures, internal vulnerability scanning triggering IDS. Document: for each false positive, note the Signature ID, the traffic that triggered it, and why it's benign.
Tune false positives using signature exclusions. For each identified false positive, choose a tuning approach: (a) Global signature disable: NSX Manager → Profiles → edit 'IDS-DMZ-Strict' → find signature by ID → toggle to 'Disabled'. Use when the signature is fundamentally irrelevant to your environment (e.g., a Windows-specific exploit on a Linux-only infrastructure). (b) Per-flow exclusion: Security → IDS/IPS → Exclusions → Add. Specify: Signature ID + Source VM + Destination VM. This suppresses the signature only for this specific traffic flow — more surgical than global disable. Use when the signature is valid generally but false-positive for one specific application. (c) Profile adjustment: switch from 'IDS-DMZ-Strict' (Crit+High+Med) to a custom profile that excludes the noisy category. After tuning, re-run the SAME test traffic from Step 4 and verify: (1) attack traffic still generates alerts, (2) legitimate traffic no longer triggers false positives.
Establish detection-mode baseline period. Before switching to IPS prevention, run in detection mode for a baseline period to build confidence: (a) Lab environment: 24-48 hours of normal application operation in IDS mode. (b) Production recommendation: 7-14 days minimum, covering all business cycle patterns (weekday/weekend, month-end processing, batch jobs). During baseline: monitor false positive rate daily. Target: zero false positives on legitimate traffic for 3+ consecutive days before switching to prevention. Document the false-positive tuning log: date, signature ID, action taken (disabled/excluded), reason.
Switch to IPS Prevention Mode. After achieving zero false positives during baseline: Edit the DFW rules → change IDS/IPS mode from 'Detect Only' to 'Detect & Prevent'. Publish. IMPORTANT: switch one rule at a time, starting with the least critical traffic flow. Monitor for 30 minutes after each switch before proceeding to the next rule. After switching Web-to-App rule to IPS: (a) Re-run SQL injection test: curl 'http://app-01:8080/search?q=1%27%20OR%201%3D1--' → connection should be RESET or timed out (IPS blocks the packet). (b) Verify legitimate traffic: curl http://app-01:8080/api/health → should succeed (not matching any IDS signature). (c) Check events: NSX Manager → Events → filter by Action='Dropped' → confirm attack traffic shows as blocked.
Verify IPS enforcement at the host level. SSH to the ESXi host running web-01. Run: nsxcli -c 'get ids-ips statistics' — shows packet inspection counts, alerts generated, and packets dropped by IPS. Check: dropped count should be > 0 after running attack traffic with IPS enabled. Run: nsxcli -c 'get ids-ips profile' — shows which IDS/IPS profiles are applied to which vNICs. Verify the IPS profile binding matches your DFW rule configuration. Also check VCF Operations (if configured) for IDS/IPS metrics: alerts per signature, dropped traffic volume, false positive trend over time.
Document IDS/IPS operational procedures. Create a runbook for ongoing IDS/IPS management: (a) Signature updates: verify daily auto-update is functioning, check for failed updates weekly. (b) False positive review: review new alerts weekly, tune as needed, document all exclusions. (c) Mode switching: document the process for switching new rules from IDS to IPS (detection period, baseline criteria, change management). (d) Incident response: when a true positive is detected in IPS mode, document the investigation workflow — event review, VM isolation, forensic data collection. (e) Signature profile review: quarterly review of profile configurations to ensure severity levels match current threat landscape.
Validation Gate
Check: IDS/IPS deployed with tuned profiles and prevention mode active
Expected: IDS/IPS enabled with current signatures, 3 severity-based profiles created and bound to DFW rules, false positives identified and tuned via signature exclusions, IPS prevention mode actively blocking attack traffic while allowing legitimate, host-level IPS statistics confirming enforcement, operational runbook documented
Common Errors
Final Validation
IDS/IPS lifecycle complete from detection through tuning to prevention
✓ IDS/IPS enabled → Feature enabled with current signature bundle (15,000+ signatures)
✓ Signature profiles → 3 profiles created with severity-appropriate filtering per zone
✓ Detection mode alerts → True positive alerts for simulated attacks, no false positives after tuning
✓ False-positive tuning → Exclusions documented with signature IDs, affected flows, and tuning rationale
✓ IPS prevention → Attack traffic actively blocked, legitimate traffic unaffected
✓ Host-level verification → nsxcli IDS/IPS statistics showing inspection counts and drops
Cleanup / Restore
• Switch DFW rules back from IPS to IDS mode (or disable IDS/IPS on rules)
• Delete IDS/IPS signature profiles
• Remove signature exclusions
• Optionally disable IDS/IPS globally if not needed for subsequent labs
Design Reflection (VCDX)
IDS/IPS deployment strategy demonstrates operational maturity in a VCDX design. Key defense points: (1) Phased deployment (detect → tune → prevent) shows you understand that security controls must be validated before enforcement — skipping this phase causes outages. (2) Per-rule IDS binding (not global) shows efficiency awareness — inspecting only DFW-allowed traffic reduces CPU overhead by 60-80% vs inspecting all traffic. (3) Severity-based profiles per zone show defense-in-depth thinking — DMZ needs aggressive detection, internal segments need balanced coverage. VCDX panelists will ask: 'How do you handle a new signature update that introduces false positives in IPS mode?'
Requirements
- Detect and prevent known attack patterns (SQL injection, XSS, directory traversal) at the vNIC level
- Zero false-positive rate on legitimate traffic before enabling prevention mode
- Automated daily signature updates from Broadcom Threat Intelligence Cloud
Constraints
- IDS/IPS requires vDefend or NSX Advanced license — additional cost
- IDS inspection adds CPU overhead on ESXi hosts (~5-10% for inspected traffic flows)
- Air-gapped environments cannot auto-update signatures — require manual bundle import process
Assumptions
- Application traffic patterns are stable enough for a 7-14 day detection baseline to be representative
- Broadcom signature updates are vetted and don't introduce mass false positives (historically reliable)
- SOC team has capacity to review IDS alerts during the detection phase
Risks
- New signature update introduces false positive in IPS mode — blocks legitimate production traffic. Mitigation: staged signature rollout, test in detection mode for 24h before auto-applying to prevention rules.
- IDS alert fatigue — SOC team ignores alerts due to volume. Mitigation: severity-based profiles, false-positive tuning, alert routing with priority filtering.
- Zero-day attacks bypass signature-based detection. Mitigation: layer with NDR behavioral analytics (Lab vdefend-03) for anomaly-based detection.
Self-Assessment Discussion Prompts
- How do you handle IDS/IPS in an environment with 500+ applications — is per-application tuning feasible?
- When should you use Distributed IDS/IPS (host-based) vs Gateway IDS/IPS (Edge-based)?
- How do you measure the CPU overhead of IDS/IPS inspection and when does it become a performance concern?
- What is the appropriate detection-mode baseline period for a healthcare environment with monthly billing cycles?
Extensions
Create a custom IDS signature for a specific application vulnerability unique to your environment
Automate false-positive tuning reports using the NSX API: GET /policy/api/v1/infra/settings/firewall/security/intrusion-services/ids-events
Design an IDS/IPS deployment plan for a 500-VM environment with 5 security zones and zone-specific profiles
Integrate IDS/IPS alerts with SIEM (Splunk/QRadar) via syslog forwarding and create correlation rules
⚠ Known Pitfalls (from Community KB)
References
- NSX 4.2 IDS/IPS Administration Guide: techdocs.broadcom.com
- vDefend Distributed IDS/IPS Design Guide: techdocs.broadcom.com
- NSX IDS/IPS Signature Management: techdocs.broadcom.com
- KB 88205 — NSX IDS/IPS Troubleshooting: https://knowledge.broadcom.com/external/article?legacyId=88205