Academy/VCAP — VCF Operations (3V0-22.25)/Lab: Design Production Alert Policy with Escalation
This lab targets VCF 9.0

Lab: Design Production Alert Policy with Escalation

VCF 9.0Advancedvcap-advanced⏱ 120 min

VCF Operations alerting — symptom definitions, dynamic thresholds, alert correlation, notification chains, automated remediation

Objectives

  • Create symptom definitions with both static and dynamic thresholds
  • Build multi-symptom alert definitions with AND/OR correlation logic
  • Configure tiered notification chains with escalation timeouts
  • Implement automated remediation actions with safety controls
  • Design maintenance windows for planned change suppression
  • Understand the full alert lifecycle: fire → take ownership → escalate (delayed email unless cancelled) → remediate → clear

Prerequisites

VCF Operations with 14+ days of metric data (required for dynamic thresholds learning), SMTP configured for email notifications, webhook endpoint available (ServiceNow, Slack, or test endpoint)

Prior labs: vcap-ops-01, vcap-ops-02

Required skills:

  • Alert design principles
  • SMTP/webhook integration concepts
  • Change management procedures

Lab Environment

VCF Operations cluster with vCenter adapter. At least one cluster with VMs that can be stress-tested to trigger alerts (stress-ng or similar). Email relay configured. Optional: ServiceNow/Slack webhook for escalation testing.

Tasks

Task 1 Production Alert Policy with Multi-Level Escalation

Production alerting follows the 'signal-to-noise' principle: every alert should be actionable. Multi-symptom correlation (CPU high AND Memory high AND Disk latency high) eliminates single-metric false positives. Dynamic thresholds adapt to each object's baseline, avoiding generic thresholds that alert on a busy-but-normal VM. Escalation ensures critical alerts don't sit Open without an owner. Automated remediation reduces MTTR for known scenarios.

Design a complete alerting framework that mirrors production operations: symptom definitions detect individual conditions, alert definitions correlate multiple symptoms to reduce false positives, notification chains escalate alerts that stay Open without an owner, and automated remediation handles well-understood scenarios — all with safety controls to prevent runaway automation.

Step 1

Create symptom definitions — static thresholds. Infrastructure Operations > Configurations > Symptom Definitions tile > Add. (a) 'High-CPU-Static': Metric=cpu.usage, Condition=> 90%, Wait Cycles=3, Cancel Cycles=3. The wait cycles mean 3 consecutive violations (15 minutes at 5-min intervals) before the symptom fires — this eliminates brief spikes. (b) 'High-Memory-Static': Metric=mem.usage, Condition=> 90%, Wait Cycles=3. (c) 'High-Disk-Latency': Metric=disk.maxTotalLatency, Condition=> 30ms, Wait Cycles=2 (disk latency issues are more urgent).

Step 2

Create symptom definitions — dynamic thresholds. Add → (d) 'CPU-Anomaly-DT': Metric=CPU|Usage, Threshold=Dynamic Threshold, Condition=Above Threshold (choose 'Abnormal' to catch deviations in both directions), Criticality=Warning. The symptom editor has no sensitivity field: dynamic thresholds are calculated per object from its history (a global setting under Administration → Global Settings → Dynamic Thresholds), and noise is tuned with the symptom's wait/cancel cycles and the alert's wait cycle. Dynamic thresholds learn each object's normal behavior over 14+ days and alert when the metric deviates significantly from its learned baseline. A VM that normally runs at 80% CPU won't alert at 82%, but a VM that normally runs at 10% will alert at 50%. (e) 'Memory-Anomaly-DT': Metric=Memory|Usage, Threshold=Dynamic Threshold, Condition=Above Threshold, Criticality=Warning.

Step 3

Create correlated alert definition. Infrastructure Operations > Configurations > Alert Definitions tile > Add. Name='VM-Performance-Degradation'. Impact=Health. Criticality=Critical. Symptoms: Add all three static symptoms with AND logic: 'High-CPU-Static AND High-Memory-Static AND High-Disk-Latency'. This means the alert only fires when ALL THREE conditions are true simultaneously — dramatically reducing false positives compared to alerting on each condition independently. Object Type=VirtualMachine.

Step 4

Create anomaly alert definition. Add → Name='VM-Behavioral-Anomaly'. Impact=Risk. Criticality=Warning. Symptoms: 'CPU-Anomaly-DT OR Memory-Anomaly-DT'. Use OR logic here because any behavioral anomaly warrants investigation. This alert catches performance changes that static thresholds miss — a VM that suddenly doubles its CPU usage from 20% to 40% won't trigger the 90% static threshold but WILL trigger the dynamic threshold.

Step 5

Configure notification chain — tiered escalation. Infrastructure Operations > Configurations > Outbound Settings tile > Add: (a) SMTP: server=mail.lab.local, port=25. (b) Webhook: URL=https://slack.webhook.example/alert (or ServiceNow endpoint). Then Notifications tile > Add: Rule 1: Alert='VM-Performance-Degradation', Criticality=Critical → Standard Email plug-in, sent immediately to ops-team@lab.local. Rule 2: same alert definition, Control State = Open, Webhook plug-in to Slack/ServiceNow — sent when the alert is raised: Delay to notify, Notify again and Max Notifications are options of the Standard Email plug-in only, so a timed escalation to Slack/ServiceNow belongs in the ITSM or paging tool. Rule 3: same criteria, Standard Email plug-in to manager@lab.local, Delay to notify = 30 minutes — the email is not sent if the alert is cancelled within those 30 minutes; set Notify again and Max Notifications on this rule for reminders (management escalation).

Step 6

Configure automated remediation. Infrastructure Operations > Configurations > Recommendations tile > create a recommendation with the action Move VM and make it the first-priority recommendation of 'VM-Performance-Degradation' (only a first-priority recommendation with an associated action can be automated). Then Infrastructure Operations > Configurations > Policy Definition tile > edit the policy applied to the test VMs > Alert / Symptom Definitions workspace > for this alert set Automate = Local > Save. Supported actions include Move VM, Power Off/On VM, Set CPU Count and Memory for VM, Delete Idle/Powered Off VM. Safety comes from design, not product knobs: scope the policy to a small custom group, leave DRS in charge of placement, and verify runs under Administration → Control Panel → Recent Tasks (Submitted By = automationAdmin).

Step 7

Configure maintenance window. Infrastructure Operations > Configurations > Maintenance Schedules tile > Add. Name='Monthly-Patch-Window'; set the days and time window (for example Saturday 22:00-06:00 UTC). A maintenance schedule has no object scope of its own and does nothing until it is part of a policy (VCF 9.0.1 docs p.4242): edit the policy applied to a custom group containing the clusters being patched and set its Maintenance Schedule element to 'Monthly-Patch-Window'. During the window VCF Operations stops collecting metrics for those objects, generates no anomalies or alerts for them and cancels their active symptoms and alerts (p.4242). This prevents alert storms during planned maintenance. Verify: bind the schedule to a test group, run a stress test on a member during the window — no alerts should fire for it.

Step 8

Test the alert pipeline — trigger a real alert. SSH to a test VM and run: stress-ng --cpu 4 --vm 2 --vm-bytes 1G --timeout 1200s (stresses CPU and memory for 20 minutes). Monitor: (a) Symptom Definitions page → 'High-CPU-Static' should show 'Active' after 15 minutes (3 × 5-min cycles); (b) If memory also exceeds 90% AND disk latency spikes, the correlated alert 'VM-Performance-Degradation' fires; (c) Email notification arrives within 60 seconds of alert; (d) The Slack/ServiceNow webhook notification arrives when the alert is raised; if the alert is not cancelled within 30 minutes, the delayed manager email is sent; (e) The Move VM action runs without user intervention only if Automate = Local is set for this alert in the policy (check Administration > Control Panel > Recent Tasks: Submitted By = automationAdmin).

Step 9

Take ownership of and close the alert. In VCF Operations → Alerts → select the 'VM-Performance-Degradation' alert → Take Ownership (assigns the alert to you). Other alert actions: Assign to, Release Ownership, Suspend, Cancel. Add notes: 'Triggered by stress test — expected behavior.' Stop the stress-ng process on the VM. Monitor: symptoms should clear after 3 cancel cycles (15 minutes). Alert auto-closes when all symptoms clear. If symptoms don't clear: manually cancel the alert and investigate why metrics remain elevated.

Step 10

Review alert audit trail. Alerts → select the alert → Timeline tab. Shows the complete lifecycle: (a) Symptoms triggered (timestamp for each), (b) Alert fired, (c) Email notification sent, (d) Slack/ServiceNow webhook and manager escalation email (+30 min), (e) Ownership taken by admin, (f) Remediation action executed (if applicable), (g) Symptoms cleared, (h) Alert auto-closed. This audit trail is critical for post-incident review and compliance documentation. Export the timeline for incident reports.

Validation Gate

Check: Complete alert pipeline operational: symptom → alert → notification → escalation → remediation → clear

Expected: Static and dynamic symptom definitions active, correlated alert definition firing correctly, email notification within 60 seconds, webhook notification to Slack/ServiceNow when the alert is raised, manager email after a 30-minute Delay to notify unless the alert is cancelled first, policy-automated Move VM recommendation executing on the scoped group, maintenance window suppressing alerts during planned work

Common Errors

Dynamic threshold alerts fire constantly (too sensitive)
Fix: There is no per-symptom sensitivity to turn; raise the symptom's Wait Cycle (and the alert's Wait Cycle) so a single spike does not trigger, or use the static-threshold symptoms for well-known limits. Also verify the object has 14+ days of metric data — DT with insufficient historical data produces erratic baselines.
Correlated alert never fires despite individual symptoms being active
Fix: AND logic requires ALL symptoms active simultaneously. If CPU spikes resolve before disk latency spikes, the correlation window is missed. Consider: (a) increase wait cycles to widen the correlation window, (b) use OR logic if any single condition warrants investigation.
Email notification not received
Fix: Check: (1) SMTP plugin configured and tested (Outbound Settings → Test); (2) notification rule matches the correct alert definition and criticality; (3) email server not rejecting due to SPF/DKIM (check SMTP server logs); (4) notification rule is enabled (not disabled during testing).
Automated remediation causes vMotion loop
Fix: The target host is also at high utilization, causing the VM to be migrated back. Fix: (1) keep DRS fully automated so placement is cluster-aware and the Move VM action is the exception, not the balancer; (2) raise the alert's wait and cancel cycles so a transient spike does not re-trigger; (3) scope Automate = Local to a small custom group while testing; (4) review Control Panel → Recent Tasks for repeated automationAdmin moves and switch Automate off if they recur.

Final Validation

Production alerting framework operational with multi-level escalation and automated remediation

✓ Static symptoms → Fire after wait cycles met, clear after cancel cycles

✓ Dynamic thresholds → Detect behavioral anomalies relative to learned baselines

✓ Correlated alerts → Fire only when all AND symptoms are simultaneously active

✓ Email notification → Delivered within 60 seconds of alert firing

✓ Escalation → Webhook notification sent when the alert is raised; manager email sent after Delay to notify = 30 minutes unless the alert is cancelled first

✓ Automated remediation → Move VM runs via policy Automate = Local; visible under Control Panel → Recent Tasks as automationAdmin

✓ Maintenance window → Suppresses alerts during scheduled period

Cleanup / Restore

• Stop stress-ng on test VM

• Cancel test alerts

• Set Automate back to Off (or inherit) for the alert in the policy

• Delete test maintenance window

• Optionally delete alert/symptom definitions if not needed

Design Reflection (VCDX)

Alert design is a key VCDX operational architecture topic. Demonstrate: (1) signal-to-noise ratio — multi-symptom correlation eliminates false positives; (2) dynamic thresholds adapt to workload profiles — no single static threshold fits all VMs; (3) escalation ensures nothing falls through the cracks; (4) automated remediation reduces MTTR while safety controls prevent runaway automation. In defense, discuss how you'd scale this from 100 to 10,000 VMs — what changes?

Requirements

  • Zero critical alerts left Open without an owner beyond 30 minutes
  • False positive rate < 5%
  • MTTR < 15 minutes for known performance scenarios

Constraints

  • Dynamic thresholds require 14+ days of learning data
  • SMTP/webhook must be reliable — notification failure is silent
  • Automated remediation limited to safe actions (vMotion, rightsizing — not reboot, delete)

Assumptions

  • Operations team monitors email/Slack during business hours
  • After-hours alerts escalate to on-call rotation
  • vMotion is safe for all VMs (no license-locked or physical-device-dependent VMs)

Risks

  • Alert fatigue from too many low-criticality alerts — team ignores all alerts
  • DT learning phase produces false positives during first 14 days
  • Remediation action makes things worse (vMotion to overloaded host)

Self-Assessment Discussion Prompts

  1. How do you measure and improve your alert signal-to-noise ratio over time?
  2. How do you tune dynamic-threshold symptoms for a mixed-workload environment when the only levers are wait/cancel cycles, criticality and the alert definition?
  3. How do you handle alerting for VMs that legitimately run at 95%+ CPU (e.g., HPC workloads)?
  4. What governance process controls who can enable automated remediation?

Extensions

Integrate alert notifications with PagerDuty for on-call rotation management

Build a ServiceNow ITSM integration that auto-creates incident tickets from critical alerts

Create a 'noise report' dashboard showing alert frequency per definition — identify and tune noisy alerts

Implement a phased DT rollout: start the dynamic symptoms with a long Wait Cycle (e.g. 6) and Criticality Info for two weeks, review the fired symptoms, then shorten the wait cycle and raise criticality

⚠ Known Pitfalls

Creating alerts on single metrics without correlation — generates 10x more false positives than multi-symptom alerts
Setting wait cycles to 1 — every 5-minute spike triggers an alert; use 3+ for production
Setting Automate = Local in a broad policy before proving the recommendation on a small custom group
Not configuring maintenance windows — planned patches trigger alert storms that erode team trust in monitoring

References

  • VCF Operations Administration Guide — Alerts and Symptoms: techdocs.broadcom.com
  • VCF Operations Outbound Plugins Guide — Email, Webhook, SNMP: techdocs.broadcom.com
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.