Academy/VMware Cloud Operations 8.x Professional (2V0-32.24)/Alert Automation & Closed-Loop Remediation
This lab targets VCF 9.0

Alert Automation & Closed-Loop Remediation

VCF 9.0Intermediatevcp-foundation⏱ 75 min

Objectives

  • Configure a multi-action alert with email, webhook and SNMP notifications, a ticketing integration, and an approval-gated remediation workflow.

Prerequisites

VCF lab environment deployed and operational

Lab Environment

VCF 9.0 lab environment with VCF Operations 9.0; UI paths in this lab follow VCF Operations 9.0. The 2V0-32.24 Cloud Operations 8.x exam itself targets vRealize Operations 8.6, vRealize Log Insight 8.8 and vRealize Suite Lifecycle Manager 8.8, so map any 8.x exam detail accordingly.

Tasks

Task 1 Alert Automation & Closed-Loop Remediation

Configure multi-action alerts with REST/SMTP/SNMP notifications and integration with ticketing system.

Step 1

Define two symptoms (Infrastructure Operations > Configurations > Symptom Definitions tile > Add); a symptom definition evaluates one condition on one metric or property, and the AND belongs in the alert definition (step 2).

  • Symptom 1: "Host Memory Usage Critical" — base object type Host System (ESX host); metric Mem|Usage (%) (mem|host_usagePct) > 90; criticality Critical
  • Symptom 2: "Host Memory Swap Used High" — base object type Host System; metric Mem|Swap Used (KB) (mem|swapused_average) > 1000000; criticality Critical
  • Wait cycle: 2 on each (the condition must hold for two 5-minute collection cycles, about 10 minutes, before the symptom triggers)
  • Cancel cycle: 2 (clear after two cycles without the condition)
  • Confirm both metric keys in the metric picker on your build (VCF 9.0.1 docs p.4295-4296, Memory Metrics for Host Systems)
Step 2

Create the alert definition (Infrastructure Operations > Configurations > Alert Definitions tile > Add).

  • Name: "Host Memory Critical Alert"
  • Symptoms: "Host Memory Usage Critical" and "Host Memory Swap Used High", evaluated with All (both must be true) (VCF 9.0.1 docs p.3175)
  • Recommendation: migrate VMs off the host (DRS) or restart the noisiest VM; attach the orchestrator remediation workflow as the recommendation's action
  • Activate the alert definition in the policy that applies to the prod cluster
  • Notification preference: once per state change (fire once, clear once)
Step 3

Configure the actions.

  • Outbound Settings (Infrastructure Operations > Configurations > Outbound Settings tile): create a Standard Email Plugin instance (SMTP host, sender, TLS), a Webhook Notification Plugin instance pointing at the ticketing system's incident API (the Rest Notification Plugin is deprecated in 9.0), and an SNMP Trap Plugin instance for the NOC trap receiver
  • Notifications (Notifications tile): create one rule per plugin for "Host Memory Critical Alert", criticality Critical, scope the prod cluster; the webhook payload template carries host name, metric values and the alert ID so the ticket is created with the right fields
  • Remediation: with the Management Pack for Orchestrator configured, the alert's recommendation shows Run Action for the "Rebalance-Host-Memory" workflow (a DRS-driven vMotion of the top memory consumer); keep it manual-run so a human approves each remediation
  • Rate limiting: set the notification rule to notify on state change only, and give the workflow a cooldown check (do not run twice on the same host within 30 minutes)
Step 4

Test the alert (inject memory pressure).

  • Pick one lab host and power on enough memory-heavy test VMs to drive host memory above 90% with active swapping (in Holodeck, nested hosts reach this quickly)
  • After one collection cycle plus the 10-minute sustain window the symptom triggers and the alert fires
  • Confirm the three notifications: email received, ticket created by the webhook (note its ID), SNMP trap seen at the receiver
  • Open the alert under Troubleshoot > Alerts, show the Action column and click Run on the remediation; approve it in the VCF Operations orchestrator client, where workflows waiting for user input are listed in the Recent Workflow Runs panel of the dashboard and in the top-right notification menu
Step 5

Validate and close the loop.

  • The workflow vMotions the top memory consumer away; host memory drops below 90%
  • After two cancel cycles the alert clears and the clear notification is sent; the webhook rule for the cancelled state updates the ticket to resolved
  • Check Administration > Control Panel > Recent Tasks for the action run and Activity > Workflow Runs in the orchestrator for the workflow outcome
  • Record the end-to-end time from symptom trigger to ticket closure and document what would need to change for unattended remediation (rate limit, blast-radius guard, change-control exemption)

Validation Gate

Check: Verify lab completion

Expected: Lab exercise completed successfully

Common Errors

Alert automation executing remediation without rate limiting
Fix: A monitoring alert that triggers auto-remediation can cascade: first alert restarts service → service restart triggers second alert → second restart disrupts ongoing recovery. Implement: cooldown period (no re-trigger within 30 minutes), maximum execution count per hour, and alert deduplication.
Closed-loop remediation without rollback capability
Fix: Auto-remediation that changes configuration (restart, scale, reconfigure) without rollback capability can make the situation worse. Implement: pre-action snapshot/backup, execution logging, and automatic rollback if the remediation doesn't resolve the alert within a configured timeframe.
Not distinguishing between symptom alerts and root cause alerts
Fix: Multiple symptom alerts (high CPU, high latency, high queue depth) may share one root cause (storage controller failure). Remediating each symptom independently wastes effort. Use VCF Operations alert correlation to group related symptoms and remediate the root cause.

Final Validation

Lab completed successfully

✓ All steps completed → No errors observed

Cleanup / Restore

• Revert to snapshot if needed

Design Reflection (VCDX)

Alert automation with closed-loop remediation demonstrates AIOps capabilities in your design. VCDX panelists test whether auto-remediation is safe, controlled, and reversible.

⚠ Known Pitfalls

Auto-remediation without cooldown — cascading restarts make outages worse.
Treating symptoms as independent root causes — wastes remediation effort and delays actual fix.
Was this page useful?
Type to search. ↑ ↓ to move, Enter to open, Esc to close.