Alert Automation & Closed-Loop Remediation
Objectives
- Configure a multi-action alert with email, webhook and SNMP notifications, a ticketing integration, and an approval-gated remediation workflow.
Prerequisites
VCF lab environment deployed and operational
Lab Environment
VCF 9.0 lab environment with VCF Operations 9.0; UI paths in this lab follow VCF Operations 9.0. The 2V0-32.24 Cloud Operations 8.x exam itself targets vRealize Operations 8.6, vRealize Log Insight 8.8 and vRealize Suite Lifecycle Manager 8.8, so map any 8.x exam detail accordingly.
Tasks
Task 1 Alert Automation & Closed-Loop Remediation
Configure multi-action alerts with REST/SMTP/SNMP notifications and integration with ticketing system.
Define two symptoms (Infrastructure Operations > Configurations > Symptom Definitions tile > Add); a symptom definition evaluates one condition on one metric or property, and the AND belongs in the alert definition (step 2).
- Symptom 1: "Host Memory Usage Critical" — base object type Host System (ESX host); metric Mem|Usage (%) (
mem|host_usagePct) > 90; criticality Critical - Symptom 2: "Host Memory Swap Used High" — base object type Host System; metric Mem|Swap Used (KB) (
mem|swapused_average) > 1000000; criticality Critical - Wait cycle: 2 on each (the condition must hold for two 5-minute collection cycles, about 10 minutes, before the symptom triggers)
- Cancel cycle: 2 (clear after two cycles without the condition)
- Confirm both metric keys in the metric picker on your build (VCF 9.0.1 docs p.4295-4296, Memory Metrics for Host Systems)
Create the alert definition (Infrastructure Operations > Configurations > Alert Definitions tile > Add).
- Name: "Host Memory Critical Alert"
- Symptoms: "Host Memory Usage Critical" and "Host Memory Swap Used High", evaluated with All (both must be true) (VCF 9.0.1 docs p.3175)
- Recommendation: migrate VMs off the host (DRS) or restart the noisiest VM; attach the orchestrator remediation workflow as the recommendation's action
- Activate the alert definition in the policy that applies to the prod cluster
- Notification preference: once per state change (fire once, clear once)
Configure the actions.
- Outbound Settings (Infrastructure Operations > Configurations > Outbound Settings tile): create a Standard Email Plugin instance (SMTP host, sender, TLS), a Webhook Notification Plugin instance pointing at the ticketing system's incident API (the Rest Notification Plugin is deprecated in 9.0), and an SNMP Trap Plugin instance for the NOC trap receiver
- Notifications (Notifications tile): create one rule per plugin for "Host Memory Critical Alert", criticality Critical, scope the prod cluster; the webhook payload template carries host name, metric values and the alert ID so the ticket is created with the right fields
- Remediation: with the Management Pack for Orchestrator configured, the alert's recommendation shows Run Action for the "Rebalance-Host-Memory" workflow (a DRS-driven vMotion of the top memory consumer); keep it manual-run so a human approves each remediation
- Rate limiting: set the notification rule to notify on state change only, and give the workflow a cooldown check (do not run twice on the same host within 30 minutes)
Test the alert (inject memory pressure).
- Pick one lab host and power on enough memory-heavy test VMs to drive host memory above 90% with active swapping (in Holodeck, nested hosts reach this quickly)
- After one collection cycle plus the 10-minute sustain window the symptom triggers and the alert fires
- Confirm the three notifications: email received, ticket created by the webhook (note its ID), SNMP trap seen at the receiver
- Open the alert under Troubleshoot > Alerts, show the Action column and click Run on the remediation; approve it in the VCF Operations orchestrator client, where workflows waiting for user input are listed in the Recent Workflow Runs panel of the dashboard and in the top-right notification menu
Validate and close the loop.
- The workflow vMotions the top memory consumer away; host memory drops below 90%
- After two cancel cycles the alert clears and the clear notification is sent; the webhook rule for the cancelled state updates the ticket to resolved
- Check Administration > Control Panel > Recent Tasks for the action run and Activity > Workflow Runs in the orchestrator for the workflow outcome
- Record the end-to-end time from symptom trigger to ticket closure and document what would need to change for unattended remediation (rate limit, blast-radius guard, change-control exemption)
Validation Gate
Check: Verify lab completion
Expected: Lab exercise completed successfully
Common Errors
Final Validation
Lab completed successfully
✓ All steps completed → No errors observed
Cleanup / Restore
• Revert to snapshot if needed
Design Reflection (VCDX)
Alert automation with closed-loop remediation demonstrates AIOps capabilities in your design. VCDX panelists test whether auto-remediation is safe, controlled, and reversible.