Health Reporting and Monitoring for VMware Cloud Foundation
Provides operational insights across availability, health, and compliance for VCF via HTML reports (PowerShell Module for VCF Reporting) and custom dashboards/alerts/notifications in VMware Cloud Foundation Operations (formerly vRealize Operations) via a Python Module for VCF Health Monitoring. Deploys a host VM in the management domain hosting both modules, which collect data from the VCF Support & Serviceability (SoS) utility and SDDC component APIs.
Key Components: NSX, SDDC Manager, vCenter, ESXi, VCF Operations
External dependencies: Host virtual machine (Photon OS or Microsoft Windows Server)
34 design decisions. The Quality column is the Academy's assessment for defence practice; the VVS documents do not classify decisions by design quality (D133).
| DD-ID | Decision | Quality (Academy view) |
|---|---|---|
| HRM-VM-CFG-001 | Deploy the host virtual machine using a supported guest operating systems (VMware Photon OS or… | Manageability |
Decision: Deploy the host virtual machine using a supported guest operating systems (VMware Photon OS or Microsoft Windows Server). Rationale: A dedicated host virtual machine is deployed to ensure isolation of the PowerShell and Python modules from other production components. Implication: The host virtual machine must be deployed, configured, and maintained outside of VMware Cloud Foundation automated workflows. Component: VM | ||
| HRM-VM-CFG-002 | Deploy the host virtual machine in the default management vSphere cluster. | Availability |
Decision: Deploy the host virtual machine in the default management vSphere cluster. Rationale: Required to communicate with SDDC Manager and VMware Cloud Foundation Operations. Implication: The host virtual machine must be able to connect to SDDC Manager and VMware Cloud Foundation Operations. Component: VM | ||
| HRM-VM-CFG-003 | Protect the host virtual machine by using vSphere High Availability. | Availability |
Decision: Protect the host virtual machine by using vSphere High Availability. Rationale: Supports the availability objective without requiring manual intervention during an ESXi host failure. Implication: None. Component: VM | ||
| HRM-VM-CFG-004 | Place the host virtual machine in a designated virtual machine folder. | Manageability |
Decision: Place the host virtual machine in a designated virtual machine folder. Rationale: Provides organization of the appliances in the management domain vSphere inventory. Implication: You must create the virtual machine folder during deployment. Component: VM | ||
| HRM-VM-CFG-005 | When using two availability zones, add the host virtual machine to the VM group of the first… | Availability |
Decision: When using two availability zones, add the host virtual machine to the VM group of the first availability zone. Rationale: Ensures that the host virtual machine runs in the primary availability zone hosts group. Implication: After the implementation of the second availability zone for the management domain, you must update the VM group for the primary availability zone virtual machines to include the host virtual machine. Component: VM | ||
| HRM-VM-CFG-006 | In an environment with multiple VMware Cloud Foundation instances, deploy the host virtual machine… | Availability |
Decision: In an environment with multiple VMware Cloud Foundation instances, deploy the host virtual machine in the management vSphere cluster in the first VMware Cloud Foundation instance. Rationale: Required to communicate with SDDC Manager in each VMware Cloud Foundation instance and VMware Cloud Foundation Operations. Implication: The host virtual machine must be able to connect to SDDC Manager in each VMware Cloud Foundation instance and VMware Cloud Foundation Operations. Component: VM | ||
| HRM-PWSH-CFG-001 | Use or install a supported edition and version of PowerShell on the host virtual machine guest… | Manageability |
Decision: Use or install a supported edition and version of PowerShell on the host virtual machine guest operating system. Rationale: The PowerShell module cmdlets may fail when run on an edition and version of PowerShell that is not supported by the PowerShell module and its dependencies. Implication: None Component: PWSH | ||
| HRM-PWSH-CFG-002 | Install the PowerShell Module for VMware Cloud Foundation Reporting and its dependencies on the… | Manageability |
Decision: Install the PowerShell Module for VMware Cloud Foundation Reporting and its dependencies on the host virtual machine. Rationale: The PowerShell Module for VMware Cloud Foundation Reporting is required to generate HTML reports. Implication: None. Component: PWSH | ||
| HRM-PY-CFG-001 | Install Python 3.x on the host virtual machine. | Manageability |
Decision: Install Python 3.x on the host virtual machine. Rationale: Python 3 is required to run the Python script that pulls data from SDDC Manager and pushes it to VMware Cloud Foundation Operations. Implication: In an environment with multiple VMware Cloud Foundation instances, multiple copies of the Python module are installed, each corresponding to a VMware Cloud Foundation instance. Component: PY | ||
| HRM-PY-CFG-002 | Install the Nagini client, a Python binding package for VMware Cloud Foundation Operations, on the… | Manageability |
Decision: Install the Nagini client, a Python binding package for VMware Cloud Foundation Operations, on the host virtual machine. Rationale: The Nagini client enables the Python module to send data to VMware Cloud Foundation Operations. Implication: Manual installation and setup depends on the host virtual machine's operating system. Component: PY | ||
| HRM-PY-CFG-003 | On the host virtual machine, schedule daily runs of the Python module to collect health data from… | Manageability |
Decision: On the host virtual machine, schedule daily runs of the Python module to collect health data from SDDC Manager and send it to VMware Cloud Foundation Operations. Rationale: Automates gathering health data. Implication: Manual installation and setup depends on the host virtual machine's operating system. Component: PY | ||
| HRM-PY-CFG-004 | Configure the default log retention for logs, generated by the Python module, to 30 days. | PerformanceManageability |
Decision: Configure the default log retention for logs, generated by the Python module, to 30 days. Rationale: Automatic cleanup of logs generated when the senddata-to-vrops.py script saves capacity on the host virtual machine's local disk and ensures old data is removed. Implication: You must manually set the log retention period by configuring the log_retention_in_days setting in the env.json file. Component: PY | ||
| HRM-VM-NET-001 | Place the host virtual machine on the management VLAN of the management domain. | Manageability |
Decision: Place the host virtual machine on the management VLAN of the management domain. Rationale: Place the host virtual machine on the same network as SDDC Manager for direct communication. Implication: None Component: VM | ||
| HRM-VM-NET-002 | Allocate statically assigned IP address from the management VLAN to the host virtual machine. | Manageability |
Decision: Allocate statically assigned IP address from the management VLAN to the host virtual machine. Rationale: Using statically assigned IP addresses ensures stability of the deployment and simplifies maintenance and tracking. Implication: Requires precise IP address management. Component: VM | ||
| HRM-VM-NET-003 | Configure forward and reverse DNS records for the host virtual machine IP address. | Manageability |
Decision: Configure forward and reverse DNS records for the host virtual machine IP address. Rationale: Ensures the host virtual machine is accessible by using a fully qualified domain name instead of using IP address only. Implication: • You must provide a DNS record for the host virtual machine IP address. • Firewalls between the host virtual machine and the DNS servers must allow DNS traffic. Component: VM | ||
| HRM-VM-NET-004 | Configure DNS servers on the host virtual machine. | Manageability |
Decision: Configure DNS servers on the host virtual machine. Rationale: Ensures the host virtual machine has accurate name resolution. Implication: • DNS infrastructure services should be highly available in the environment. • Firewalls between the appliance and the DNS servers must allow DNS traffic. • You must provide two or more DNS servers unless a DNS geographic load balancing is active. Component: VM | ||
| HRM-VM-NET-005 | Configure NTP servers for the host virtual machine. | Manageability |
Decision: Configure NTP servers for the host virtual machine. Rationale: • Ensures that the host virtual machine has accurate time synchronization. • Assists in the prevention of time mismatch between the host virtual machine and any dependencies. Implication: NTP infrastructure services should be highly available in the environment. Firewalls between the host virtual machine and the NTP servers must allow NTP traffic. You must provide two or more NTP servers unless an NTP geographic load balancing is active. Component: VM | ||
| HRM-VM-LCM-001 | Manage the updates for the host virtual machine's guest operating system using your organization's… | Manageability |
Decision: Manage the updates for the host virtual machine's guest operating system using your organization's tools and processes. Rationale: Update the host virtual machine in accordance with your organizations processes and policies to ensure security and critical fixes are applied in a timely manner. Implication: The host virtual machine is not managed by SDDC Manager. Component: VM | ||
| HRM-LCM-001 | Manually update PowerShell Module for VMware Cloud Foundation Reporting when new versions are… | Manageability |
Decision: Manually update PowerShell Module for VMware Cloud Foundation Reporting when new versions are available. Rationale: Updating the PowerShell Module for VMware Cloud Foundation Reporting when new versions are released ensures the latest features and bug fixes are applied. Implication: None Component: HRM | ||
| HRM-LCM-002 | Manually update the Python Module for VMware Cloud Foundation Health Monitoring in VMware Cloud… | Manageability |
Decision: Manually update the Python Module for VMware Cloud Foundation Health Monitoring in VMware Cloud Foundation Operations. See the README.md in the GitHub repository. Rationale: Updating the Python module for VMware Cloud Foundation Health Monitoring in VMware Cloud Foundation Operations when new versions are released ensures the latest features and bug fixes are applied. Implication: None Component: HRM | ||
| HRM-SEC-001 | Limit the use of local accounts for interactive or API access and solution integration. | SecurityManageability |
Decision: Limit the use of local accounts for interactive or API access and solution integration. Rationale: Local accounts are not specific to user identity and do not offer complete auditing from an endpoint back to the user identity. Implication: You must define and manage service accounts, security groups, group membership, and security controls in Active Directory. Component: HRM | ||
| HRM-SEC-002 | Limit the scope and privileges for accounts used for interactive or API access and solution… | SecurityManageability |
Decision: Limit the scope and privileges for accounts used for interactive or API access and solution integration. Rationale: The principle of least privilege is a critical aspect of access management and must be part of a comprehensive defense-indepth security strategy. Implication: You must define and manage custom roles and security controls to limit the scope and privileges used for interactive access or solution integration. Component: HRM | ||
| HRM-SEC-003 | Assign an SDDC Manager role to a designated service account. | Security |
Decision: Assign an SDDC Manager role to a designated service account. Rationale: To provide least privilege access to SDDC Manager you assign the service account to a role. Implication: None. Component: HRM | ||
| HRM-SEC-004 | Assign a custom VMware Cloud Foundation Operations role to a designated service account. | Security |
Decision: Assign a custom VMware Cloud Foundation Operations role to a designated service account. Rationale: To provide least privilege access to VMware Cloud Foundation Operations you assign the service account to a custom role. Implication: You must maintain the custom role required for service account of your organization. Component: HRM | ||
| HRM-SEC-005 | If the SDDC Manager service account is changed, update the user credentials in the sddc_manager… | SecurityManageability |
Decision: If the SDDC Manager service account is changed, update the user credentials in the sddc_manager section of the env.json. Rationale: You must manually reestablish authentication to SDDC Manager after the service account is changed (including a password change) to ensure that the Python Module for VMware Cloud Foundation Health Monitoring has the correct credentials and access. Implication: You must update the user credentials manually. Component: HRM | ||
| HRM-SEC-006 | If the VMware Cloud Foundation Operations service account is changed, update the user credentials… | SecurityManageability |
Decision: If the VMware Cloud Foundation Operations service account is changed, update the user credentials in the vrops section of the env.json file. Rationale: You must manually reestablish authentication to VMware Cloud Foundation Operations after service account is changed (including a password change) to ensure that the Python Module for VMware Cloud Foundation Health Monitoring has the correct credentials and access. Implication: You must update the user credentials manually. Component: HRM | ||
| HRM-SEC-007 | Encrypt the passwords for SDDC Manager and VMware Cloud Foundation Operations service accounts by… | Security |
Decision: Encrypt the passwords for SDDC Manager and VMware Cloud Foundation Operations service accounts by running encryptpasswords.py Python script. Rationale: Password encryption enhances the security of the communication between the applications. Implication: You must manually run the Python script to encrypt the passwords. Component: HRM | ||
| HRM-PWSH-SEC-001 | Assign the ADMIN role to an Active Directory user account in each SDDC Manager instance for… | AvailabilitySecurityManageability |
Decision: Assign the ADMIN role to an Active Directory user account in each SDDC Manager instance for application-to-application communication between the PowerShell Module for VMware Cloud Foundation Reporting and SDDC Manager. Rationale: To generate reports by using the PowerShell Module for VMware Cloud Foundation Reporting, the service account requires the ADMIN role for least privilege access. Implication: You must maintain the life cycle and availability of the service account outside of the SDDC stack. Component: PWSH | ||
| HRM-PY-SEC-001 | Create a custom role in VMware Cloud Foundation Operations and assign it to an Active Directory… | AvailabilitySecurityManageability |
Decision: Create a custom role in VMware Cloud Foundation Operations and assign it to an Active Directory user account for application-to-application communication between the Python Module for VMware Cloud Foundation Health Monitoring and VMware Cloud Foundation Operations. Rationale: A custom role with least privileges is required to provide access to the REST API to push custom metrics to VMware Cloud Foundation Operations. Implication: • You must maintain the life cycle and availability of the service account outside of the SDDC stack. • You must maintain the synchronization and availability of the service account in Workspace ONE Access. Component: PY | ||
| HRM-PY-SEC-002 | Import the service account to the Everyone user group in VMware Cloud Foundation Operations. | Security |
Decision: Import the service account to the Everyone user group in VMware Cloud Foundation Operations. Rationale: The Everyone user group has no roles and scopes. You need to assign the scope and custom role to the service account. Implication: No restrictions to limit access in VMware Cloud Foundation Operations. Component: PY | ||
| HRM-PY-SEC-003 | Assign the scope of permissions to the custom role in VMware Cloud Foundation Operations. | Security |
Decision: Assign the scope of permissions to the custom role in VMware Cloud Foundation Operations. Rationale: Provide the limited permission to required adapter instances. Implication: • Limits access to objects to a custom role in VMware Cloud Foundation Operations. • This narrows the service account access to only NSX, vCenter, VMware Cloud Foundation, and vSAN adapter instance objects. Component: PY | ||
| HRM-VM-SEC-001 | Configure the local user password expiration policy for the host virtual machine. | Security |
Decision: Configure the local user password expiration policy for the host virtual machine. Rationale: You configure the local user password expiration policy for the host virtual machine to align with the requirements of your organization. Implication: You must manage the local user password expiration settings on the host virtual machine. Component: VM | ||
| HRM-VM-SEC-002 | Configure the local user password complexity policy for the host virtual machine. | Security |
Decision: Configure the local user password complexity policy for the host virtual machine. Rationale: You configure the local user password complexity policy for the host virtual machine to align with the requirements of your organization. Implication: You must manage the local user password complexity settings on the host virtual machine. Component: VM | ||
| HRM-VM-SEC-003 | Configure the local user account lockout policy for the host virtual machine. | Security |
Decision: Configure the local user account lockout policy for the host virtual machine. Rationale: You configure the local user account lockout policy for the host virtual machine to align with the requirements of your organization. Implication: You must manage the local user account lockout settings on the host virtual machine. Component: VM | ||
Prerequisites
- VCF version listed in Support Matrix
- Environment configured per Before You Apply This Guidance
- Health Reporting and Monitoring tab completed in VCF Planning and Preparation Workbook
- VCF instance healthy and fully operational
- DNS forward/reverse records
- AD DCs available and service accounts created
- VCF integrated with AD over LDAP (see IAM validated solution)
- VCF integrated with Intelligent Operations (VCF Operations deployed)
Implementation Procedure
PowerShell Implementation
- Install PowerShell modules (PowerCLI 13.2.1+, vSphere.SsoAdmin 1.3.9+, ImportExcel 7.8.5+, PowerVCF 2.4.0+, PowerValidatedSolutions 2.11.0+)
- Import-Module PowerValidatedSolutions; Test-PowerValidatedSolutionsPrereq
- Create folder structure (validatedSolutions, certificates, binaries, generatedJsons)
- Run Start-ValidatedSolutionMenu with the workbook
- From the main menu, enter 11. (HRM) Health Reporting and Monitoring
- Enter 01. Generate JSON Specification File
- Enter 02. Verify Prerequisites
- Enter 05. End-to-End Deployment
Note: automated PowerShell implementation supports only preparing the VCF instance; Host VM configuration and VCF Operations configuration must be done manually through the UI (VVS 1.0 p.130).
External Services / Integration Points
- Active Directory (AD)
- DNS
- NTP
Open Source Prerequisites
- Networking: Host VM connectivity to the SDDC Manager of each VCF instance and to VCF Operations
- PowerShell: supported guest OS and supported PowerShell version
- Python: Python 3.x and the required Python libraries installed
Implementation Options
| Method | Description |
|---|---|
| PowerShell automation | Prepares the VCF instance through the PowerValidatedSolutions menu |
| UI / component interfaces | Manual deployment and configuration, including the Host VM and VCF Operations |
Personas
| Persona | Responsibility | Component role or group |
|---|---|---|
| Site Resiliency Engineer (SRE) | Generate HTML health reports; perform Operations actions except user and cluster management | SDDC Manager: ADMIN; VCF Operations: PowerUser |
Day-2 Operations Tasks
Operational verification — PowerShell reporting
After installation, host VM patching or module updateOn the host VM run the PowerShell Module for VCF Reporting against SDDC Manager with the ADMIN service account (HRM-PWSH-SEC-001) and confirm an HTML report is generated; the host VM must reach SDDC Manager and VCF Operations (HRM-VM-CFG-002).
Operational verification — Python health monitoring
After installation or module updateRun the Python Module for VCF Health Monitoring and confirm the metrics appear in VCF Operations under the custom-role service account (HRM-PY-SEC-001/002); verify Python 3.x and the Nagini client are installed (HRM-PY-CFG-001/002).
Service-account credential changes
Whenever the SDDC Manager or VCF Operations service account changesUpdate the sddc_manager or vrops section of env.json (HRM-SEC-005/006) and re-run encrypt-passwords.py so the passwords are stored encrypted (HRM-SEC-007); re-run a report to confirm.
Host VM password policies
Per policyConfigure and review the local user password expiration, complexity and lockout policies on the host VM (HRM-VM-SEC-001..003); these are managed on the guest OS, not by SDDC Manager.
Host VM life cycle
Per organisation patch cyclePatch the Photon OS or Windows Server guest with the organisation tools (HRM-VM-LCM-001); the host VM is outside SDDC Manager life cycle; protect it with vSphere HA (HRM-VM-CFG-003) and, with two availability zones, keep it in the first-AZ VM group (HRM-VM-CFG-005).
Module updates
When new versions are releasedManually update the PowerShell Module for VCF Reporting (HRM-LCM-001) and the Python Module for VCF Health Monitoring per the GitHub README (HRM-LCM-002); re-run operational verification.
Certificate trust
When CA or component certificates changeImport the CA root certificate into the host VM trust store so the PowerShell and Python modules connect to SDDC Manager and VCF Operations without TLS errors.
Monitoring Points
- Verify host VM is online and accessible via FQDN
- Verify PowerShell module generates HTML reports against current VCF inventory
- Verify Python module daily run succeeds; metrics land in VCF Operations as custom metrics on corresponding objects
- Verify VCF Health dashboards populate correctly; drill into VCF Health Rollup
- Verify alerts/notifications trigger when thresholds exceeded
- MonitoringThis solution IS the monitoring; integrates natively with VCF Operations for dashboards/alerts/notifications
Troubleshooting
Likely Panelist Questions
Q: Why a dedicated host VM outside SDDC Manager?
HRM-VM-CFG-001/002: isolates the PowerShell and Python modules from production components and keeps them in the management cluster near SDDC Manager and VCF Operations; the trade-off is that the VM is deployed, patched and protected outside VCF automation (vSphere HA, organisation patching).
Q: How is least privilege applied?
HRM-SEC-001/002 limit local accounts and scope; the PowerShell service account needs ADMIN in SDDC Manager (HRM-PWSH-SEC-001), the Python service account a custom VCF Operations role (HRM-PY-SEC-001); credentials sit encrypted in env.json (HRM-SEC-007).
Q: Who patches what?
Guest OS by the organisation (HRM-VM-LCM-001); modules manually from GitHub (HRM-LCM-001/002); nothing here is in the SDDC Manager life cycle.
Failure Scenarios
Trade-off Analysis
One host VM for several VCF instances vs one per instance
Chosen: Single host VM with one Python module copy per instance (HRM-PY-CFG-001 implication)
Justification: Fewer VMs to patch; multiple module copies and credentials to maintain in env.json.
Service account with ADMIN vs lower role
Chosen: ADMIN in SDDC Manager (HRM-PWSH-SEC-001)
Justification: Reporting needs it; compensated by a dedicated AD account, encrypted credentials and host VM password policies.
Quiz — Health Monitoring
- ImportExcel
- PowerVCF
- Python Module for VCF Health Monitoring in VCF Operations
- PowerShell Module for VCF Reporting
- 1 vCPU, 1 GB RAM, 20 GB disk
- 2 vCPU, 4 GB RAM, 60 GB disk
- 1 vCPU, 2 GB RAM, 662 MB thin/16 GB thick
- 4 vCPU, 8 GB RAM, 100 GB disk
- VCF Compute Health
- VCF Hardware Compatibility
- VCF Version Health
- VCF Connectivity Health
- One per VCF instance
- One per vCenter
- Two — primary and standby
- One, always — it aggregates all instances
- HRM-VM-CFG-003 (HA)
- HRM-VM-NET-001 (mgmt VLAN)
- HRM-VM-CFG-005 (VM group for first AZ)
- HRM-VM-CFG-004 (VM folder)
- Compliance only
- Only Configuration report
- Only Health report
- System overview, Health, Alert, Configuration, Upgrade precheck
- 90
- 30
- 14
- 7
- Ansible
- PyVmomi
- Nagini
- PowerShell Core
- SDDC Manager default
- VMware Marketplace management pack
- Built into VCF Operations
- Predefined dashboards installed by the Python module
- VCF Connectivity Health
- VCF DNS Health
- VCF Compute Health
- VCF Networking Health
- SDDC Manager
- vCenter Server
- ESXi hosts
- External Active Directory
- Continuous real-time streaming
- On demand only
- Every 5 minutes
- Scheduled daily runs
- 04. (IAM)
- 11. (HRM)
- 12. (ILA)
- 05. (DRI)
- Manual (HRM-LCM-001)
- Handled by ImportExcel
- Automated by VCF Operations
- Automated by SDDC Manager
- Only Photon OS
- Any Linux
- Only Windows Server
- Photon OS or Microsoft Windows Server
Flashcards — Health Monitoring
Labs
Lab 1: Deploy Host VM and Install PowerShell Reporting Module
Deploy a Photon OS host VM in management domain, install PowerShell Module for VCF Reporting, generate a Health report.
Starting State: VCF 5.2 management domain operational; Photon OS OVA downloaded; AD service account with SDDC Manager ADMIN role; DNS/NTP configured.
- Deploy Photon OS OVA in default management vSphere cluster, 1 vCPU/2 GB/16 GB thick
- Configure static IP on mgmt VLAN, DNS, NTP, FQDN A/PTR records
- Protect with vSphere HA; in multi-AZ, add to first AZ VM group
- SSH to VM; install PowerShell 7.2+
- Install-Module PowerVCF, PowerValidatedSolutions, VMware.PowerCLI from PowerShell Gallery
- Connect-VCFManager -fqdn <sddc_mgr_fqdn> -username <ad_svc_acct>
- Invoke-VcfHealthReport -sddcManagerFqdn ... -sddcManagerUser ... -reportPath /reports -allDomains
- Review generated HTML health report for management + workload domains
Lab 2: Install Python Health Monitoring Module and Import VCF Health Dashboards to VCF Operations
Install Python module on host VM, configure env.json, schedule daily run, import dashboards into VCF Operations.
Starting State: Host VM from Lab 1; VCF Operations deployed and AD-integrated; AD service account holding a custom least-privilege VCF Operations role, scoped to the NSX, vCenter, VMware Cloud Foundation and vSAN adapter instances (HRM-PY-SEC-001 to -003).
- Install Python 3.x, pip, Nagini client and required libraries
- Clone/download Python Module for VCF Health Monitoring
- Configure env.json: sddc_manager_fqdn, sddc_manager_credentials, vrops_fqdn, vrops_credentials, log_retention_in_days=30
- Run send-data-to-vrops.py manually — verify custom metrics appear in VCF Operations (look up metric on vCenter object)
- Schedule daily cron job: 0 2 * * * python /path/send-data-to-vrops.py
- Import VCF Health dashboards into VCF Operations (VCF Health dashboard group)
- Verify VCF Health Rollup dashboard populates; drill into VCF Compute/Networking/Storage health
Lab 3: Configure Alerts and Notifications for VCF Certificate Expiry
Use VCF Health Certificate Health data to configure an alert definition that emails administrators 30 days before cert expiry.
Starting State: VCF Health dashboards installed; VCF Operations with SMTP configured.
- In VCF Operations, navigate to Alerts > Alert Definitions
- Create new alert definition: base object = vCenter Server; symptom = VCF Certificate Health custom metric (days_to_expiry) < 30
- Set impact, criticality, recommendation text
- Create a Notification Rule: alert definition = new cert expiry alert; email recipients = SRE distribution list
- Simulate alert by temporarily lowering threshold; verify email delivery
- Restore threshold; document alert as runbook item