To set the stage, a short overview of what this is all about. In vSphere you can, since vSphere 4, disable and enable alarm actions for all the managed entities. This option is available from the vSphere client

and from the vSphere Web client.

But how to automate these actions, and more importantly in this case, how to report on the active settings ? Like always PowerCLI to the rescue.

The script

function Set-AlarmActionState {
<#  
.SYNOPSIS  Enables or disables Alarm actions   
.DESCRIPTION The function will enable or disable
  alarm actions on a vSphere entity itself or recursively
  on the entity and all its children.
.NOTES  Author:  Luc Dekens  
.PARAMETER Entity
  The vSphere entity.
.PARAMETER Enabled
  Switch that indicates if the alarm actions should be
  enabled ($true) or disabled ($false)
.PARAMETER Recurse
  Switch that indicates if the action shall be taken on the
  entity alone or on the entity and all its children.
.EXAMPLE
  PS> Set-AlarmActionState -Entity $cluster -Enabled:$true
#>

  param(
    [CmdletBinding()]
    [Parameter(Mandatory=$true,ValueFromPipeline=$true)]
    [VMware.VimAutomation.ViCore.Impl.V1.Inventory.InventoryItemImpl]$Entity,
    [switch]$Enabled,
    [switch]$Recurse
  )

  begin{
    $alarmMgr = Get-View AlarmManager 
  }

  process{
    if($Recurse){
      $objects = @($Entity)
      $objects += Get-Inventory -Location $Entity
    }
    else{
      $objects = $Entity
    }
    $objects | %{
      $alarmMgr.EnableAlarmActions($_.Extensiondata.MoRef,$Enabled)
    }
  }
}

function Get-AlarmActionState {
<#  
.SYNOPSIS  Returns the state of Alarm actions.    
.DESCRIPTION The function will return the state of the
  alarm actions on a vSphere entity or on the the entity
  and all its children
.NOTES  Author:  Luc Dekens  
.PARAMETER Entity
  The vSphere entity.
.PARAMETER Recurse
  Switch that indicates if the state shall be reported for
  the entity alone or for the entity and all its children.
.EXAMPLE
  PS> Get-AlarmActionState -Entity $cluster -Recurse:$true
#>

  param(
    [CmdletBinding()]
    [Parameter(Mandatory=$true,ValueFromPipeline=$true)]
    [VMware.VimAutomation.ViCore.Impl.V1.Inventory.InventoryItemImpl]$Entity,
    [switch]$Recurse = $false
  )

  process {
    $Entity = Get-Inventory -Id $Entity.Id
    if($Recurse){
      $objects = @($Entity)
      $objects += Get-Inventory -Location $Entity
    }
    else{
      $objects = $Entity
    }

    $objects |
    Select Name,
    @{N="Type";E={$_.GetType().Name.Replace("Impl","").Replace("Wrapper","")}},
    @{N="Alarm actions enabled";E={$_.ExtensionData.alarmActionsEnabled}}
  }
}

Annotations

Line 23: We declare the Entity parameter as a base Inventory  object. That way you can pass all types of vSphere entities to the function.

Line 33-39,71-77: In the $objects variable the function collects all the entities involved. In case Recurse is True, all the child entities are obtained through the Get-Inventory cmdlet.

Line 41: The function calls the EnableAlarmActions method to change the alarm actions state of each entity. The value of the Enabled switch determines if the alarm actions will be enabled or disabled.

Line 70: This line makes sure we have the latest situation for the entity. Remember that the objects PowerCLI produces are not updated automatically when something changes in the entity.

Line 81: To make the Type property a bit more readable, the Impl and Wrapper suffixes are removed.

Sample usage

The Get-AlarmActionState function is quite easy to use. In the first example we will ask for the state of the alarms on a cluster

$cluster = Get-Cluster Cluster1
Get-AlarmActionState -Entity $cluster -Recurse:$false

The result is displayed on the console as follows.

The function can also be used in a pipeline construct. The following lines will do exactly the same as the previous example.

Get-Cluster Cluster1 |
Get-AlarmActionState -Recurse:$false

To change the state of the alarm actions, we can use the Set-AlarmActionState function. The following example will disable the alarm actions on Cluster1 and all its children. To confirm the change, the samepl calls the Get-AlarmActionState function to display the state.

$cluster = Get-Cluster Cluster1
$cluster | Set-AlarmActionState -Enabled:$false -Recurse:$true
$cluster | Get-AlarmActionState -Recurse:$true

In the console output we can see that the state of the alarm actions has effectively been changed. And this for the cluster and all its child entities.

Another vSphere feature we can now use in our automation scripts.
Enjoy !

And of course now I found in a matter of minutes what had cost me fruitless hours before.

While pondering the problem I realised that the acknowledgement couldn’t be stored in the Alarm itself. So I decided to have another look at the object on which the Alarm was defined.

The alarm I looked at was the infamous Non-VI workload detected on the datastore.

The acknowledgement info is stored under the DeclaredAlarmState property of the ManagedEntity object.

The following little script will look at all datastores and list those where the alarm has been acknowledged.

foreach($ds in Get-Datastore){
	$ds.ExtensionData.DeclaredAlarmState | where {$_.Acknowledged} | `
		Select @{N="Datastore";E={$ds.Name}},
			@{N="Alarm";E={(Get-View -Id $_.Alarm).Info.Name}},
			@{N="Acknowledged By";E={$_.AcknowledgedByUser}},
			@{N="Time";E={$_.AcknowledgedTime}}
}

This produces output like this (provided you acknowledged the alarm on some datastores of course).

The example script only showed how to retrieve acknowledgements for datastores, but since the DeclaredAlarmState property
belongs to the ManagedEntity object, you can use this logic for any vSphere object.
Thanks to Eric and William for making me have a 2nd look at this.

There is no easy way to move an alarms from one entity to another entity. No drag-and-drop in the vSphere Client, no Move-Alarm cmdlet in PowerCLI.

A practical example, you have developed this fantastic new alarm and for testing purposes you had defined it on a single virtual machine. Now the tests are done and your alarm is ready for production. But there is apparently no easy way to move your new alarm to the root of your vSphere environment.

Luckily we have PowerCLI and we can easily create a function to fill this gap.

The script

The function is called Move-Alarm and has the following syntax:

Move-Alarm -Alarm <Alarm> -From <ViContainer> -To <ViContainer[]> -DeleteOriginal

–Alarm: the MoRef of the Alarm object you want to move
–From: the ViContainer from where the alarm should be moved
–To: one or more ViContainer objects to which the alarm should be moved
–DeleteOriginal: a switch that determines what happens with the original alarm. Default is $false

Update 23/02/10

While using the script I discovered that the length of the name of an alarm can not exceed 80 characters. This seems to be an undocumented “feature”.

A second problem in the original script was that when you “move” to more than one destination, there would be an error message on the second “move”. That was because you can not have two alarms with the same name in your vCenter. The solution I used was to add the name of the destination entity as a suffix to the name of the new alarm.

The third problem was the Status property in each EventAlarmExpression object. Since the script copies the AlarmInfo object from the “from” alarm to the AlarmSpec object for the “to” alarm, the Status property has to blanked out explicitly.

Set-Variable -Name alarmLength -Value 80 -Option "constant"

$from = Get-Folder -Name "Datacenters" | Get-View
$to1 = Get-Folder -Name "Folder1" | Get-View
$to2 = Get-Folder -Name "Folder2" | Get-View

function Move-Alarm{
	param($Alarm, $From, $To, [switch]$DeleteOriginal = $false)

	$alarmObj = Get-View $Alarm
	$alarmMgr = Get-View AlarmManager

	if($deleteOriginal){
		$alarmObj.RemoveAlarm()
	}
	else{
		$updateAlarm = New-Object VMware.Vim.AlarmSpec
		$updateAlarm = $alarmObj.Info
		$oldName = $alarmObj.Info.Name
		$oldState = $alarmObj.Info.Enabled
		$oldDescription = $alarmObj.Info.Description
		$suffix = " (moved to " + ([string]($to | %{$_.Name + ","})).TrimEnd(",") + ")"
		if(($oldName.Length + $suffix.Length) -gt $alarmLength){
			$newName = $oldName.Substring(0, $alarmLength - $suffix.Length) + $suffix
		}
		else{
			$newName = $oldName + $suffix
		}
		$updateAlarm.Name =  $newName
		$updateAlarm.Enabled = $false
		$updateAlarm.Description += ("`rOriginal name: " + $oldName)
		$updateAlarm.Expression.Expression | %{
			if($_.GetType().Name -eq "EventAlarmExpression"){
				$_.Status = $null
				$needsChange = $true
			}
		}

		$alarmObj.ReconfigureAlarm($updateAlarm)

		$alarmObj.Info.Name = $oldName
		$alarmObj.Info.Enabled = $oldState
		$alarmObj.Info.Description = $oldDescription
	}

	$newAlarm = New-Object VMware.Vim.AlarmSpec
	$newAlarm = $alarmObj.Info

	$oldName = $alarmObj.Info.Name
	$oldDescription = $alarmObj.Info.Description

	foreach($destination in $To){
		if($To.Count -gt 1){
			$suffix = " (" + $destination.Name + ")"
			if(($oldName.Length + $suffix.Length) -gt $alarmLength){
				$newName = $oldName.Substring(0, $alarmLength - $suffix.Length) + $suffix
			}
			else{
				$newName = $oldName + $suffix
			}
			$newAlarm.Name = $newName
			$newAlarm.Description += ("`rOriginal name: " + $oldName)
		}
		$newAlarm.Expression.Expression | %{
			if($_.GetType().Name -eq "EventAlarmExpression"){
				$_.Status = $null
				$needsChange = $true
			}
		}

		$alarmMgr.CreateAlarm($destination.MoRef,$newAlarm)
		$newAlarm.Name = $oldName
		$newAlarm.Description = $oldDescription
	}
}

$alarmMgr = Get-View AlarmManager

$alarms = $alarmMgr.GetAlarm($from.MoRef)
$alarms | % {
	Move-Alarm -Alarm $_ -From (Get-View $_) -To $to1,$to2 -DeleteOriginal:$false
}

Annotations

Line 1: The maximum length of the name of an Alarm defined as a constant.

Line 3-5: Variables that will be used later on in the script as parameters to the function. In the sample I’m moving all alarms from the vCenter root (hidden entity “Datacenters”) to two folders called “Folder1” and “Folder2”

Line 8: Note that the function doesn’t use strongly types parameters for the -From and -To parameters. This is because the ViObjects on which alarms can be defined can be of several types (Datacenter, Folder, HostSystem…). This also allows to pass an array of ViObjects for the -To parameter.

Line 14: If the -DeleteOriginal switch is set to $true, the original alarm will be deleted.

Line 17-35: If the -DeleteOriginal switch is set to $false the original Alarm will be kept but it will be renamed since there can not be multiple alarms with the same name.

Line 24,48: If the length of name with the suffix exceeds 80 characters, the script will remove sufficient characters from the original alarmname to be able to add the suffix.

Line 29: If the original alarm is kept it will be renamed. The original will get a suffix “(moved to <dest>)”

Line 30: If the original alarm is kept it will be “disabled”.

Line 31,54: The script adds the original alarmname to the Description field.

Line 32-37,64-69: Since the script copies the AlarmSpec object from the original alarm, the Status field for all the EventAlarmExpression objects needs to blanked out.

Line 52: The -To parameter is handled in a foreach loop since the function allows more than one destination. The function can be used as a 1-to-n move in this case.

Line 80-81: A sample use case of the Move-Alarm function. All Alarms defined in the root of the vCenter will be “moved” to “Folder1” and “Folder2”. The original Alarm will not be deleted but it will be renamed and disabled.

Samples

Before the run of the above sample use of the script, the alarms defined on the vCenter root look like this.

After the first call of the Move-Alarm function the first alarm has been changed and disabled.

And on both target folders the alarm has been created as this screenshot from the alarms on Folder1 shows.

This is a rather simple function but it can be quite useful at times. I hope it can be of use to some of you.

The article shows a very elegant solution how to move your guests to “safer havens”, the moment one of the hosts in the cluster starts experiencing hardware problems.
The elegance of Cody’s solution is that he uses maintenance mode to force vMotion on all the powered-on guests on the host that experiences HW problems.

The condition for the solution to work is of course that the host needs to be a node in a DRS-enabled cluster where the Automation Level is set to Fully automated.

Since I thought that this was a good example of some of the alarm-related posts I did in the past, I created a script so you can automate the creation of Cody’s alarm.

$dcName = "MyDatacenter"
$mailto = "lucd@lucd.info"

$alarmMgr = Get-View AlarmManager
$entity = Get-Datacenter $dcName | Get-View

# AlarmSpec
$alarm = New-Object VMware.Vim.AlarmSpec
$alarm.Name = "Generic Host Health Alert"
$alarm.Description = "When host reports HW health, place host in maintenance mode"
$alarm.Enabled = $TRUE

$alarm.action = New-Object VMware.Vim.GroupAlarmAction

# Action 1 - Send email
$trigger1 = New-Object VMware.Vim.AlarmTriggeringAction
$trigger1.action = New-Object VMware.Vim.SendEmailAction
$trigger1.action.ToList = $mailTo
$trigger1.action.Subject = "HW Health - Set host in maintenance mode"
$trigger1.Action.CcList = ""
$trigger1.Action.Body = ""

# Action 2 - Enter maintenance mode
$trigger2 = New-Object VMware.Vim.AlarmTriggeringAction
$trigger2.action = New-Object VMware.Vim.MethodAction
$trigger2.action.Name = "EnterMaintenanceMode_Task"

# Action 2 - Arguments
$arg1 = New-Object VMware.Vim.MethodActionArgument
# timeout
$arg1.value = 0
# evacuatePoweredOffVms
$arg2 = New-Object VMware.Vim.MethodActionArgument
$arg2.value = $false

$trigger2.action.argument += $arg1
$trigger2.action.argument += $arg2

# Transition - yellow --> red
$trans = New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec
$trans.StartState = "yellow"
$trans.FinalState = "red"

$trigger1.TransitionSpecs += $trans
$trigger2.TransitionSpecs += $trans

$alarm.action.action += $trigger1
$alarm.action.action += $trigger2

# Expression - Hardware Health Changed
$expression = New-Object VMware.Vim.EventAlarmExpression
$expression.EventType = "EventEx"
$expression.eventTypeId = "com.vmware.vc.cim.CIMGroupHealthStateChanged"
$expression.objectType = "HostSystem"
$expression.status = "red"

$alarm.expression = New-Object VMware.Vim.OrAlarmExpression
$alarm.expression.expression += $expression

$alarm.setting = New-Object VMware.Vim.AlarmSetting
$alarm.setting.reportingFrequency = 0
$alarm.setting.toleranceRange = 0

# Create alarm.
$alarmMgr.CreateAlarm($entity.MoRef, $alarm)

Annotations

Line 1: The script creates the alarm on a datacenter but you can easily replace that by any object that supports alarm definitions.

Line 26: When the alarm is fired this action will make a call to the EnterMaintenanceMode_Task.

Line 29-35: The EnterMaintenanceMode_Task method requires two arguments. For the first argument, called timeout, we specify 0 (zero) which indicates there is no timeout. The second argument, called evacuatePoweredOffVms, is a Boolean value that indicates if powered-off guests should also be migrated. We pass $false, meaning that the powered-off guest shall not be migrated.

Line 52-56: See my earlier post, called Alarm expressions – Part 2 : Event alarms, for more information on how to list the available events.

The result of the script is a new alarm identical to the one Cody created in the vSphere client.

With this script you can now include the creation of this alarm in your automation process.

And the script has also shown that it is more flexible than the vSphere client. Try for example changing the parameters of the method (in this case EnterMaintenanceMode_Task) you use in the Actions.

The current PowerCLI build (version 4 update 1 – build 208462) unfortunately has no cmdlets to work with alarms. There are some alarm-related cmdlets available in the VI Toolkit for Windows Community Extensions. But none of these provides the functionality Charlie wanted to have.

To start with alarm action maintenance, you should be able to report on what is currently defined in the vCenter.

As a reminder, the following types of alarm actions are available in the current vSphere version.

1. Create a task
2. Execute a “method” on a specific entity
3. Run a script
4. Send an email
5. Send an SNMP trap

An alarm can have one or more of the above defined.

Inventory

The following script produces a CLIXML file containing all the existing alarms and for each of them a selection of their properties, including the action(s) that are defined.

Why CLIXML and not to a plain XML or CSV file ? The answer is simple: “structure“.

While a CSV file is handy to export object that all have the same layout, it’s quite cumbersome to export objects that have varying content. Same is valid for the ConvertTo-Xml cmdlet that came with PowerShell v2.

Since an alarm can have one or more actions of different types (see above), the easiest serialisation solution was CLIXML and to use the built-in Export-CliXML cmdlet.

One of the disadvantages of the CLIXML format; it’s not obvious for a user to read this type of XML file !

function Get-AlarmAction{
	param($action)

	$list = @()
	if($action -ne $null){
		switch($action.GetType().Name){
			"GroupAlarmAction"{
				$action.action | % {
					$_.Action | % {
						$list += Get-AlarmAction $_
					}
				}
			}
			"CreateTaskAction"{
				$list += New-Object PSObject -Property @{
					Type = $_
					TaskTypeId = $action.taskTypeId
					Cancelable = %action.Cancelable
				}
			}
			"MethodAction"{
				$list += New-Object PSObject -Property @{
					Type = $_
					Argument = $action.argument
					Name = $action.name
				}
			}
			"RunScriptAction"{
				$list += New-Object PSObject -Property @{
					Type = $_
					Script = $action.script
				}
			}
			"SendEmailAction"{
				$list += New-Object PSObject -Property @{
					Type = $_
					Body = $action.taskTypeId
					ccList = $action.Cancelable
					Subject = $action.Subject
					toList = $action.toList
				}
			}
			"SendSNMPAction"{
				$list += New-Object PSObject -Property @{
					Type = $_
				}
			}
		}
	}
	$list
}

$alarmMgr = Get-View AlarmManager

$alarms = $alarmMgr.GetAlarm($null)
$report = @()
$alarms | % {
	$alarm = Get-View $_
	$report += New-Object PSObject -Property @{
		Name = $alarm.Info.Name
		Description = $alarm.Info.Description
		Entity = (Get-View $alarm.Info.Entity).Name
		Enabled = $alarm.Info.Enabled
		Action = Get-AlarmAction $alarm.Info.Action
	}
}

$report | Export-Clixml "C:\Alarm-actions.xml" -NoClobber

Annotations

Line 1: This function is called recursively (for the GroupAlarmAction). All actions will be returned in an array of PSObjects.

Line 6: This switch handles all possibilities that can be found in the Action property.

Line 7-13: A GroupAlarmAction can contain one or more of the other “basic” actions. This case loops through each of them and calls the Get-AlarmAction function recursively.

Line 14-47: Handles each of the “base” actions and stores the specific properties for each of them in a PSObject. Note that the New-Object format that is used here is only valid in PowerShell v2 RTM.

Line 55: If the GetAlarm method is called with the $null parameter it will return all alarms defined in the vCenter.

Line 68: The report with all the alarms is saved to an XML file. The -NoClobber parameter makes sure n existing file is not overwritten.

As you can notice from the following extract, the content of the file is rather hard to read if you’re not used to the CLIXML format..

The file was opened with XMLPad in the Tabular view.

Quick report

To get a quick listing of the file contents you can execute the following line

Import-Clixml "C:\Alarm-actions.xml"

This will produce a listing in the console window looking like this.

Which is definitely more legible than the CLIXML file itself.

Add an action

The following script will allow you to add an action on a selected set of alarms.

# Alarms to be changed
$tgtAlarms = "Test1","Test2"
# $tgtAlarms = "Test2"

#=================================
# Uncomment one of the following actions
# Fill in the required parameters
#=================================
# New createtask action
# $actType = "CreateTaskAction"
# $actTaskTypeId =
# $actCancelable =
#-- -------------------------------
# New method action
# $actType = "MethodAction"
# $actArgument =
# $actName =
#---------------------------------
# New runscript action
# $actType = "RunScriptAction"
# $actScript =
#---------------------------------
# New sendmail action
$actType = "SendEmailAction"
$actTo = "luc@lucd.info"
$actCc = ""
$actSubject = "Alarm email"
$actBody = ""
#---------------------------------
# New sendSNMP action
# $actType = "SendSNMPAction"
#=================================

function Set-AlarmAction{
	param($alarm)

	switch($actType){
		"CreateTaskAction"{
			$trigger = New-Object VMware.Vim.AlarmTriggeringAction
			$trigger.action = New-Object VMware.Vim.CreateTaskAction
			$trigger.action.Cancelable = $actCancelable
			$trigger.action.taskTypeId = $actTaskTypeId
		}
		"MethodAction"{
			$trigger = New-Object VMware.Vim.AlarmTriggeringAction
			$trigger.action = New-Object VMware.Vim.MethodAction
			$trigger.action.Argument = $actArgument
			$trigger.action.Name = $actName
		}
		"RunScriptAction"{
			$trigger = New-Object VMware.Vim.AlarmTriggeringAction
			$trigger.action = New-Object VMware.Vim.RunScriptAction
			$trigger.action.Script = $actScript
		}
		"SendEmailAction"{
			$trigger = New-Object VMware.Vim.AlarmTriggeringAction
			$trigger.action = New-Object VMware.Vim.SendEmailAction
			$trigger.action.ToList = $actTo
			$trigger.action.Subject = $actSubject
			$trigger.Action.CcList = $actCc
			$trigger.Action.Body = $actBody
		}
		"SendSNMPAction"{
			$trigger = New-Object VMware.Vim.AlarmTriggeringAction
			$trigger.action = New-Object VMware.Vim.SendSNMPAction
		}
	}
# Transition a - yellow --> red
	$transa = New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec
	$transa.StartState = "green"
	$transa.FinalState = "yellow"
# Transition b - red --> yellow
	$transb = New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec
	$transb.StartState = "yellow"
	$transb.FinalState = "red"
	$trigger.TransitionSpecs += $transa
	$trigger.TransitionSpecs += $transb

	if($alarm.Info.Action -eq $null){
		$action = New-Object VMware.Vim.GroupAlarmAction
	}
	else{
		if($alarm.Info.Action.GetType().Name -ne "GroupAlarmAction"){
			$action = New-Object VMware.Vim.GroupAlarmAction
			$action.action += $alarm.Info.Action
		}
		else{
			$action = $alarm.Info.Action
		}
	}
	$action.action += $trigger

	$spec = New-Object VMware.Vim.AlarmSpec
	$spec.Action = $action
	$spec.actionFrequency = $alarm.Info.ActionFrequency
	$spec.Description = $alarm.Info.Description
	$spec.Enabled = $alarm.Info.Enabled
	$spec.Expression = $alarm.Info.Expression
	$spec.Name = $alarm.Info.Name
	$spec.Setting = $alarm.Info.Setting

	$alarm.ReconfigureAlarm($spec)
}

# Get all alarms
$alarmMgr = Get-View AlarmManager
$alarms = $alarmMgr.GetAlarm($null)

# For all selected alarms
$alarms | % {
	$alarm = Get-View $_
	if($tgtAlarms -contains $alarm.Info.Name){
		Set-AlarmAction $alarm
	}
}

Annotations

Line 2: The variable $tgtAlarms is an array of strings where each element is the name of an alarm where you want to add an action.

Line 5-32: Uncomment one of the actions. This will be the action that is added to the alarms defined in $tgtAlarms.

Line 37-67: In this switch construction the new action is set up.

Line 68-77: The new action requires one or more triggers, in other words when will the action be executed. In the script I defined the triggers to be fired when the alarm goes from green to yellow and when the alarm goes from yellow to red. If you require other triggers change the contents of the AlarmTriggeringAction object.

Line 79-91: These lines make sure the new action is added in the correct place. Note that the script uses the GroupAlarmAction to combine all the actions. This to avoid the problem I mentioned in Alarm expressions – Part 1 : Metric alarms where your alarm settings are not editable anymore in the vSphere client.

Line 93-100: The AlarmSpec object is set up. Except for the Action property all properties are copied from the existing alarm.

Line 102: The ReconfigureAlarm method is used to add the new action.

Line 105-107: Get all the defined alarms.

Line 110-115: A simple loop to find all the alarms that were specified in the $tgtAlarms variable. For each of the specified alarms the Set-AlarmAction is called.

After the script is executed you will see the new action on the selected alarms.

Notice the triggers in the four columns on the right.

In this part I will show you how to create alarms when one or more specific events occur in your vSphere environment. More specifically I will show you how to create an alarm that will fire when someone adds or removes a license from your vCenter.

Warning: this function will not work in vCenter 4.x when you are using the builtin licensing instead of the separate License Manager. I’ll post an update soon.

The first step for creating alarms based on this type of triggers is to define which event(s) to use. As I showed in Events, Dear Boy, Events – Part 2, there are an enormous number of possible events to chose from.

With the CSV file from that blog entry it’s easy to find out which events, related to licensing, are available. I used the following script for this

$targetStr = "license"
Import-Csv "C:\Events.csv" -UseCulture | where {$_.Name -match $targetStr -or $_.Description -match $targetStr}

From the output it is clear that I will have to use events from the EventEx type.

In the EventAlarmExpression description it says that one needs to pass an eventTypeId. Unfortunately the Reference Guide doesn’t explain how to get such an eventTypeId. After some trial and error I discovered you can get that property via the Extensionmanager. The following short script list all the extensions registered in your vSphere environment and shows their eventTypeIds.

$extMgr = Get-View ExtensionManager
$extMgr.ExtensionList | %{
if($_.EventList -ne $null){
$_.Key
$_.EventList | %{
"`t" + $_.EventId
}
}
}

The output of this script shows what I need.

The following script puts it all together and creates the alarm.

$dcName = "MyDatacenter"
$mailto = "luc.dekens@lucd.info"

$alarmMgr = Get-View AlarmManager
$entity = Get-Datacenter $dcName | Get-View

# AlarmSpec
$alarm = New-Object VMware.Vim.AlarmSpec
$alarm.Name = "Add/remove license"
$alarm.Description = "Add or remove a license"
$alarm.Enabled = $TRUE

# Action - Send email
$alarm.action = New-Object VMware.Vim.GroupAlarmAction

$trigger1 = New-Object VMware.Vim.AlarmTriggeringAction
$trigger1.action = New-Object VMware.Vim.SendEmailAction
$trigger1.action.ToList = $mailTo
$trigger1.action.Subject = "License added/removed"
$trigger1.Action.CcList = ""
$trigger1.Action.Body = ""

# Transition 1a - yellow --> red
$trans1a = New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec
$trans1a.StartState = "yellow"
$trans1a.FinalState = "red"

# Transition 1b - red --> yellow
$trans1b = New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec
$trans1b.StartState = "red"
$trans1b.FinalState = "yellow"

$trigger1.TransitionSpecs += $trans1a
$trigger1.TransitionSpecs += $trans1b

$alarm.action.action += $trigger1

# Expression 1 - License added
$expression1 = New-Object VMware.Vim.EventAlarmExpression
$expression1.EventType = $null
$expression1.eventTypeId = "com.vmware.license.AddLicenseEvent"
$expression1.objectType = "Datacenter"
$expression1.status = "yellow"

# Expression 2 - License removed
$expression2 = New-Object VMware.Vim.EventAlarmExpression
$expression2.EventType = $null
$expression2.eventTypeId = "com.vmware.license.RemoveLicenseEvent"
$expression2.objectType = "Datacenter"
$expression2.status = "yellow"

$alarm.expression = New-Object VMware.Vim.OrAlarmExpression
$alarm.expression.expression += $expression1
$alarm.expression.expression += $expression2

$alarm.setting = New-Object VMware.Vim.AlarmSetting
$alarm.setting.reportingFrequency = 0
$alarm.setting.toleranceRange = 0

# Create alarm.
$alarmMgr.CreateAlarm($entity.MoRef, $alarm)

Annotations:

Line 40 & 47: although the SDK Reference says that the eventTypeId property replaces the EventType property you have to explicitly set the EventType property to $null. Otherwise the CreateAlarm method will fail

You can now automate the creation of your event-driven alarms.

This blog entry shows how you can quickly get a list of permitted counterIds (and instances) for a specific entity. And it will show how to create some “impossible” alarms !

In the vSphere environment all things “performance” are managed by the PerformanceManager. In the PerformanceManager object the perfCounter property contains a list of all the performance counters the system supports.

But not every entity supports every performance counter. That is why I will use the QueryAvailablePerfMetric method to obtain the list of metrics for a specific entity.

The following script will produce a CSV file with the available metrics per entity type (cluster, host, virtual machine, datastore…) per interval.

Update September 8th 2010: This is a complete rewrite of the “Metrics” script. There was in fact no need to query the metrics for each historical interval, it was sufficient to query the metrics for “Current” (realtime) and “Summary” (historical). The new script also contains some tests which will avoid error messages when a specific entity is not present on the vCenter.

The metrics

function Get-PerfCounterList{
	param($pm)

	$list = @{}
	$pm.PerfCounter | % {
		$obj = "" | Select Name, Level, Summary
		$obj.Name = $_.GroupInfo.key + "." + $_.NameInfo.key + "." + $pm.Description.CounterType[$_.RollupType].Key

		$obj.Level = $_.Level
		$obj.Summary = $_.NameInfo.Summary
		$list[$_.Key] = $obj
	}
	$list
}

function Get-StatId{
param($entity, $pm, $perfCounterList)

$result = @()

$perfProvider = $pm.QueryPerfProviderSummary($entity.MoRef)

if($perfProvider.SummarySupported){
	$perfMetrics = $pm.QueryAvailablePerfMetric($entity.MoRef,$null,$null,$null)
	if($perfMetrics){
		foreach($metric in $perfMetrics){
			$row = "" | Select Entity, Interval, CounterId, Stat, Instance, Level, Summary
			$row.Entity = $entity.GetType().Name
			$row.Interval = "Aggregate"
			$row.CounterId = $metric.CounterId
			$row.Stat = $perfCounterList[$metric.CounterId].Name
			$row.Instance = $metric.Instance
			$row.Level = $perfCounterList[$metric.CounterId].Level
			$row.Summary = $perfCounterList[$metric.CounterId].Summary
			$result += $row
		}
	}
}

if($perfProvider.CurrentSupported){
	$perfMetrics = $pm.QueryAvailablePerfMetric($entity.MoRef,$null,$null,$perfProvider.refreshRate)
	foreach($metric in $perfMetrics){
		$row = "" | Select Entity, Interval, CounterId, Stat, Instance, Level, Summary
		$row.Entity = $entity.GetType().Name
		$row.Interval = "RealTime"
		$row.CounterId = $metric.CounterId
		$row.Stat = $perfCounterList[$metric.CounterId].Name
		$row.Instance = $metric.Instance
		$row.Level = $perfCounterList[$metric.CounterId].Level
		$row.Summary = $perfCounterList[$metric.CounterId].Summary
		$result += $row
	}
}
$result
}

$pm = Get-View (Get-View ServiceINstance).Content.PerfManager
$perfCounterList = Get-PerfCounterList $pm

$report = @()

# Datacenter
$dc = Get-Datacenter | Select-Object -First 1 | Get-View
$report += (Get-StatId $dc $pm $perfCounterList)

# Datastore
$ds = Get-Datastore | Select-Object -First 1 | Get-View
if($ds){
	$report += (Get-StatId $ds $pm $perfCounterList)
}

# VirtualMachine
$vm = Get-VM | where{$_.PowerState -eq 'PoweredOn'} | Select-Object -First 1 | Get-View
if($vm){
	$report += (Get-StatId $vm $pm $perfCounterList)
}

# HostSystem
$vmhost = Get-VmHost | where{$_.State -eq 'Connected'} | Select-Object -First 1 | Get-View
if($vmhost){
	$report += (Get-StatId $vmhost $pm $perfCounterList)
}

# ClusterComputeResource
$cluster = Get-Cluster | Select-Object -First 1 | Get-View
if($cluster){
	$report += (Get-StatId $cluster $pm $perfCounterList)
}

# ResourcePool
$respool = Get-ResourcePool | Select-Object -First 1 | Get-View
$report += (Get-StatId $respool $pm $perfCounterList)

$report | Export-Csv "C:\Stat-Ids.csv" -NoTypeInformation -UseCulture

Annotations

Line 1-13: in this function I create a hash table where the key is performance counter Id and the value is an object that contains specific information about the metric: the composite name (group-name-rollup), the statistics level where it is available and the summary description of the metric

Line 21: In the PerfProvider for an entity we can find out if the entity has “realtime” and/or “aggregate” metrics

Line 23-38: this part of the function retrieves the available “aggregate” metrics if they exist

Line 24: the QueryAvailablePerfMetric method is where the script finds out which metrics are available for a specific entity.

Line 40-53: this part of the function retrieves the available “realtime”metrics if they exist

Line 62-92: the metrics are collected for all entities that have performance metrics. You can leave out specific entities if you don’t want those metrics in the list.

Line 73,79: a powered-off guest or a disconnected host will not return all the possible metrics. By adding a where-clause this problem is avoided.

This resulting list makes it easy to define MetricAlarmExpression objects. In fact you can now create alarms that you can not create via the vSphere client !

As an example, suppose you want to be alerted when one of your ESX servers is receiving network traffic over a specific watermark.

In the vSphere client you can create an alarm for Network Usage, but that will be the total of incoming and outgoing network traffic. And here I want to create an alarm for incoming network traffic only.

From the CSV file it’s simple to find that the script will need to use performance counter Id 102.

Note that the instances you see in this CSV file are not  necessarily all the possible instances you can encounter in your vSphere environment.

For example there could be a virtual machine with 10 hard disks connected. If that guest was not used to compile the available metrics for the VirtualMachine object, you would not see for example all the available instances in the virtualdisk group !

In the screenshot you can see that the script used a guest with 2 virtual hard disks to compile the CSV.

The alarm

The following script will create the alarm.

$esxName = "esx1.lucd.info"
$mailTo = "luc.dekens@lucd.info"

$alarmMgr = Get-View AlarmManager
$entity = Get-VmHost $esxName | Get-View

# AlarmSpec
$alarm = New-Object VMware.Vim.AlarmSpec
$alarm.Name = "Net received rate"
$alarm.Description = "Testing network related alarms"
$alarm.Enabled = $TRUE

# Action - Send email
$alarm.action = New-Object VMware.Vim.GroupAlarmAction

$trigger = New-Object VMware.Vim.AlarmTriggeringAction
$trigger.action = New-Object VMware.Vim.SendEmailAction
$trigger.action.ToList = $mailTo
$trigger.action.Subject = "Net received alarm"
$trigger.Action.CcList = ""
$trigger.Action.Body = ""

# Transition a - yellow --> red
$transa = New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec
$transa.StartState = "yellow"
$transa.FinalState = "red"

# Transition b - red --> yellow
$transb = New-Object VMware.Vim.AlarmTriggeringActionTransitionSpec
$transb.StartState = "red"
$transb.FinalState = "yellow"

$trigger.TransitionSpecs += $transa
$trigger.TransitionSpecs += $transb

$alarm.action.action += $trigger

# Expression - Network data receive rate
$expression = New-Object VMware.Vim.MetricAlarmExpression
$expression.Metric = New-Object VMware.Vim.PerfMetricId
$expression.Metric.CounterId = 102
$expression.Metric.Instance = ""
$expression.Operator = "isAbove"
$expression.Red = 300
$expression.Yellow = 150
$expression.Type = "HostSystem"

$alarm.expression = New-Object VMware.Vim.OrAlarmExpression
$alarm.expression.expression += $expression

$alarm.setting = New-Object VMware.Vim.AlarmSetting
$alarm.setting.reportingFrequency = 0
$alarm.setting.toleranceRange = 0

# Create alarm.
$alarmMgr.CreateAlarm($entity.MoRef, $alarm)

Annotations

Line 44-45: the watermarks are defined in KBps. The values in the script are values for demonstration purposes only. Correct these for your own environment.

In the vSPhere client you will see the following warning when you edit the settings of the alarm and select the Triggers tab.

But rest assured, this alarm works and will fire when the metric goes over the yellow and red watermarks.