Events – Part 3 : Auditing VM device changes
In a comment on my Events, Dear Boy, Events – Part 2 post, Sham was wondering if it was possible to find out which device was actually changed on a guest.
Sham was able to find out who did a change, at what time and on which machine, but he also wanted to know what exactly was changed.
Since this is the kind of audit information that I would like to have as well I had a closer look. And luckily the VmReconfiguredEvent object had all the required information in the configSpec property.
The configSpec property contains a VirtualMachineConfigSpec object. This is the same object that you use when you call the CreateVM_Task or ReconfigVM_Task method.
That means that all the device information is stored under the deviceChange property as an array of VirtualDeviceConfigSpec objects.
Using my standard Task & Event script it was rather simple to generate a basic report on device changes.
Update May 20th 2010:
- introduce task- and event-reader loops to handle high numbers of tasks and events in the requested time period
- add code to monitor CPU and memory changes on guests
$hours = 36 # Number of hours back
$tasknumber = 999 # Windowsize for task collector
$eventnumber = 100 # Windowsize for event collector
$report = @()
$taskMgr = Get-View TaskManager
$eventMgr = Get-View eventManager
$tFilter = New-Object VMware.Vim.TaskFilterSpec
$tFilter.Time = New-Object VMware.Vim.TaskFilterSpecByTime
$tFilter.Time.beginTime = (Get-Date).AddHours(-$hours)
$tFilter.Time.timeType = "startedTime"
$tCollector = Get-View ($taskMgr.CreateCollectorForTasks($tFilter))
$dummy = $tCollector.RewindCollector
$tasks = $tCollector.ReadNextTasks($tasknumber)
while($tasks){
$tasks | where {$_.Name -eq "ReconfigVM_Task"} | % {
$task = $_
$eFilter = New-Object VMware.Vim.EventFilterSpec
$eFilter.eventChainId = $task.EventChainId
$eCollector = Get-View ($eventMgr.CreateCollectorForEvents($eFilter))
$events = $eCollector.ReadNextEvents($eventnumber)
while($events){
$events | % {
$event = $_
switch($event.GetType().Name){
"VmReconfiguredEvent" {
$event.ConfigSpec.DeviceChange | % {
if($_.Device -ne $null){
$report += New-Object PSObject -Property @{
VMname = $task.EntityName
Start = $task.StartTime
Finish = $task.CompleteTime
Result = $task.State
User = $task.Reason.UserName
Device = $_.Device.GetType().Name
Operation = $_.Operation
}
}
}
}
Default {}
}
}
$events = $eCollector.ReadNextEvents($eventnumber)
}
$ecollection = $eCollector.ReadNextEvents($eventnumber)
# By default 32 event collectors are allowed. Destroy this event collector.
$eCollector.DestroyCollector()
}
$tasks = $tCollector.ReadNextTasks($tasknumber)
}
# By default 32 task collectors are allowed. Destroy this task collector.
$tCollector.DestroyCollector()
$report | Sort-Object -Property Start | Export-Csv "C:\VM-device-operations.csv" -NoTypeInformation -UseCulture
Annotation
Line 1: define how far back in time you want the report to go
Line 2-3: these values limit the windows of the Task and the Event collectors. For me these values produce the best result, but depending on the size of your environment, you might have to change them to get a better performance from the script
Line 10-13: configure the filter for the Task collector
Line 21: we’re only interested in ReconfigVM_Task tasks
Line 24: find all events related to the task
Line 33: report on all devices under the DeviceChange property
Line 54 & 60: there is a limit of 32 Task and 32 Event collectors per session. Clean up the collectors otherwise your next run of the script might fail
Result
The CSV file that is produced by the script looks like this.
This version of the script only looked superficially at the device changes, but since we have access to the complete VirtualMachineConfigSpec object, the script can go a lot farther.
For each device it can report the actual settings (depending of course on the device type).
And besides the devices, we can also produce reports on subjects like
- changes in CPU masks
- hot add and remove of CPUs and memory
- changes in network traffic shaping
- changes in the VMware Tools configuration
- addition or removal of the guest to a vApp
- …
Let me know if you have any other suggestions for reporting on changes made to the guests.
CPU and memory changes
Skywalker (see comments) wanted to know how to monitor for memory changes.
This is in fact quite straightforward since the information is available in the ConfigSpec property of the VmReconfiguredEvent object. The only “trick” is that you need to know how the system indicates what is changed. For the NumCPUs and MemoryMB properties, if they are not equal to zero the value has been changed.
Unfortunately the task nor event keeps track of what the original value was.
To adapt the script to report on number of CPUs and memory size changes, replace lines 33-45 with the following code.
$event.ConfigSpec | % {
if($_.NumCPUs -ne 0 -or $_.MemoryMB -ne 0){
$report += New-Object PSObject -Property @{
VMname = $task.EntityName
Start = $task.StartTime
Finish = $task.CompleteTime
Result = $task.State
User = $task.Reason.UserName
Memory = &{if($_.MemoryMB -ne 0){$_.MemoryMB}else{""}}
NumCPU = &{if($_.NumCPUs -ne 0){$_.NumCPUs}else{""}}
}
}
}
This produces a CSV file like this:



Is there a way to tell whether the resource was added or subtracted from the VM?
I want to be able to track who is adding and who is deleting how much of CPU and memory.
any ideas?
Hi jkb5054, sort of
The event only has the new value, not the original value.
So you’ll be able to see who changed CPU and/or memory, but you will not be able to tell what the original value was.
Try this
$hours = 24 # Number of hours back
$tasknumber = 999 # Windowsize for task collector
$eventnumber = 100 # Windowsize for event collector
$report = @()
$taskMgr = Get-View TaskManager
$eventMgr = Get-View eventManager
$tFilter = New-Object VMware.Vim.TaskFilterSpec
$tFilter.Time = New-Object VMware.Vim.TaskFilterSpecByTime
$tFilter.Time.beginTime = (Get-Date).AddHours(-$hours)
$tFilter.Time.timeType = "startedTime"
$tCollector = Get-View ($taskMgr.CreateCollectorForTasks($tFilter))
$dummy = $tCollector.RewindCollector
$tasks = $tCollector.ReadNextTasks($tasknumber)
while($tasks){
$tasks | where {$_.Name -eq "ReconfigVM_Task"} | % {
$task = $_
$eFilter = New-Object VMware.Vim.EventFilterSpec
$eFilter.eventChainId = $task.EventChainId
$eCollector = Get-View ($eventMgr.CreateCollectorForEvents($eFilter))
$events = $eCollector.ReadNextEvents($eventnumber)
while($events){
$events | % {
$event = $_
switch($event.GetType().Name){
"VmReconfiguredEvent" {
$report += New-Object PSObject -Property @{
VMname = $task.EntityName
Start = $task.StartTime
Finish = $task.CompleteTime
Result = $task.State
User = $task.Reason.UserName
NumCPU = $event.ConfigSpec.NumCPUs
NumCoresPerSocket = $event.ConfigSpec.NumCoresPerSocket
MemoryMB = $event.ConfigSpec.MemoryMB
}
}
Default {}
}
}
$events = $eCollector.ReadNextEvents($eventnumber)
}
# By default 32 event collectors are allowed. Destroy this event collector.
$eCollector.DestroyCollector()
}
$tasks = $tCollector.ReadNextTasks($tasknumber)
}
# By default 32 task collectors are allowed. Destroy this task collector.
$tCollector.DestroyCollector()
$report | Sort-Object -Property Start
How do use this script against one VM alone? i.e. want to see all the “reconfigure” events for one VM alone.
Hi Luc,
Is it possible to also detect hard disk changes? i.e. increasing the existing disk size, or adding another disk to a VM?
thanks.
Hi LucD,
when i run the first script i get a 0 Kb output file. i did make a few changes to cpu and memory before i ran the script
@LucD
Luc,
It seems that the column “Numcpu” is empty when I made 2 changes (more than 1 change) at a time. For instance, removed virtual floppy, and added 1 vCPU to the VM.
Hi Skywalker, just did a similar test myself and it looks indeed as if the event generated by such an action is missing the memory change.
This looks like a bug. I’ll open an SR.
Luc.
@Skywalker
brilliant!!
Hi Luc,
This script can’t report on memory change to guest OS. How to modify the script to get this done? Thanks in advance.
Yes it can. See the updates.
Luc,
Just wanted to say a big thanks for posting this.
My script was called via a virtualcenter alarm when a vmreconfigured event was initiated and so I was relying on the environment variables that the alarm passed back to provide this information.
The code youve written clearly demontrates its possible to get the information out if you have the vmreconfigured event object so I guess Im going to have to authenticate back into VC to get the relevant event object to identify the change.
Hopefully this will allow for the reporting of changes in real time.
Ill let you know how I get on…
Is there a way to report on who started a VM? Currently the GUI doesn’t match who did the power on of a VM.
Yes, you can. Even dedicated a new post to the reply.
See Events – Part 4 : Who started that VM ?