Nice feature, but what for those of us that want to automate this ?

Update October 28th 2012: Take into account that the datastorecluster is not connected to a host that is part of a cluster. Skip the HA heartbeat test.

Update April 23th 2012: Use the RetrieveDasAdvancedRuntimeInfo method to find the actual datastores that are used for the heartbeat.

What to look for ?

From the popup, it is quite obvious what specific feature can prohibit the unmount of a datastore.

But if you look at Cormac’s post called What could be writing to a VMFS when no Virtual Machines are running?, there are a couple of other settings that might influence the ability to unmount a datastore.

In Cormac’s post you’ll find that a Distributed Virtual Switch configuration file or the presence of a scratch partition might also cause IO that would prevent a datastore unmount.

So I decided to expand the function with a check for those 2 additional causes.

The script

function Get-DatastoreUnmountStatus{
  <#
.SYNOPSIS  Check if a datastore can be unmounted.
.DESCRIPTION The function checks a number of prerequisites
  that need to be met to be able to unmount a datastore.
.NOTES  Author:  Luc Dekens
.PARAMETER Datastore
  The datastore for which you want to chekc the conditions.
  You can pass the name of the datastore or the Datastore
  object returned by Get-Datastore
.EXAMPLE
  PS> Get-DatastoreUnmountStatus -Datastore DS1
.EXAMPLE
  PS> Get-Datastore | Get-DatastoreUnmountStatus
#>
  param(
    [CmdletBinding()]
    [Parameter(Mandatory=$true,ValueFromPipeline=$true)]
    [PSObject[]]$Datastore
  )

  process{
    foreach($ds in $Datastore){
      if($ds.GetType().Name -eq "string"){
        $ds = Get-Datastore -Name $ds
      }
      $parent = Get-View $ds.ExtensionData.Parent
      New-Object PSObject -Property @{
        Datastore = $ds.Name
        # No Virtual machines
        NoVM = $ds.ExtensionData.VM.Count -eq 0
        # Not in a Datastore Cluster
        NoDastoreClusterMember = $parent -isnot [VMware.Vim.StoragePod]
        # Not managed by sDRS
        NosDRS = &{
          if($parent -is [VMware.Vim.StoragePod]){
            !$parent.PodStorageDrsEntry.StorageDrsConfig.PodConfig.Enabled
          }
          else {$true}
        }
        # SIOC disabled
        NoSIOC = !$ds.StorageIOControlEnabled
        # No HA heartbeat
        NoHAheartbeat = &{
          $hbDatastores = @()
          $cls = Get-View -ViewType ClusterComputeResource -Property Host |
          where{$_.Host -contains $ds.ExtensionData.Host[0].Key}
          if($cls){
            $cls | %{
              (                $_.RetrieveDasAdvancedRuntimeInfo()).HeartbeatDatastoreInfo | %{
                $hbDatastores += $_.Datastore
              }
            }
            $hbDatastores -notcontains $ds.ExtensionData.MoRef
          }
          else{$true}
        }
        # No vdSW file
        NovdSwFile = &{
          New-PSDrive -Location $ds -Name ds -PSProvider VimDatastore -Root '\' | Out-Null
          $result = Get-ChildItem -Path ds:\ -Recurse |
          where {$_.Name -match '.dvsData'}
          Remove-PSDrive -Name ds -Confirm:$false
          if($result){$false}else{$true}
        }
        # No scratch partition
        NoScratchPartition = &{
          $result = $true
          $ds.ExtensionData.Host | %{Get-View $_.Key} | %{
            $diagSys = Get-View $_.ConfigManager.DiagnosticSystem
            $dsDisks = $ds.ExtensionData.Info.Vmfs.Extent | %{$_.DiskName}
            if($dsDisks -contains $diagSys.ActivePartition.Id.DiskName){
              $result = $false
            }
          }
          $result
        }
      }
    }
  }
}

Annotations

Line 24-26: A simple Object By Name implementation. It allows to pass the datastores by name or by object to the function.

Line 27: The parent of the datastore is used in a couple of tests further on in the script

Line 31: Test if there are any VMs or Templates present on the datastore

Line 33: Test if the datastore belongs to a Storage Cluster

Line 35-40: Test if the datastore is managed by Storage DRS. The Enabled property in the StorageDrsPodConfigInfo object is used as the indicator.

Line 42: Test if SIOC is enabled for the datastore

Line 44-57: Test if the datastore is one of the Heartbeat Datastores. The test first gets all the clusters known in vCenter and then filters out those clusters that have hosts on which the datastore is visible.

Line 48,56: If the host is not part of a cluster, the HA heartbeat test is considered as successful.

Line 59-65: Test to see if the is a Distributed Virtual Switch configuration file present on the datastore.

Line 67-77: Test if the datastore is used as a scratch partition.

Sample usage

The function can be used in a stand-alone or in a pipeline construct.

Get-DatastoreUnmountStatus -Datastore DS1

The pipeline construct could look like this

Get-Datastore DS* | Get-DatastoreUnmountStatus

The function produces 1 object per datastore. Each of the properties in the object represents 1 test that the function executes. Something like this.

The output shows us that we will probably not be able to unmount datastore DS1, because:

• the datastore is part of a Datastore Cluster
• the datastore is managed by Storage DRS
• there are VMs (or Templates) on the datastore
• the datastore is used by the Datastore Heartbeat
• the datastore has SIOC enabled

The functions to solve this and allow an unmount of the datastore will follow in some future posts.

Enjoy !

A recent post on Yellow-Bricks, called Enable Storage IO Control on all Datastores! got quite a few comments and Tweets.

I was intrigued by one of the comments on Twitter that stated that the users didn’t understand what SIOC was all about. From several posts on SIOC I came to understand that the non-VI workload event would be fired when SIOC doesn’t see any latency improvements when it throttles the storage queue. Simple enough, but is there any data available that can make this visible ?

So I decided to try and pull some performance data from the vSphere environment to help me understand what is going on when SIOC is activated and more specifically if there is any performance data that seems to explain why the NonVIWorkloadDetectedOnDatastoreEvent event is fired.

I started by looking at the performance metric to see if there were any that had anything to do with SIOC. The only ones I could find were 2 metrics in the Datastore group.

The next preparatory step was to look at the NonVIWorkloadDetectedOnDatastoreEvent event. This event extends the DatastoreEvent, which adds the datastore property to the basic Event object. From a preliminary report on this event it was clear that the NonVIWorkloadDetectedOnDatastoreEvent event is fired against a Datastore. There is no specific host information present in the event.

I envisaged a function that would be able to return the SIOC performance data for a specific datastore on a host but also for one or more datastores in a cluster. Since this ment that the resulting array would have a variable number of columns, I decided to use the Add-Type cmdlet to create a customised object each time the function is called. See my LUN report – datastore, RDM and node visibility post for another example of this technique.

The script

#requires -version 2
#requires -pssnapin VMware.VimAutomation.Core -version 4.1

function Get-SiocStat{
<#
.SYNOPSIS
	Returns SIOC related performance data for one or more
	datastores
.DESCRIPTION
	The function returns an array with SIOC performance data for
	one or more datastore over period in time.
	The data also contains all NonVIWorkloadDetectedOnDatastoreEvent
	events that occurred during the requested interval.
.NOTES
	Author:  Luc Dekens
.PARAMETER VMHostName
	The name of the ESX(i) host for which you request the data.
.PARAMETER ClusterName
	The SIOC performance data will be collected for all shared
	datastores, or for the datastore passed with DatastoreName,
	for each node in the cluster.
.PARAMETER DatastoreName
	The name of one or more datastores for which you want to retrieve
	the SIOC performance data.
	If no DatastoreName is provided, the function will return
	SIOC performance data for all the shared datastores on the
	entity.
.PARAMETER Start
	Start of the interval for which SIOC performance data will be
	collected. The default is 1 day back.
.PARAMETER Finish
	End of the interval for which SIOC performance data will be
	collected. The default is now.
.EXAMPLE
	PS> $stats= Get-SiocStat -DatastoreName MyDS -Start $start
.EXAMPLE
	PS> $stats = Get-SiocStat -HostName MyEsx -DatastoreName MyDS
.EXAMPLE
	PS> $stats = Get-SiocStat -ClusterName MyCluster -Start $start -Finish $finish
.EXAMPLE
	PS> $stats = Get-SiocStat -ClusterName MyCluster -DatastoreName MyDS
#>

	[CmdletBinding(DefaultParametersetName="Host")]
	param(
	[string]$DatastoreName = "*",
	[DateTime]$Start,
	[DateTime]$Finish,
	[Parameter(ParameterSetName="Host")]
	[string]$HostName,
	[Parameter(ParameterSetName="Cluster")]
	[string]$ClusterName
	)

	process{
		$dsTab = @{}
		$report = @()
		$hugeSamplesNumber = 99999

		if($psCmdlet.ParameterSetName -eq "Cluster"){
			$esx = Get-Cluster -Name $ClusterName | Get-VMHost
		}
		else{
			$esx = Get-VMHost -Name $HostName
		}
		if(!$Finish){
			$Finish = Get-Date
		}
		if(!$Start){
			$Start = $Finish.AddDays(-1)
		}

		Get-Datastore -Name $DatastoreName -VMHost $esx | `
		  where{$_.Type -eq "VMFS" -and $_.Extensiondata.Summary.MultipleHostAccess} | %{
			$dsTab[$_.Extensiondata.Info.Vmfs.Uuid] = $_.Name
		}
		$metrics = "datastore.sizeNormalizedDatastoreLatency.average","datastore.datastoreIops.average"

		# Create the type to hold the info
		$DSsiocDef = "public string Timestamp;`n"
		$rndName = "DSsioc" + (Get-Random -Maximum 99999)
		$DSsiocDef = "public struct " + $rndName + "{`n" + $DSsiocDef
		$dsTab.GetEnumerator() | Sort-Object -Property Value | %{
			$DSsiocDef += ("`n`tpublic bool " + $_.Value + "_alarm" + ";")
		}
		foreach($esxHost in $esx){
			$shortName = $esxHost.Name.Split('.')[0]
			$dsTab.GetEnumerator() | Sort-Object -Property Value | %{
				$DSsiocDef += ("`n`tpublic long " + $shortName + '_' + $_.Value + "_latency" + ";")
				$DSsiocDef += ("`n`tpublic long " + $shortName + '_' + $_.Value + "_iops" + ";")
			}
		}
		$DSsiocDef += "`n}"

		Add-Type -Language CsharpVersion3 -TypeDefinition $DSsiocDef

		$events = Get-VIEvent -Start $start -Finish $Finish -MaxSamples $hugeSamplesNumber | `
		  where {$_.GetType().Name -eq 'NonVIWorkloadDetectedOnDatastoreEvent'}
		$stats = Get-Stat -Entity $esx -Stat $metrics -Start $start -Finish $Finish -Instance @($dsTab.Keys)
		$groups = $stats | Sort-Object -Property Timestamp | Group-Object -Property Timestamp
		$groups | %{
			$row = New-Object $rndName
			$row.Timestamp = $_.Group[0].Timestamp
			$_.Group | %{
				$shortName = $_.Entity.Name.Split('.')[0]
				$property = $shortName + "_" + $dsTab[$_.Instance] + "_" + $_.MetricId.Split('.')[1]
				$property = $property.Replace('sizenormalizeddatastorelatency','latency')
				$property = $property.Replace('datastoreiops','iops')
				$row.$property = $_.Value
			}
			foreach($nonVIevent in $events){
				if($dsTab.Values -contains $nonVIevent.Datastore.Name -and `
				   $_.Group[0].Timestamp -le $nonVIevent.CreatedTime -and `
					 ($_.Group[0].Timestamp.AddSeconds($_.Group[0].IntervalSecs)) -gt $nonVIevent.CreatedTime){
					$property = $nonVIevent.Datastore.Name + "_alarm"
					$row.$property = $true
				}
			}
			$report += $row
		}
		$report
	}
}

Annotations

Line 49,51: The function has 2 parameter sets, one called Host and the other called Cluster. This avoids incorrect calls where you pass a Hostname and a Clustername

Line 60-62: When the function is called with the Cluster parameter set, the script will get all the ESX(i) hosts that are present in the cluster.

Line 66-71: The default start and/or finish for the interval are calculated when these values are not provided in the call to the function.

Line 73-76: A hash table is created to translate the datastore UUID to a datastorename. This translation is needed because the Instance returned by the Get-Stat cmdlet uses the datastore UUID.

Line 79-95: Define a custom object to hold all the data. Each datastore will have the following properties: <datastorename>_alarm, <hostname>_<datastorename>_latecy and <hostname>_<datastorename>_iops. Notice that the script adds a random number to the name of the new object to avoid errors on multiple runs of the script. There is currently no way that I know of to remove a type that was created by Add-Type besides stopping/starting the PowerShell session.

Line 97-98: Collects all the non-VI-workload events for the interval.

Line 99-100: Collects all the statistical data for the SIOC-related metrics.

Line 101-118: Creates and populates an object for each interval that was returned by the Get-Stat cmdlet.

Line 121: The function returns an array with customised objects.

Sample runs

As I already mentioned the function has two parameter sets.

The ‘Host‘ parameter set can be used like this

$esxName = "esx41.test.local"
$start = [DateTime]"1/23/2011 14:00"
$finish = $start.AddHours(2)
Get-SiocStat -HostName $esxName -Start $start -Finish $finish | `
Export-Csv "C:\sioc-report.csv" -NoTypeInformation -UseCulture

This will produce a CSV file that looks something like this

You can see that the host has 4 datastores. Needless to say that a report on latency and IOPS over 30 minute intervals is of no real use for looking at SIOC.

The ‘Cluster‘ parameter set will include by default performance data for all datastores for each node in the cluster.

Watch out, this can produce huge CSV file. For example a 5-node cluster with 8 shared datastores will produce a CSV file with 89 columns. When you use the Cluster parameter set it is advised to look at 1 or more specific datastores. This can be done like this.

$clusterName = "CLUS1"
$dsName = "ds1"

$start = (Get-Date).AddHours(-1)
Get-SiocStat -ClusterName $clusterName -DatastoreName $dsName -Start $start | `
Export-Csv "C:\ds1-sioc-report.csv" -NoTypeInformation -UseCulture

This produces a report like that will look something like this. The sample comes from a 3-node cluster.

Interpretation of the data

Now that I had an easy way to produce these reports I decided to do some testing.

To force some non-VI workload I started a VCB backup for a guest.

As expected this produced the Alarm for the non-VI workload. But I’m somewhat confused by the data I see in the report.

The VCB backup released the disk lease at 21:24:01.

In the report that I produced with the Get-SiocStat function, I see the Alarm being fired nearly 1 minute later. I could understand that SIOC uses a safety margin to decide if the latency decreased after SIOC throttled the storage queue depth.

But I don’t understand why I see an enormous increase in latency after the VCB disk lease is released.

And it’s not the Get-SiocStat function that makes an error, because the performance graphs for the datastore in the vSphere client seem to indicate the same thing.

Can anyone shed some light on what I see here ?

On a side note, I think it would be useful if SIOC provided a bit more information about what it is doing. Just an Alarm is a bit sparse. A metric that returns the queue depth would be a good start.

• What to Expect When You Enable Storage I/O Controls in ESX 4.1 by Alex Bakman
• Storage I/O Fairness, SIOC, tying up some loose ends and Storage IO Control Best Practices all by Duncan Epping

The only thing missing was a way to automate everything surrounding SIOC. And so I decided to write a couple of functions to fill that gap.

The script

#requires -pssnapin VMware.VimAutomation.Core -version 4.1
function Get-Sioc{
<#
.SYNOPSIS
	Get the Storage IO Control settings for a host or a datastore
.DESCRIPTION
	When called against a VMHost, the cmdlet will return the SIOC
	settings on the host
	When called against a Datastore, the cmdlet will return the SIOC
	settings for all hosts on which the datastore is accessible
.NOTES
	Authors:      Luc Dekens
.PARAMETER VMHost
	On or more hosts
.PARAMETER Datastore
	On or more datastores
.EXAMPLE
	PS> Get-Sioc -VMHost (Get-VMHost)
.EXAMPLE
	PS> Get-VMHost -Name "esx41" | Get-Sioc
.EXAMPLE
	PS> Get-Sioc -Datastore (Get-Datastore)
.EXAMPLE
	PS> Get-Datastore -Name "DS1" | Get-Sioc
#>
	param(
		[parameter(ParameterSetName = "VMHost",
			valuefrompipeline = $true,position = 0,
			HelpMessage = "Enter a host")]
			[VMware.VimAutomation.ViCore.Impl.V1.Inventory.VMHostImpl[]]$VMHost,
		[parameter(ParameterSetName = "Datastore",
			valuefrompipeline = $true, position = 0,
			HelpMessage = "Enter a datastore")]
			[VMware.VimAutomation.ViCore.Impl.V1.DatastoreManagement.DatastoreImpl[]]$Datastore
	)
	begin{
		switch($PsCmdlet.ParameterSetName){
			"VMHost"{
				$si = Get-View ServiceInstance
			}
			"Datastore"{}
		}
	}
	process{
		switch($PsCmdlet.ParameterSetName){
			"VMHost"{
				$VMHost | %{
					$result = $si.Client.VimService.QueryIORMConfigOption(
						[VMWare.Vim.VIConvert]::ToVim41($si.Content.StorageResourceManager),
						[VMWare.Vim.VIConvert]::ToVim41( $_.Extensiondata.MoRef))
					$row = "" | Select Name,SIOCSupported,SIOCStateDefault,
										SIOCThresholdMinimum,SIOCThresholdMaximum,SIOCThresholdDefault
					$row.Name = $_.Name
					$row.SIOCSupported = $result.enabledOption.supported
					$row.SIOCStateDefault = $result.enabledOption.defaultValue
					$row.SIOCThresholdMinimum = $result.congestionThresholdOption.min
					$row.SIOCThresholdMaximum = $result.congestionThresholdOption.max
					$row.SIOCThresholdDefault = $result.congestionThresholdOption.defaultValue
					$row
				}
			}
			"Datastore"{
				$Datastore | %{
					$row = "" | Select Name,SIOCEnabled,SIOCThreshold
					$row.Name = $_.Name
					$row.SIOCEnabled = $_.Extensiondata.iormConfiguration.enabled
					$row.SIOCThreshold = $_.Extensiondata.iormConfiguration.congestionThreshold
					$row
				}
			}
		}
	}
}

function Set-Sioc{
<#
.SYNOPSIS
	Enables/disables Storage IO Control for a a datastore
.DESCRIPTION
	The function enables or disables SIOC for a datastore.
.NOTES
	Authors:      Luc Dekens
.PARAMETER Datastore
	On or more datastores
.PARAMETER Enabled
	A switch that defines if SIOC will be enabled or disabled
.PARAMETER Threshold
	Specify the threshold
.EXAMPLE
	PS> Set-Sioc -Datastore (Get-Datastore -Name "DS1") -Enabled:$true
.EXAMPLE
	PS> Get-Datastore | Set-SIOC -Enabled:$true -Threshold 30
.EXAMPLE
	PS> Get-Datastore | Set-SIOC -Enabled:$false
#>
	param(
		[parameter(
		valuefrompipeline = $true,position = 0,
		HelpMessage = "Enter a datastore")]
		[VMware.VimAutomation.ViCore.Impl.V1.DatastoreManagement.DatastoreImpl[]]$Datastore,
		[switch]$Enabled,
		[int]$Threshold = 30
	)
	begin{
	$si = Get-View ServiceInstance
	$spec = New-Object VMware.Vim.StorageIORMConfigSpec
	$spec.congestionThreshold = $Threshold
	$spec.enabled = $Enabled
	}

	process{
		$Datastore | %{
			$taskMoRef = $si.Client.VimService.ConfigureDatastoreIORM_Task(
				[VMWare.Vim.VIConvert]::ToVim41($si.Content.StorageResourceManager),
				[VMWare.Vim.VIConvert]::ToVim41($_.Extensiondata.MoRef),
				[VMWare.Vim.VIConvert]::ToVim41($spec))
			$task = Get-View ([VMWare.Vim.VIConvert]::ToVim($taskMoRef))
			while ("running","queued" -contains $task.Info.State){
				$task.UpdateViewData("Info.State")
			}
		}
	}
}

Annotations

Line 27-34: Since you have SIOC settings on a Datastore and on a VMHost, I use 2 parameter sets.

Line 37: Via the $PsCmdlet.ParameterSetName the script can find out with which parameter set the function was called.

Line 39: When the function was called with the VMHost parameter set, we need the ServiceInstance object to get to the StorageResourceManager.

Line 48-50: The StorageResourceManager object is not included in the PowerCLI 4.1 framework. The script uses the method, I also used in Script vSphere 4.1 AD Authentication to get to the StorageResourceManager object and use the QueryIORMConfigOption method. Thanks again to Yasen for providing us with this method.

Line 62: If the function is called with the Datastore parameter set, the SIOC properties can be found in the StorageIORMInfo object.

Line 102: Notice that the script uses a default threshold of 30 ms, which is apparently the default except for SSD storage.

Line 113-116: Again, to call the ConfigureDatastoreIORM_Task method, the script uses the method provided by Yasen.

Samples

The Get-SIOC function can be used with two parameter sets. First the VMHost parameter set will retrieve the SIOC settings on a specific host.

Get-SIOC (Get-VMHost esx41)

This will return the SIOC settings for that specific host. Your output will look something like this.

You can of course use the Get-SIOC function a pipeline as well.

Get-VMHost | Get-SIOC | ft -Autosize

This will give a nice tabular overview

With the Datastore parameter set you get the SIOC settings for a specific datastore

Get-SIOC -Datastore DS1

And again this can be used in a pipeline

Get-Datastore | Get-SIOC

To change your SIOC settings on a datastore you can do

Set-SIOC  -Datastore (Get-Datastore DS1) -Enabled:$true -Threshold 30

Or in a pipeline

Get-Datastore -Name "DS*" | Set-SIOC -Enabled:$true -Threshold 30

Notice that the Set-SIOC function doesn’t return anything !