I took my old script, massaged it a bit and gave it a more contemporary look and feel.
Just for info, the SearchDatastoreSubFolders method is relatively slow. So scanning a couple of datastores for orphaned files might take a bit of time. Be patient

The Script

function Get-VmwOrphan{
<#
.SYNOPSIS
  Find orphaned files on a datastore
.DESCRIPTION
  This function will scan the complete content of a datastore.
  It will then verify all registered VMs and Templates on that
  datastore, and compare those files with the datastore list.
  Files that are not present in a VM or Template are considered
  orphaned
.NOTES
  Author:  Luc Dekens
.PARAMETER Datastore
  The datastore that needs to be scanned
.EXAMPLE
  PS> Get-VmwOrphan -Datastore DS1
.EXAMPLE
  PS> Get-Datastore -Name DS* | Get-VmwOrphan
#>

  [CmdletBinding()]
  param(
    [parameter(Mandatory=$true,ValueFromPipeline=$true)]
    [PSObject[]]$Datastore
  )
  
  Begin{
    $flags = New-Object VMware.Vim.FileQueryFlags
    $flags.FileOwner = $true
    $flags.FileSize = $true
    $flags.FileType = $true
    $flags.Modification = $true
    
    $qFloppy = New-Object VMware.Vim.FloppyImageFileQuery
    $qFolder = New-Object VMware.Vim.FolderFileQuery
    $qISO = New-Object VMware.Vim.IsoImageFileQuery
    $qConfig = New-Object VMware.Vim.VmConfigFileQuery
    $qConfig.Details = New-Object VMware.Vim.VmConfigFileQueryFlags
    $qConfig.Details.ConfigVersion = $true
    $qTemplate = New-Object VMware.Vim.TemplateConfigFileQuery
    $qTemplate.Details = New-Object VMware.Vim.VmConfigFileQueryFlags
    $qTemplate.Details.ConfigVersion = $true
    $qDisk = New-Object VMware.Vim.VmDiskFileQuery
    $qDisk.Details = New-Object VMware.Vim.VmDiskFileQueryFlags
    $qDisk.Details.CapacityKB = $true
    $qDisk.Details.DiskExtents = $true
    $qDisk.Details.DiskType = $true
    $qDisk.Details.HardwareVersion = $true
    $qDisk.Details.Thin = $true
    $qLog = New-Object VMware.Vim.VmLogFileQuery
    $qRAM = New-Object VMware.Vim.VmNvramFileQuery
    $qSnap = New-Object VMware.Vim.VmSnapshotFileQuery
    
    $searchSpec = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
    $searchSpec.details = $flags
    $searchSpec.Query = $qFloppy,$qFolder,$qISO,$qConfig,$qTemplate,$qDisk,$qLog,$qRAM,$qSnap
    $searchSpec.sortFoldersFirst = $true
  }
  
  Process{
    foreach($ds in $Datastore){
      if($ds.GetType().Name -eq "String"){
        $ds = Get-Datastore -Name $ds
      }

# Only shared VMFS datastore
      if($ds.Type -eq "VMFS" -and $ds.ExtensionData.Summary.MultipleHostAccess -and $ds.State -eq 'Available'){
        Write-Verbose -Message "$(Get-Date)`t$((Get-PSCallStack)[0].Command)`tLooking at $($ds.Name)"
  
# Define file DB
        $fileTab = @{}

# Get datastore files
        $dsBrowser = Get-View -Id $ds.ExtensionData.browser
        $rootPath = "[" + $ds.Name + "]"
        $searchResult = $dsBrowser.SearchDatastoreSubFolders($rootPath, $searchSpec) | Sort-Object -Property {$_.FolderPath.Length}
        foreach($folder in $searchResult){
          foreach ($file in $folder.File){
            $key = "$($folder.FolderPath)$(if($folder.FolderPath[-1] -eq ']'){' '})$($file.Path)"
            $fileTab.Add($key,$file)

            $folderKey = "$($folder.FolderPath.TrimEnd('/'))"
            if($fileTab.ContainsKey($folderKey)){
              $fileTab.Remove($folderKey)
            }
          }
        }
  
# Get VM inventory
        Get-VM -Datastore $ds | %{
          $_.ExtensionData.LayoutEx.File | %{
            if($fileTab.ContainsKey($_.Name)){
              $fileTab.Remove($_.Name)
            }
          }
        }
  
# Get Template inventory
        Get-Template | where {$_.DatastoreIdList -contains $ds.Id} | %{
          $_.ExtensionData.LayoutEx.File | %{
            if($fileTab.ContainsKey($_.Name)){
              $fileTab.Remove($_.Name)
            }
          }
        }

# Remove system files & folders from list
        $systemFiles = $fileTab.Keys | where{$_ -match "] \.|vmkdump"}
        $systemFiles | %{
          $fileTab.Remove($_)
        }

# Organise remaining files
        if($fileTab.Count){
          $fileTab.GetEnumerator() | %{
            $obj = [ordered]@{
              Name = $_.Value.Path
              Folder = $_.Name
              Size = $_.Value.FileSize
              CapacityKB = $_.Value.CapacityKb
              Modification = $_.Value.Modification
              Owner = $_.Value.Owner
              Thin = $_.Value.Thin
              Extents = $_.Value.DiskExtents -join ','
              DiskType = $_.Value.DiskType
              HWVersion = $_.Value.HardwareVersion
            }
            New-Object PSObject -Property $obj
          }
          Write-Verbose -Message "$(Get-Date)`t$((Get-PSCallStack)[0].Command)`tFound orphaned files on $($ds.Name)!"
        }
        else{
          Write-Verbose -Message "$(Get-Date)`t$((Get-PSCallStack)[0].Command)`tNo orphaned files found on $($ds.Name)."
        }
      }
    }
  }
}

Annotations

Line 28-57: The script constructs the SearchDatastoreSubFolders parameter in the Begin block. This parameter will be the same for all datastores.

Line 56: To get maximum detail for the returned files, the script uses all available FileQuery variations

Line 62-64: Cheap OBN implementation

Line 67: The script only looks at shared VMFS datastores

Line 71: This hash table will be used to determine which file/folder is orphaned or not. First all the files the method fins are placed in the hash table. Then the files that belong to VMs and Templates are removed. Finally some system files are removed. What is left are orphaned files.

Line 76: Get all the files on the datastore with the SearchDatastoreSubFolders method

Line 77-87: Enter the files in the hash table

Line 83-85: Take care of the folder entries. If a file inside a folder is encountered, the script also removes the folder itself from the hash table

Line 90-96: Remove all files that belong to registered VMs from the hash table

Line 99-105: Remove all files that belong to registered Templates from the hash table

Line 108-111: Remove system files. For now these all files that are located in folders that start with a dot, or in a folder named vmkdump.

Line 114-128: If there are entries left in the hash table, collect more information and create an ordered object containing that information.

Sample Usage

The use of this function is rather straightforward, call the function with a datastorename or a datastore object.

Get-VmwOrphan -Datastore MyDS1

or

$ds = Get-Datastore -Name MyDS1

Get-VmwOrphan -Datastore $ds

The function also accepts the datastore objects via the pipeline. You can do

Get-Cluster -Name MyCluster | Get-Datastore | Get-VmwOrphan

The objects that are returned by the function, depending on the type of file, contain information about the file. The following sample output shows how some properties are specific to certain file types.

If you want to create an orphaned files report for multiple datastores, it is handy to capture the results for each datastore in a separate worksheet in an Excel spreadsheet.

The following sample code uses Doug Finke‘s ImportExcel module to accomplish that.

$reportName = 'C:\orphan-report.xlsx'

foreach($ds in (Get-Cluster -Name MyCluster | Get-Datastore | Get-VmwOrphan | Group-Object -Property {$_.Folder.Split(']')[0].TrimStart('[')})){
    $ds.Group | Export-Excel -Path $reportName -WorkSheetname $ds.Name -AutoSize -AutoFilter -FreezeTopRow
}

The resulting spreadsheet has all the orphaned files, with a separate worksheet for each datastore that has orphaned files.

Enjoy!

Recently there were a couple of threads on this subject in the VMTN PowerCLI Community, so I decided to streamline my quick-and-dirty scripts into something more presentable, and create a PowerShell module to bundle the functions. I present the VMFSIncrease module!
The VMFSIncrease module will also be my first contribution to the PowerCLI Community Repository! More on that further on in this post.

* _{The expand and extend functions for a VMFS datastore depend on the availability of free space on the VMFS datastore extents and/or the availability of free LUNs}

The Increase button

The Increase button in fact offers two different methods to increase the size of a datastore. The definitions of two methods according to the

• Expand: increases the capacity of an existing VMFS datastore by expanding (increasing the size of) an existing extent of the datastore
• Extend: Increases the capacity of an existing VMFS datastore by adding new extents to the datastore

Perhaps the following schematic makes it a bit more clear.

The schematic shows a VMFS datastore with one extent, on LUN A. If we want to increase the Used capacity, we can do an Expand, on the same extent (LUN A). Or we can do an Extend on a new LUN, in this case LUN B.In the case of the Extend, the VMFS datastore will now have two extents, LUN A and LUN B.

Note that for the Expand and the Extent you do not need to use all of the free space. The following is perfectly possible.

The Increase button presents the user with a series of candidate LUNs. The Expand and Extend candidate LUNs are intermixed in that list. The underlying API method makes sure that the LUNs are not already used by another datastore or by a RDM disk.

The Module

The module is available in my GitHub VMFSIncrease repository.

Sample runs

Reporting

The Get-VmfsDatastoreInfo displays basic information about the extents and the partitions that make up the datastore.

Get-VmfsDatastoreInfo -Datastore TestDS

This produces something like this

The Datastore lives on one extent, and on that extent there are three partitions. The partition marked with (2) is the part where the VMFS Datastore lives. Part (3) is the free space, and part (1) is for alignment.

To check how we can increase the capacity of the Datastore, we can run

Get-VmfsDatastoreIncrease -Datastore TestDS

This returns something like this

The returned information shows that there are two options to increase the capacity of the Datastore. In (1) we see that we can Expand the extent by 80GB. In (2) we see that we can Extend the capacity by 100GB by adding a new extent to the Datastore.

Expand

The Help page for the New-VmfsDatastoreIncrease function already contains several examples on how to use the function. But to give you an idea what is possible, just a few examples.

We have the possibility to add up to 80GB. If we don’t specify a size, all the free space on the extent is used.

New-VmfsDatastoreIncrease -Datastore 'TestDS' -Expand

If we do specify how much we want to increase the capacity, only that specific amount of space from the free space will be taken.

New-VmfsDatastoreIncrease -Datastore 'TestDS' -Expand -IncreaseSizeGB 25

If we have a Datastore that spans more than one extent, we can, through the Canonicalname parameter, indicate on which extent the increase shall be taken.

New-VmfsDatastoreIncrease -Datastore 'TestDS' -Expand -IncreaseSizeGB 15 -CanonicalName 'naa.60050123017775af1800000000000011'

Extend

As we explained in the beginning of the post, with the Extend switch we indicate that we want to increase the capacity on a new extent. Again we can take the full extent.

New-VmfsDatastoreIncrease -Datastore 'TestDS' -Extend -CanonicalName 'naa.60050123017775af1800000000000012'

Interesting to note, if you don’t specify a LUN, through the Canonicalname parameter, the function will order the available LUNs alphanumerically on their Canonicalname, and take the first one. So the following is perfectly possible, provided of course there are free LUNs available.

New-VmfsDatastoreIncrease -Datastore 'TestDS' -Extend

And, as with the Expand option, you do not need to use all of the available space with the Extend option. You can control how much space is taken through the IncreaseSizeGB parameter.

New-VmfsDatastoreIncrease -Datastore 'TestDS' -Extend -IncreaseSizeGB 50 -CanonicalName 'naa.60050123017775af1800000000000012'

The option is also available without specifying a LUN. The function uses the same logic as was mentioned before, it will order the available LUNs alphanumerically on their Canonicalname, and take the first one.

New-VmfsDatastoreIncrease -Datastore 'TestDS' -Extend -IncreaseSizeGB 35

PowerCLI Community Repository

Since a couple of days the long awaited PowerCLI Community Repository is live! More info can be found in the Updating the VMware PowerCLI Community Repository! post over on the PowerCLI Blog.

If you have any scripts, functions, modules… that might be useful for the community, please submit them there.

The VMFSIncrease module will be available on the PowerCLI Community Repository shortly.

Enjoy, and share!

The files I encountered were named like this:

• vmware64-core*.gz
• vmware-vmx-zdump.*

There isn’t a lot of information available on what exactly these files are used for, besides that they seem to be created when the VM Monitor encounters a crash or a serious problem.

Since these files were quite old, and since I didn’t have any open tickets with VMware, I decided to remove these files. But of course in the PowerCLI way with a function

The Script

function Get-Dump {
<#  
.SYNOPSIS  Finds dump files on datastores   
.DESCRIPTION The function will look for dump files on
  one or more datastores. The dump filenames contain "zdump"
  or "core"
.NOTES  Author:  Luc Dekens  
.PARAMETER Datastore
  The datastore(s) the function has to search.
.EXAMPLE
  PS> Get-Dump -Datastore DS1
.EXAMPLE
  PS> Get-Datastore -Name DS* | Get-Dump
#>

  param(
    [Parameter(ValueFromPipeline=$true)]
    [PSObject[]]$Datastore
  )
  
  begin {
    $spec = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
    $spec.MatchPattern = "*zdump*","*core*"
    $spec.details = New-Object VMware.Vim.FileQueryFlags
    $spec.details.fileType = $true
    $spec.details.fileSize = $true
    $spec.details.modification = $true
    $spec.details.fileOwner = $true
    $spec.sortFoldersFirst = $true
  }
  process {
    if(!$Datastore){
      $Datastore = Get-Datastore
    }
    else{
      $Datastore | %{
        $Datastore = if($_ -is [System.String]){
          Get-Datastore -Name $_
        }
        else {$_}
      }
    }
    $Datastore | %{
      $dsBrowser = Get-View $_.ExtensionData.Browser
      $task = $dsBrowser.SearchDatastoreSubFolders("[" + $_.Name + "] ",$spec)
      $task | Where {$_.File} | Select -Property FolderPath -ExpandProperty File |
        where {$_ -isnot "VMware.Vim.FolderFileInfo"} |
        Select Path,Modification,FileSize,FolderPath
    }
  }
}

function Remove-Dump {
<#  
.SYNOPSIS  Removes files on datastores   
.DESCRIPTION The function removes files on datastores. The
  filenames need to be provided as objects returned by the
  Get-Dump function.
.NOTES  Author:  Luc Dekens  
.PARAMETER File
  The file(s) to remove
.EXAMPLE
  PS> Get-Dump -Datastore DS1 | Remove-Dump
.EXAMPLE
  PS> Get-Datastore -Name DS* | Remove-Dump -Confirm:$false 
#>

  [cmdletBinding(SupportsShouldProcess=$true,ConfirmImpact='High')]
  param(
    [Parameter(ValueFromPipeline=$true)]
    [PSObject[]]$File
  )

  begin {
    $fileMgr = Get-View FileManager
  }
  
  process {
    $File | %{
      $dsName = $_.FolderPath.Split(' ')[0].TrimStart('[').TrimEnd(']')
      $dc = (Get-Datastore -Name $dsName).Datacenter.ExtensionData.MoRef
      $name = $_.FolderPath + $_.Path
      if($pscmdlet.ShouldProcess($dsName,"Deleting file $name")){
        $fileMgr.DeleteDatastoreFile($name,$dc)
      }
    }
  }
}

Annotations

Line 18: The function accepts one or more datastores. The datastores can be passed as datastore objects that are returned by the Get-Datastore cmdlet, or they can be passed as datastorenames.

Line 22-29: The HostDatastoreBrowserSearchSpec that is used to find the dump files. In the MatchPattern property the function defines the name patterns that should be looked for. Since this specification always is the same, we define it in the Begin section of the function.

Line 32-42: The function accepts datastore objects or datastore names, these lines take care of converting datastore names to datastore objects. If no Datastore value was passed, the function will search all the datastores.

Line 45: The function uses the SearchDatastoreSubFolders to find the dump files. The same logic could also be done through the PowerCLI datastore provider. But my tests showed that the SearchDatastoreSubFolders method is 20 times faster !

Line 48: The function returns object that have the basic information about the file; the name, the path, the modification time and the size.

Line 68: The Remove-Dump function supports the WhatIf and Confirm parameters.

Line 75: The FileManager is preferred over the HostDatastoreBrowser to remove files.

Line 80-81: Since the function provides the filepath as “[datastore] folder/file“, it has to pass the datacenter MoRef to the DeleteDatastoreFile method.

Line 83: This handles the use of the WhatIf parameter.

Sample Usage

The functions are very simple to use. To look for dump files on a specific datastore, you can do

$DS = Get-Datastore -Name ds1
Get-Dump -Datastore $DS

This will, provided there are dump files present on that datastore, return an object for each dump file. The output on the console looks something like this.

The remove the dump files you can just place the returned objects on the pipeline and call the Remove-Dump function.

Get-Datastore -Name ds* | Get-Dump | Remove-Dump -Confirm:$false

You can of course remove files a bit more intelligently. You could for example remove all dump files that are more than 1 year old.

Get-Dump | Where {$_.Modification -lt $date} | Remove-Dump -Confirm:$false

Note that the Modifcation property holds the UTC time.

Enjoy !

Since there exists a SDK method that makes retrieving free SCSI LUNs quite easy, the function I came up with isn’t too complex.

But it should help you in further automating the setup of your datastores.

The Script

function Get-FreeScsiLun {

<#  
.SYNOPSIS  Find free SCSI LUNs  
.DESCRIPTION The function will find the free SCSI LUNs
  on an ESXi server
.NOTES  Author:  Luc Dekens  
.PARAMETER VMHost
    The VMHost where to look for the free SCSI LUNs  
.EXAMPLE
   PS> Get-FreeScsiLun -VMHost $esx
.EXAMPLE
   PS> Get-VMHost | Get-FreeScsiLun
#>

  param (
  [parameter(ValueFromPipeline = $true,Position=1)]
  [ValidateNotNullOrEmpty()]
  [VMware.VimAutomation.Client20.VMHostImpl]
  $VMHost
  )

  process{
    $storMgr = Get-View $VMHost.ExtensionData.ConfigManager.DatastoreSystem

    $storMgr.QueryAvailableDisksForVmfs($null) | %{
      New-Object PSObject -Property @{
        VMHost = $VMHost.Name
        CanonicalName = $_.CanonicalName
        Uuid = $_.Uuid
        CapacityGB = [Math]::Round($_.Capacity.Block * $_.Capacity.BlockSize / 1GB,2)
      }
    }
  }
}

Annotations

Line 24: The HostDatastoreSystem provides access to the method we are using

Line 26: The QueryAvailableDisksForVmfs method is called with a $null argument. That way we tell the method that we do not want to extend an existing datastore, but that we want to retrieve all free LUNs that can be used to create a datastore.

Line 27-32: The result is returned as an array of HostScsiDisk objects. With the New-Object cmdlet we pass the information that you can eventually use in the New-Datastore cmdlet.

Sample Usage

The use of this function is quite simple as the following example will show

$esx = Get-VMHost -Name MyEsx

Get-FreeScsiLun -VMHost $esx | Format-List

The result looks something like this. You can use the CanonicalName property in the Path parameter of the New-Datastore cmdlet.

You can also use the function in a pipeline construct. For example something like this

Get-Cluster -Name MyCluster |
Get-VMHost |
Get-FreeScsiLun | Format-List

On my test cluster with 2 nodes this will return the following results. And as we can see the free SCSI LUNs are visible on both nodes.

Enjoy !

But getting the individual vDisk statistics is a bit more tricky compared to getting the datastore statistics, as I showed in that post. The “disk” metrics hold the information, but the Instance that points to the MoRef value of a VM makes it a bit more tricky to retrieve.

Be forewarned, the “disk” metrics hold usage data for all the vDisks that a specific VM has on a specific datastore. You will not be able to get individual vDisk statistics, unless the vDisks are stored on different datastores !

On the positive side, the “disk” metrics will allow you to see how your vDisks increase in size over time. For your Thick vDisks that increase will be by expanding them, and for your Thin vDisks it will also show the natural growth.

The Script

$datastoreName = "MyDS"
$targetVM = "VM1","VM2","VM3"
$monthsBack = 3
$interval = "HI4"

$ds = (Get-Datastore -Name $datastoreName).ExtensionData
$metric = "disk.used.latest"
$start = (Get-Date).AddMonths(-$monthsBack)

$vmTab = @{}

$localCulture = Get-Culture

Get-VM -Name $targetVM | %{
  $vmTab.Add($_.Id.Split('-')[2],$_.Name)
}

&{Get-Stat2 -entity $ds -interval $interval -stat $metric -QueryInstances |
  where {$_.Instance -match "^\d+$" -and $vmTab.ContainsKey($_.Instance)} | %{
    Get-Stat2 -entity $ds -Instance $_.Instance -stat $metric -interval $interval -Start $start |
    Select @{N="VM";E={$vmTab[$_.Instance]}},Timestamp,Entity,Value,Unit,Interval
  }
} |
Group-Object -Property Timestamp | ForEach-Object -Process {
  $newObj = New-Object PSObject -Property @{
    Timestamp = [datetime]::Parse($_.Name,$localCulture)
  }
  $_.Group | %{
    Add-Member -InputObject $newObj -Name $_.VM -Value ($_.Value/1MB) -MemberType NoteProperty
  }
  $newObj
} |
Sort-Object -Property Timestamp |
Export-Csv "C:\vdisk.csv" -NoTypeInformation -UseCulture

Annotations

Line 1: The datastore for which the script will produce a report

Line 2: The script looks for specific VMs on the datastore

Line 4: The Historical Interval 4 holds 1 entry per day, which is sufficient for monitoring vDisk growth. If you need finer samples change it to another Historical Interval, but note that the “disk” metrics are only available for the historical intervals.

Line 12: This will allow the script to interpret dates correctly, in other words it allows the script later on to read dates in the format “mm/dd/yyyy” and in the “dd/mm/yyyy” format.

Line 14-16: For the VMs we want to report upon, the script fetches the Value property of its MoRef. This Value corresponds with the Instance that will be returned with the performance data, and that allows the script to link the Instance to the VM name.

Line 18-19: With the QueryInstances parameter the script requests all the available Instances, in other words VM specific performance data. The script checks if the Instance is in the MoRef.Value format and if the Instance corresponds with one of the VMs we want to report upon.

Line 20-21: For all the qualifying Instances the script fetches the performance data.

Line 24-32: The performance data is grouped on the Timestamp. All the individual performance objects with a specific Timestamp, each for 1 Instance (or 1 VM), are combined into 1 object with a property for each Instance.

Line 26: The Timestamp became a string, in a format defined by the locale, in the Group-Object earlier on. To interpret the date string correctly the script calls the Parse method and passes the locale object.

Sample Usage

The script uses my Get-Stat2 function I have used before. Make sure that function is known before calling the script above. The latest version of my Get-Stat2 function can be downloaded below.

[wpfilebase tag=file id=11]

The CSV file that is produced by the script looks like this:

If you have a PowerShell module or snapin that allows you to produce graphs, you can make the result a bit fancier.

I replaced the Export-Csv line with the Out-Chart cmdlet from PowerGadgets.

Out-Chart -Gallery Lines `
  -AllSeries_MarkerSize 0 `
  -Title "vDisk monitoring" `
  -AxisX_Title_Text "Date" `
  -AxisY_Title_Text "Space (GB)" `
  -Values $targetVM `
  -Label {([datetime]::Parse($_.Timestamp,$localCulture)).ToString("dd-MM")}

Notice how I use the Parse method to interpret the dates correctly for the locale where the script runs.
The resulting picture looks as follows.

This vDisk reporting script can be expanded in many ways. We could for example find all created and removed VMs, which would give us a complete report of which VMs have hsitorically been using your datastore space. But that will be for a later post.

Enjoy !

Nice feature, but what for those of us that want to automate this ?

Update October 28th 2012: Take into account that the datastorecluster is not connected to a host that is part of a cluster. Skip the HA heartbeat test.

Update April 23th 2012: Use the RetrieveDasAdvancedRuntimeInfo method to find the actual datastores that are used for the heartbeat.

What to look for ?

From the popup, it is quite obvious what specific feature can prohibit the unmount of a datastore.

But if you look at Cormac’s post called What could be writing to a VMFS when no Virtual Machines are running?, there are a couple of other settings that might influence the ability to unmount a datastore.

In Cormac’s post you’ll find that a Distributed Virtual Switch configuration file or the presence of a scratch partition might also cause IO that would prevent a datastore unmount.

So I decided to expand the function with a check for those 2 additional causes.

The script

function Get-DatastoreUnmountStatus{
  <#
.SYNOPSIS  Check if a datastore can be unmounted.
.DESCRIPTION The function checks a number of prerequisites
  that need to be met to be able to unmount a datastore.
.NOTES  Author:  Luc Dekens
.PARAMETER Datastore
  The datastore for which you want to chekc the conditions.
  You can pass the name of the datastore or the Datastore
  object returned by Get-Datastore
.EXAMPLE
  PS> Get-DatastoreUnmountStatus -Datastore DS1
.EXAMPLE
  PS> Get-Datastore | Get-DatastoreUnmountStatus
#>
  param(
    [CmdletBinding()]
    [Parameter(Mandatory=$true,ValueFromPipeline=$true)]
    [PSObject[]]$Datastore
  )

  process{
    foreach($ds in $Datastore){
      if($ds.GetType().Name -eq "string"){
        $ds = Get-Datastore -Name $ds
      }
      $parent = Get-View $ds.ExtensionData.Parent
      New-Object PSObject -Property @{
        Datastore = $ds.Name
        # No Virtual machines
        NoVM = $ds.ExtensionData.VM.Count -eq 0
        # Not in a Datastore Cluster
        NoDastoreClusterMember = $parent -isnot [VMware.Vim.StoragePod]
        # Not managed by sDRS
        NosDRS = &{
          if($parent -is [VMware.Vim.StoragePod]){
            !$parent.PodStorageDrsEntry.StorageDrsConfig.PodConfig.Enabled
          }
          else {$true}
        }
        # SIOC disabled
        NoSIOC = !$ds.StorageIOControlEnabled
        # No HA heartbeat
        NoHAheartbeat = &{
          $hbDatastores = @()
          $cls = Get-View -ViewType ClusterComputeResource -Property Host |
          where{$_.Host -contains $ds.ExtensionData.Host[0].Key}
          if($cls){
            $cls | %{
              (                $_.RetrieveDasAdvancedRuntimeInfo()).HeartbeatDatastoreInfo | %{
                $hbDatastores += $_.Datastore
              }
            }
            $hbDatastores -notcontains $ds.ExtensionData.MoRef
          }
          else{$true}
        }
        # No vdSW file
        NovdSwFile = &{
          New-PSDrive -Location $ds -Name ds -PSProvider VimDatastore -Root '\' | Out-Null
          $result = Get-ChildItem -Path ds:\ -Recurse |
          where {$_.Name -match '.dvsData'}
          Remove-PSDrive -Name ds -Confirm:$false
          if($result){$false}else{$true}
        }
        # No scratch partition
        NoScratchPartition = &{
          $result = $true
          $ds.ExtensionData.Host | %{Get-View $_.Key} | %{
            $diagSys = Get-View $_.ConfigManager.DiagnosticSystem
            $dsDisks = $ds.ExtensionData.Info.Vmfs.Extent | %{$_.DiskName}
            if($dsDisks -contains $diagSys.ActivePartition.Id.DiskName){
              $result = $false
            }
          }
          $result
        }
      }
    }
  }
}

Annotations

Line 24-26: A simple Object By Name implementation. It allows to pass the datastores by name or by object to the function.

Line 27: The parent of the datastore is used in a couple of tests further on in the script

Line 31: Test if there are any VMs or Templates present on the datastore

Line 33: Test if the datastore belongs to a Storage Cluster

Line 35-40: Test if the datastore is managed by Storage DRS. The Enabled property in the StorageDrsPodConfigInfo object is used as the indicator.

Line 42: Test if SIOC is enabled for the datastore

Line 44-57: Test if the datastore is one of the Heartbeat Datastores. The test first gets all the clusters known in vCenter and then filters out those clusters that have hosts on which the datastore is visible.

Line 48,56: If the host is not part of a cluster, the HA heartbeat test is considered as successful.

Line 59-65: Test to see if the is a Distributed Virtual Switch configuration file present on the datastore.

Line 67-77: Test if the datastore is used as a scratch partition.

Sample usage

The function can be used in a stand-alone or in a pipeline construct.

Get-DatastoreUnmountStatus -Datastore DS1

The pipeline construct could look like this

Get-Datastore DS* | Get-DatastoreUnmountStatus

The function produces 1 object per datastore. Each of the properties in the object represents 1 test that the function executes. Something like this.

The output shows us that we will probably not be able to unmount datastore DS1, because:

• the datastore is part of a Datastore Cluster
• the datastore is managed by Storage DRS
• there are VMs (or Templates) on the datastore
• the datastore is used by the Datastore Heartbeat
• the datastore has SIOC enabled

The functions to solve this and allow an unmount of the datastore will follow in some future posts.

Enjoy !

With VMFS5 comes a bunch of new features. Just to name a few:

• 64TB VMFS Volumes in 1 extent
• 64TB physical RDM
• Unified block size of 1MB
• Support for more files (> 100000)

For a complete list of the features that VMFS5 introduces, have a look at Cormac‘s post, called vSphere 5.0 Storage Features Part 1 – VMFS-5.

The vSphere 5 Client has an option to upgrade your VMFS3 datastores to VMFS5 datastores.

Handy option, but we want to automate this of course

And as a bonus I’ll throw in a function to get the partition information from a datastore.

The script

function ConvertTo-Vmfs5{
<#
.SYNOPSIS  Upgrade a datastore to VMFS5
.DESCRIPTION The function will convert the datastores that
  are passed to it to VMFS5.
.NOTES  Author:  Luc Dekens
.PARAMETER Datastore
  The datastore(s) to be upgraded
  The parameter accepts a string or an object returned by the
  Get-Datastore cmdlet.
.EXAMPLE
  PS> ConvertTo-Vmfs5 -Datastore "DS*"
.EXAMPLE
  PS> Get-Datastore -Name "DS1" | ConvertTo-Vmfs5
#>

  param(
  [CmdletBinding()]
  [parameter(Mandatory = $true, ValueFromPipeline = $true)]
  [PSObject[]]$Datastore
  )

  process{
    foreach($ds in $Datastore){
      if($ds.getType().Name -eq "string"){
        $ds = Get-Datastore -Name $ds
      }
      $poweredHosts = $ds.ExtensionData.Host | %{Get-View -Id $_.Key} |
      where {$_.Runtime.PowerState -eq "PoweredOn"}
      
      $oldHost = $poweredHosts | 
      where {$_.Capability.SupportedVmfsMajorVersion -notcontains 5}
      if($oldHost){
        Write-Warning "One of the connected hosts doesn't support VMFS5"
        exit
      }
      $vmfsPath = $ds.ExtensionData.Host[0].MountInfo.Path
      $storSys = Get-View ($poweredHosts | Get-Random).ConfigManager.StorageSystem

      $storSys.UpgradeVmfs($vmfsPath)
    }
  }
}

function Get-VmfsPartitionInfo{
<#
.SYNOPSIS  Retrieves partition info for a datastore
.DESCRIPTION The function will retrieve partition information
  for one or more datastores
.NOTES  Author:  Luc Dekens
.PARAMETER Datastore
  The datastore(s) for which the partition info shall be
  returned. The parameter accepts a string or an object
  returned by the Get-Datastore cmdlet.
.EXAMPLE
  PS> Get-VmfsPartitionInfo -Datastore "DS*"
.EXAMPLE
  PS> Get-Datastore -Name "DS1" | Get-VmfsPartitionInfo
#>
  param(
  [CmdletBinding()]
  [parameter(Mandatory = $true, ValueFromPipeline = $true)]
  [PSObject[]]$Datastore
  )

  process{
    foreach($ds in $Datastore){
      if($ds.getType().Name -eq "string"){
        $ds = Get-Datastore -Name $ds
      }
      $poweredHosts = $ds.ExtensionData.Host | %{Get-View -Id $_.Key} |
      where {$_.Runtime.PowerState -eq "PoweredOn"}
      $storSys = Get-View ($poweredHosts | Get-Random).ConfigManager.StorageSystem

      $ds.ExtensionData.Info.Vmfs.Extent | %{
        $devicePath = "/vmfs/devices/disks/" + $_.DiskName
        $storSys.RetrieveDiskPartitionInfo($devicePath) | %{
          New-Object PSObject -Property @{
            Name = $ds.Name
            Version = $ds.FileSystemVersion
            Device = $_.DeviceName
            PartitionFormat = $_.Spec.PartitionFormat
            BlockSizeMB = $ds.ExtensionData.Info.Vmfs.BlockSizeMb
            Blocks = $_.Layout.Total.Block
          }
        }
      }
    }
  }
}

Annotations

Line 25-27: A rather simple Object By Name implementation.

Line 28-29: Collect the hosts, connected to the datastore, that are powered on. The reason is that we can’t use the UpgradeVmfs method via the StorageSystem of a powered off host.

Line 31-36: Check if all the powered on and connected hosts support VMFS5. If not, leave the function.

Line 37: Get the devicepath to the datastore.

Line 38: Get the HostStorageSystem. Note that the function randomly selects 1 host from the powered on hosts. There is no real reason to do this, but I wanted to spread the method call over all available hosts.

Line 40: Convert the datastore to VMFS5

Line 68-70: Another example of my rather simple Object By Name implementation.

Line 71-73: We randomly take 1 of the connected hosts to fetch the HostStorageSystem.

Line 75: We loop through each VMFS extent of the datastore

Line 76-77: The function uses the RetrieveDiskPartitionInfo method to fetch the partition information.

Line 78-85: The function fetches the information for each partition and puts it on the pipeline as a PSObject.

Sample usage

In our vSphere 4 environment we have a number of VMFS3 datastores.

First, I use the Get-VmfsPartition function to check the partitioninfo of each datastore.

Get-Datastore DS* | Get-VmfsPartitionInfo | ft -AutoSize

The result.

Notice the BlockSizeMB column in the output, all 4 possible blocksizes for VMFS3 datastores are present.

Let’s upgrade these VMFS3 datastores to VMFS5 datastores.

Get-Datastore DS* | ConvertTo-Vmfs5

The convert function produces no output, but we can check what the new properties are with the Get-VmfsPartitionInfo function.

The VMFS version is 5.54, so the conversion worked.

But the attentive viewer might have noticed a couple of values he didn’t expect.

• The PartitionFormat still says ‘MBR’. Shouldn’t that be ‘GPT’ ?
• The BlocksizeMB stayed the same as before the VMFS5 conversion. Shouldn’t that be 1MB for VMFS5 datastores ?

The answer to both questions can be found in Cormac’s post called vSphere 5.0 Storage Features Part 1 – VMFS-5.

• A converted datastores keeps the ‘MBR’ format until it grows bigger than 2TB.
• A converted datastores keeps the original VMFS3 blocksize.

And I want to repeat Cormac’s final advise in his post, if you have the luxury to do so, it’s better to create new VMFS5 datastores instead of converting VMFS3 datastores !

VIProperty

Derived from the Get-VmfsPartitionInfo function, I created a New-VIProperty definition to return the format of the first datastore partition.

New-VIProperty -Name PartitionFormat -ObjectType Datastore `
  -Value {
    param($ds)

    $storSys = Get-View (Get-View $ds.ExtensionData.Host[0].Key).ConfigManager.StorageSystem
    $partInfo = $storSys.RetrieveDiskPartitionInfo("/vmfs/devices/disks/" + 
      $ds.ExtensionData.Info.Vmfs.Extent[0].DiskName)
    $partInfo[0].Spec.PartitionFormat
  } -Force | Out-Null

Note that you will only get the partition format (MBR or GPT) when you use this in a vSphere 5, or higher, environment. The partitionFormat property in the HostDiskPartitionSpec object is only available since API 5.

Enjoy !

The FDM roles

The following script will show,for each cluster node, which FDM role it holds.

function Get-DasHostState{
  param(
  [PSObject]$Cluster
  )

  if($Cluster.GetType().Name -eq "string"){
    $Cluster = Get-Cluster -Name $Cluster
  }
  Get-View $cluster.ExtensionData.Host -Property Name,Runtime.DasHostState | %{
    New-Object PSObject -Property @{
      VMHost = $_.Name
      DasHostState = $_.RunTime.DasHostState.State
      StateReporter = (Get-View -Id $_.RunTime.DasHostState.StateReporter -Property Name).Name
    }
  }
}

To call the function you can pass a clustername or the output of a Get-Cluster cmdlet. Something like this.

Get-DasHostState -Cluster Cluster1

or this

$clus = Get-Cluster -Name Cluster1
Get-DasHostState -Cluster $clus

The output looks like this

Note that there are more DasHostState values, besides master and connectedToMaster, possible. The complete list can be found in the ClusterDasFdmAvailabilityState enumeration.

And you can of course provide this functionality as a New-VIProperty.

New-VIProperty -Name "DasHostState" -ObjectType Cluster -Value {
  param($cluster)

  Get-View $cluster.ExtensionData.Host -Property Name,Runtime.DasHostState | %{
    New-Object PSObject -Property @{
      VMHost = $_.Name
      DasHostState = $_.RunTime.DasHostState.State
      StateReporter = (Get-View -Id $_.RunTime.DasHostState.StateReporter).Name
    }
  }
} -Force | Out-Null

Now you can find the FDM states with a call like this

Get-Cluster -Name London | Select -ExpandProperty DasHostState

The output will look exactly the same as the output that comes out of the Get-DasHostState function.

The heartbeat datastore

Another HA novelty in vSphere 5 is that it will now use, besides a network-based heartbeat, a datastore heartbeat, to check the presence of the nodes. To find out which of the shared datastores is used as the heartbeat datastore, you can use the following function.

function Get-DasHostState{
  param(
  [PSObject]$Cluster
  )

  if($Cluster.GetType().Name -eq "string"){
    $Cluster = Get-Cluster -Name $Cluster
  }

  $cluster.ExtensionData.RetrieveDasAdvancedRuntimeInfo()  | %{
    $_.HeartbeatDatastoreInfo | %{
      New-Object PSObject -Property @{
        Datastore = (Get-View -Id $_.Datastore).Name
        VMHosts = [string]::Join(',',($_.Hosts | %{(Get-View -Id $_).Name}))
      }
    }
  }
}

The function can be called again with the name of a cluster or with the object returned by the Get-Cluster cmdlet.

Get-DasHostState -Cluster Cluster1

The output looks something like this.

And of course, the same functionality as a New-VIProperty.

New-VIProperty -Name HeartbeatDatastore -ObjectType Cluster -Value {
  param($cluster)

  $cluster.ExtensionData.RetrieveDasAdvancedRuntimeInfo() | %{
    [String]::Join(',',($_.HeartbeatDatastoreInfo |
    %{(Get-View -Id $_.Datastore).Name}))
  }
} -Force | Out-Null

You can now use the HeartbeatDatastore property to get the datastore that is used for the heartbeat.

Get-Cluster -Name Cluster1 | Select Name, HeartbeatDatastore

Which will result in

Enjoy !

Have a look at David Davis‘s post, called Using VMware vSphere Storage Views, for more information on what you can do with the Storage Views.

Some time ago I got a question from Andrew how the Multipathing Status presented in the Storage Views could be detected and reported upon by a PowerCLI script. What looked rather simple at first, turned out to be a bit more difficult than I anticipated.

The primary reason I created this function, is that it will allow to schedule the creation of the Storage Views – Datastores values at regular intervals. A feature that is unfortunately missing from the vCenter Client plug-in.

With those saved values it is easy to produce reports that show who is consuming the storage over time.

The script

function Get-StorageViewDatastore{
<#
.SYNOPSIS  Retrieve the "Storage Views - Show all Datastores" values
.DESCRIPTION The function calculates and returns all the values
  that you get in the Storage Views - Show all Datastores panel.
.NOTES  Author:  Luc Dekens
.PARAMETER VMHost
  Specify the ESX(i) server for which you want to retrieve
  the Storage Views values.
.PARAMETER Cluster
  Specify the Cluster for which you want to retrieve
  the Storage Views values.
.PARAMETER Datacenter
  Specify the Datacenter for which you want to retrieve
  the Storage Views values.
.PARAMETER VM
  Specify the VM for which you want to retrieve
  the Storage Views values.
.PARAMETER ResourcePool
  Specify the ResourcePool for which you want to retrieve
  the Storage Views values.
.EXAMPLE
  PS> Get-StorageViewDatastore -VMHost MyCluster
.EXAMPLE
  PS> Get-Datacenter MyDC | Get-StorageViewDatastore
#>

  [CmdletBinding()]
  param(
  [parameter(ParameterSetName="VMHost",Mandatory = $true)]
  [PSObject]$VMHost,
  [parameter(ParameterSetName="Cluster",Mandatory = $true)]
  [PSObject]$Cluster,
  [parameter(ParameterSetName="Datacenter",Mandatory = $true)]
  [PSObject]$Datacenter,
  [parameter(ParameterSetName="VM",Mandatory = $true)]
  [PSObject]$VM,
  [parameter(ParameterSetName="ResourcePool",Mandatory = $true)]
  [PSObject]$ResourcePool
  )

  process{
    # Get the datastores and ESX(i) servers
    switch($PSCmdlet.ParameterSetName){
      "Datacenter" {
        if($Datacenter.GetType().Name -eq "String"){
          $Datacenter = Get-Datacenter -Name $Datacenter
        }
        $vms = Get-VM -Location $Datacenter
        $esx = Get-VMHost -Location $Datacenter
        $datastores = Get-Datastore -VMHost $esx
      }
      "Cluster" {
        if($Cluster.GetType().Name -eq "String"){
          $Cluster = Get-Cluster -Name $Cluster
        }
        $vms = Get-VM -Location $cluster
        $esx = Get-VMHost -Location $Cluster
        $datastores = Get-Datastore -VMHost $esx
      }
      "VMHost" {
        if($VMHost.GetType().Name -eq "String"){
          $VMHost = Get-VMHost -Name $VMHost
        }
        $vms = Get-VM -Location $VMHost
        $esx = $VMHost
        $datastores = Get-Datastore -VMHost $VMHost
      }
      "VM" {
        if($VM.GetType().Name -eq "String"){
          $VM = Get-VM -Name $VM
        }
        $vms = $vm
        $esx = $vm.VMHost
        $datastores = Get-Datastore -Name ($vms | Get-HardDisk -DiskType flat | %{$_.Filename.Split(']')[0].TrimStart('[')} | Sort-Object -Unique) | Sort-Object -Property Name
      }
      "ResourcePool" {
        if($ResourcePool.GetType().Name -eq "String"){
          $ResourcePool = Get-ResourcePool -Name $ResourcePool
        }
        $vms = Get-VM -Location $ResourcePool
        $esx = $vms  | %{$_.VMHost} | Sort-Object -Property Name -Unique
        $datastores = Get-Datastore -Name ($vms | Get-HardDisk -DiskType flat | %{$_.Filename.Split(']')[0].TrimStart('[')} | Sort-Object -Unique) | Sort-Object -Property Name
      }
    }

    # Create some helper collections
    $esxMoRef = $esx | %{$_.ExtensionData.MoRef}
    $vmMoRef = $vms | %{$_.ExtensionData.MoRef}

    # Set up a HBA hash table
    $hbaFCNodeWWN = @{}
    Get-VMHostHba -VMHost $esx -Type FibreChannel | %{
      $hbaFCNodeWWN["{0:X}" -f $_.NodeWorldWideName] = 0
    }

    # Handle all datastores
    foreach($ds in $datastores){
      $mpStatus = $usedSpace = $snapSpace = ""
      if($ds.Type -eq "VMFS"){
        $lun = $ds.ExtensionData.Info.Vmfs.Extent | %{$_.DiskName} |
          %{Get-ScsiLun -CanonicalName $_ -VmHost $esx -ErrorAction SilentlyContinue}
        $hbaFCCopy = $hbaFCNodeWWN.Clone()
        # Count Active and Standby paths
        # If there are 2 or more paths, the multi-path status is fully redundant
        Get-ScsiLunPath -ScsiLun $lun | where {"Active" ,"Standby" -contains $_.State} | %{
          $hbaFCCopy[$_.LunPath.Split(':')[0].Split('.')[1]] += 1
        }
        if(($hbaFCCopy.Values | where {$_} | Measure-Object).Count -gt 1){
          $mpStatus = "Full Redundancy"
        }
        else{
          $mpStatus = "Partial/No Redundancy"
        }
      }

      # Get the total used, snapshot, swap, shared and other space
      $usedSpace = 0
      $snapSpace = 0
      $vdiskSpace = 0
      $swapSpace = 0
      $otherSpace = 0
      $sharedSpace = 0
      if($ds.ExtensionData.Vm){
        Get-View $ds.ExtensionData.Vm | where {$vmMoRef -contains $_.MoRef -and $esxMoRef -contains $_.Runtime.Host} | %{
          $snapIndex = @()
          $dsPattern = '\[' + $ds.Name + '\]'
          $usedSpace += ($_.Storage.PerDatastoreUsage | where {$_.Datastore -eq $ds.ExtensionData.MoRef}).Committed
          if($_.LayoutEx.Snapshot){
            $snapIndex = $_.layoutEx.Disk | %{$_.Chain[1..($_.Chain.Count - 1)]} | %{$_.FileKey}
          }
          $snapIndex += $_.layoutEx.File | where {"snapshotList","snapshotData" -contains $_.Type -and $_.Name -match $dsPattern} | %{$_.Key}
          if($snapIndex){
            $snapSpace += ($_.LayoutEx.File | where {$snapIndex -contains $_.Key -and $_.Name -match $ds.Name} | Measure-Object -Property Size -Sum).Sum
          }
          $vdiskIndex = $_.layoutEx.File | where {"diskDescriptor","diskExtent" -contains $_.Type -and $_.Name -match $dsPattern} | %{$_.Key}
          if($vdiskIndex){
            $vdiskSpace += ($_.LayoutEx.File | where {$vdiskIndex -contains $_.Key} | Measure-Object -Property Size -Sum).Sum
          }
          $swapIndex = $_.LayoutEx.File | where {$_.Type -eq "swap" -and $_.Name -match $dsPattern} | %{$_.Key}
          if($swapIndex){
            $swapSpace += ($_.LayoutEx.File | where {$swapIndex -contains $_.Key} | Measure-Object -Property Size -Sum).Sum
          }
          $otherIndex = $_.LayoutEx.File | where {"log","config","extendedConfig","nvram","core" -contains $_.Type -and $_.Name -match $dsPattern} | %{$_.Key}
          if($otherIndex){
            $otherSpace += ($_.LayoutEx.File | where {$otherIndex -contains $_.Key} | Measure-Object -Property Size -Sum).Sum
          }
          $vmds = $_.Storage.PerDatastoreUsage | where {$_.Datastore -eq $ds.ExtensionData.MoRef}
          if($vmds){
            $sharedSpace += ($vmds.Committed - $vmds.Unshared)
          }
        }
      }

      # Collect all values for Storage Views - Datastores
      New-Object PSObject -Property @{
        VMHost = $esx.Name
        Datastore = $ds.Name
        "File System Type" = $ds.Type.Replace("NFS","NAS")
        "Connectivity Status" = &{if($ds.Accessible){"Up"}else{"Down"}}
        "Multipathing Status" = $mpStatus
        Capacity = Get-FriendlyUnit -Value ($ds.CapacityMB * 1MB)
        Free = Get-FriendlyUnit -Value ($ds.FreeSpaceMB * 1MB)
        Used = Get-FriendlyUnit -Value $usedSpace
        Snapshot = Get-FriendlyUnit -Value $snapSpace
        vDisk = Get-FriendlyUnit -Value ($vdiskSpace - $snapSpace)
        Swap = Get-FriendlyUnit -Value $swapSpace
        Other = Get-FriendlyUnit -Value $otherSpace
        Shared = Get-FriendlyUnit -Value $sharedSpace
      }
    }
  }
}

Annotations

Line 29-40: The function uses 5 parameter sets that allow you to get the Storage Views – Datastores values for a VM, an ESX(i) server, a ResourcePool, a Cluster or a Datacenter.

Line 44-85: For each of the parameter sets, this Switch block calculates the VMs, the ESX(i) host(s) and the datastore(s) that need to be queried. Notice that each of the 5 parameter sets support a kind of Object By Name (OBN), that means you can pass the name or the object itself.

Line 88-89: The script will use these MoRef collections later on to determine if a specific VM needs to be included in the calculations for the Storage Views values.

Line 92-95: To determine the Multipathing Status the script first creates a hash table for all the FC HBAs in the ESX(i) servers. The key for this hash table is the WWN, in hex format, for the HBA.

Line 98-171: The main loop over all the datastores that the script needs to handle.

Line 101-102: The script collects all the LUNs that are part of the datastore.

Line 103: The Multipathing logic works with a copy of the hash table that was created earlier.

Line 106-108: In the hash table the script increments the value for each LUN path that is Active or Standby.

Line 109-114: If a LUN has more then 1 path, the Multipathing Status is considered as Full Redundancy. Otherwise the status is Partial/No Redundancy.

Line 118-153: The script calculates all values that are related to the storage in use by the VMs.

Line 125: For a VM to be taken into account, the VM must use storage on the datastore and must be hosted by one of the ESX(i) host.

Line 127: This Regular Expression will be used later on to verify the a specific file is located on the datastore. Note that the square brackets need to be escaped.

Line 128: The Space Used value is the sum of all the Committed space for each of the VMs on that datastore.

Line 129-135: The space used by snapshots is the sum of all the VMDK snapshot files and the snapshotList and the snapshotData files.

Line 132: Note that we check for each file used for the Snapshot Space calculation, if it is located on the datastore with the -match operator. This is required since the administrator or user can define that snapshot files need to be stored on another datastore.

Line 136-139: All the files used for the vDisks of a VM.

Line 140-143: All swap related files

Line 144-147: All other files that are used by the VM.

Line 148-151: This block calculates the Shared Space for a VM. Examples of shared space usage are linked clones that are created from a snapshot with the createNewChildDiskBacking option for the diskMoveType (see the VirtualMachineRelocateSpec).

Line 156-170: Create the object, per datastore, with all the values. To have similar values that are presented in the Storage Views panel, the script uses the Get-FriendlyUnits function from my Friendly Units post.

Line 166: Note that for the Virtual Disk Space value we need to subtract the Snapshot Space from the total disk space.

Samples

You can pass the function the name of the object for which you want the Storage Views – Datastore values.

Get-StorageViewDatastore -Cluster MyCluster |
Select-Object Datastore,"File System Type","Connectivity Status","Multipathing Status",
  @{N="Capacity";E={"{0,7:f2} {1,2}" -f $_.Capacity.Value,$_.Capacity.Unit}},
  @{N="Free Space";E={"{0,7:f2} {1,2}" -f $_.Free.Value,$_.Free.Unit}},
  @{N="Space Used";E={if($_.Used.Value){"{0,7:f2} {1,2}" -f $_.Used.Value,$_.Used.Unit}}},
  @{N="Snapshot Space";E={if($_.Snapshot.Value){"{0,7:f2} {1,2}" -f $_.Snapshot.Value,$_.Snapshot.Unit}}},
  @{N="Virtual Disk Space";E={if($_.vDisk.Value){"{0,7:f2} {1,2}" -f $_.vDisk.Value,$_.vDisk.Unit}}},
  @{N="Swap Space";E={if($_.Swap.Value){"{0,7:f2} {1,2}" -f $_.Swap.Value,$_.Swap.Unit}}},
  @{N="Other VM Space";E={if($_.Other.Value){"{0,7:f2} {1,2}" -f $_.Other.Value,$_.Other.Unit}}},
  @{N="Shared Space";E={if($_.Shared.Value){"{0,7:f2} {1,2}" -f $_.Shared.Value,$_.Shared.Unit}}} |
Format-Table -AutoSize

Or you can pass the actual object.

$dc = Get-Datacenter -Name MyDC
Get-StorageViewDatastore -Datacenter $dc |
Select-Object Datastore,"File System Type","Connectivity Status","Multipathing Status",
  @{N="Capacity";E={"{0,7:f2} {1,2}" -f $_.Capacity.Value,$_.Capacity.Unit}},
  @{N="Free Space";E={"{0,7:f2} {1,2}" -f $_.Free.Value,$_.Free.Unit}},
  @{N="Space Used";E={if($_.Used.Value){"{0,7:f2} {1,2}" -f $_.Used.Value,$_.Used.Unit}}},
  @{N="Snapshot Space";E={if($_.Snapshot.Value){"{0,7:f2} {1,2}" -f $_.Snapshot.Value,$_.Snapshot.Unit}}},
  @{N="Virtual Disk Space";E={if($_.vDisk.Value){"{0,7:f2} {1,2}" -f $_.vDisk.Value,$_.vDisk.Unit}}},
  @{N="Swap Space";E={if($_.Swap.Value){"{0,7:f2} {1,2}" -f $_.Swap.Value,$_.Swap.Unit}}},
  @{N="Other VM Space";E={if($_.Other.Value){"{0,7:f2} {1,2}" -f $_.Other.Value,$_.Other.Unit}}},
  @{N="Shared Space";E={if($_.Shared.Value){"{0,7:f2} {1,2}" -f $_.Shared.Value,$_.Shared.Unit}}} |
Export-Csv "C:\SV-MyDC.csv" -NoTypeInformation -UseCulture

The previous example also showed how the results can be saved to a CSV file.
Such a CSV file looks as follows.

Enjoy the function !

A quick browse through the available metrics, on the SDK Reference PerformanceManager page, showed that these metrics are indeed available.

With the disk.capacity.latest, disk.provisioned.latest and disk.used.latest metrics this should be a simple script. But was it ? As it turned out there are a few gotchas!

Getting there

The SDK Reference, on the Storage Capacity page, mentions that these metrics are available for virtual machines and for datastores. In this case we obviously want the metrics for the datastore.

Get-Stat doesn’t do datastores.

Which brings us immediately to the first problem, the current Get-Stat cmdlet doesn’t accept datastore entities. No problem, I have an older post, called Get-Stat2 : another way of getting at the statistical data, which I should be able to adapt to retrieve datastore statistics.

The script uses a number of methods on the PerformanceManager object, to retrieve the counterId and available instances for the metrics. Once it has these it calls the QueryPerf method to get the actual statistical data.

Next problem!

The “optional value not set” error

On each call I seemed to get an error, stating “optional value not set”. A quick search through the VMTN posts showed that I was not the first one seeing that error when using datastore metrics. Unfortunately none of these posts provided me with a workable solution.

After some trial and error, I discovered that the problem was caused by the intervalId property. Although the SDK Reference states that you can either use begintime/endtime or intervalId, it seems that for datastores the PerformanceManager methods only want you to use beginTime and endTime.

This required some fundamental changes to the Get-Stat2 script, but in the end I got it working.

Instances. What instances ?

As with most metrics, you have the option to use the instance property, to identify which specific statistical data you want to retrieve. According to the SDK Reference page, the Storage Capacity metrics support the following instances.

The VMid is the value property of the MoRef of a specific virtual machine.

The script

function Get-Stat2 {
<#
.SYNOPSIS  Retrieve vSphere statistics
.DESCRIPTION The function is an alternative to the Get-Stat cmdlet.
  It's primary use is to provide functionality that is missing
  from the Get-Stat cmdlet.
.NOTES  Author:  Luc Dekens
.PARAMETER Entity
  Specify the VIObject for which you want to retrieve statistics
  This needs to be an SDK object
.PARAMETER Start
  Start of the interval for which to retrive statistics
.PARAMETER Finish
  End of the interval for which to retrive statistics
.PARAMETER Stat
  The identifiers of the metrics to retrieve
.PARAMETER Instance
  The instance property of the statistics to retrieve
.PARAMETER Interval
  Specify for which interval you want to retrieve statistics.
  Allowed values are RT, HI1, HI2, HI3 and HI4
.PARAMETER MaxSamples
  The maximum number of samples for each metric
.PARAMETER QueryMetrics
  Switch to indicate that the function should return the available
  metrics for the Entity specified
.PARAMETER QueryInstances
  Switch to indicate that the function should return the valid instances
  for a specific Entity and Stat
.EXAMPLE
  PS> Get-Stat2 -Entity $vm.Extensiondata -Stat "cpu.usage.average" -Interval "RT"
#>

  [CmdletBinding()]
  param (
  [parameter(Mandatory = $true,  ValueFromPipeline = $true)]
  [PSObject]$Entity,
  [DateTime]$Start,
  [DateTime]$Finish,
  [String[]]$Stat,
  [String]$Instance = "",
  [ValidateSet("RT","HI1","HI2","HI3","HI4")]
  [String]$Interval = "RT",
  [int]$MaxSamples,
  [switch]$QueryMetrics,
  [switch]$QueryInstances)

  # Test if entity is valid
  $EntityType = $Entity.GetType().Name

  if(!(("HostSystem",
        "VirtualMachine",
        "ClusterComputeResource",
        "Datastore",
        "ResourcePool") -contains $EntityType)) {
    Throw "-Entity parameters should be of type HostSystem, VirtualMachine, ClusterComputeResource, Datastore or ResourcePool"
  }

  $perfMgr = Get-View (Get-View ServiceInstance).content.perfManager

  # Create performance counter hashtable
  $pcTable = New-Object Hashtable
  $keyTable = New-Object Hashtable
  foreach($pC in $perfMgr.PerfCounter){
    if($pC.Level -ne 99){
      if(!$pctable.containskey($pC.GroupInfo.Key + "." + $pC.NameInfo.Key + "." + $pC.RollupType)){
        $pctable.Add(($pC.GroupInfo.Key + "." + $pC.NameInfo.Key + "." + $pC.RollupType),$pC.Key)
        $keyTable.Add($pC.Key, $pC)
      }
    }
  }

  # Test for a valid $Interval
  if($Interval.ToString().Split(" ").count -gt 1){
    Throw "Only 1 interval allowed."
  }

  $intervalTab = @{"RT"=$null;"HI1"=0;"HI2"=1;"HI3"=2;"HI4"=3}
  $dsValidIntervals = "HI2","HI3","HI4"
  $intervalIndex = $intervalTab[$Interval]

  if($EntityType -ne "datastore"){
    if($Interval -eq "RT"){
      $numinterval = 20
    }
    else{
      $numinterval = $perfMgr.HistoricalInterval[$intervalIndex].SamplingPeriod
    }
  }
  else{
    if($dsValidIntervals -contains $Interval){
      $numinterval = $null
      if(!$Start){
        $Start = (Get-Date).AddSeconds($perfMgr.HistoricalInterval[$intervalIndex].SamplingPeriod - $perfMgr.HistoricalInterval[$intervalIndex].Length)
      }
      if(!$Finish){
        $Finish = Get-Date
      }
    }
    else{
      Throw "-Interval parameter $Interval is invalid for datastore metrics."
    }
  }

  # Test if QueryMetrics is given
  if($QueryMetrics){
    $metrics = $perfMgr.QueryAvailablePerfMetric($Entity.MoRef,$null,$null,$numinterval)
    $metricslist = @()
    foreach($pmId in $metrics){
      $pC = $keyTable[$pmId.CounterId]
      $metricslist += New-Object PSObject -Property @{
        Group = $pC.GroupInfo.Key
        Name = $pC.NameInfo.Key
        Rollup = $pC.RollupType
        Id = $pC.Key
        Level = $pC.Level
        Type = $pC.StatsType
        Unit = $pC.UnitInfo.Key
      }
    }
    return ($metricslist | Sort-Object -unique -property Group,Name,Rollup)
  }

  # Test if start is valid
  if($Start -ne $null -and $Start -ne ""){
    if($Start.gettype().name -ne "DateTime") {
      Throw "-Start parameter should be a DateTime value"
    }
  }

  # Test if finish is valid
  if($Finish -ne $null -and $Finish -ne ""){
    if($Finish.gettype().name -ne "DateTime") {
      Throw "-Start parameter should be a DateTime value"
    }
  }

  # Test start-finish interval
  if($Start -ne $null -and $Finish -ne $null -and $Start -ge $Finish){
    Throw "-Start time should be 'older' than -Finish time."
  }

  # Test if stat is valid
  $unitarray = @()
  $InstancesList = @()

  foreach($st in $Stat){
    if($pcTable[$st] -eq $null){
      Throw "-Stat parameter $st is invalid."
    }
    $pcInfo = $perfMgr.QueryPerfCounter($pcTable[$st])
    $unitarray += $pcInfo[0].UnitInfo.Key
    $metricId = $perfMgr.QueryAvailablePerfMetric($Entity.MoRef,$null,$null,$numinterval)

    # Test if QueryInstances in given
    if($QueryInstances){
      $mKey = $pcTable[$st]
      foreach($metric in $metricId){
        if($metric.CounterId -eq $mKey){
          $InstancesList += New-Object PSObject -Property @{
            Stat = $st
            Instance = $metric.Instance
          }
        }
      }
    }
    else{
      # Test if instance is valid
      $found = $false
      $validInstances = @()
      foreach($metric in $metricId){
        if($metric.CounterId -eq $pcTable[$st]){
          if($metric.Instance -eq "") {$cInstance = '""'} else {$cInstance = $metric.Instance}
          $validInstances += $cInstance
          if($Instance -eq $metric.Instance){$found = $true}
        }
      }
      if(!$found){
        Throw "-Instance parameter invalid for requested stat: $st.`nValid values are: $validInstances"
      }
    }
  }
  if($QueryInstances){
    return $InstancesList
  }

  $PQSpec = New-Object VMware.Vim.PerfQuerySpec
  $PQSpec.entity = $Entity.MoRef
  $PQSpec.Format = "normal"
  $PQSpec.IntervalId = $numinterval
  $PQSpec.MetricId = @()
  foreach($st in $Stat){
    $PMId = New-Object VMware.Vim.PerfMetricId
    $PMId.counterId = $pcTable[$st]
    if($Instance -ne $null){
      $PMId.instance = $Instance
    }
    $PQSpec.MetricId += $PMId
  }
  $PQSpec.StartTime = $Start
  $PQSpec.EndTime = $Finish
  if($MaxSamples -eq 0 -or $numinterval -eq 20){
    $PQSpec.maxSample = $null
  }
  else{
    $PQSpec.MaxSample = $MaxSamples
  }
  $Stats = $perfMgr.QueryPerf($PQSpec)

  # No data available
  if($Stats[0].Value -eq $null) {return $null}

  # Extract data to custom object and return as array
  $data = @()
  for($i = 0; $i -lt $Stats[0].SampleInfo.Count; $i ++ ){
    for($j = 0; $j -lt $Stat.Count; $j ++ ){
      $data += New-Object PSObject -Property @{
        CounterId = $Stats[0].Value[$j].Id.CounterId
        CounterName = $Stat[$j]
        Instance = $Stats[0].Value[$j].Id.Instance
        Timestamp = $Stats[0].SampleInfo[$i].Timestamp
        Interval = $Stats[0].SampleInfo[$i].Interval
        Value = $Stats[0].Value[$j].Value[$i]
        Unit = $unitarray[$j]
        Entity = $Entity.Name
        EntityId = $Entity.MoRef.ToString()
      }
    }
  }
  if($MaxSamples -eq 0){
    $data | Sort-Object -Property Timestamp -Descending
  }
  else{
    $data | Sort-Object -Property Timestamp -Descending | select -First $MaxSamples
  }
}

Annotations

Line 49: Determine the type of entity that was passed to the function

Line 62-71: Create a hash table with all the available metrics and their metricId

Line 79: Datastore statistics are only available from Historical Interval 2 (HI2) onwards.

Line 82-103: Datastore metrics don’t seem to like the intervalId property. To avoid this problem, the script transforms the interval to a Start and Finish time, provided there were no explicit Start or Finish parameters passed.

Line 94: For the Start time, the script substracts 1 interval duration from the length of the historical interval.

Line 106-122: These lines handle the QueryMetrics switch

Line 148-150: Test if a valid metric was passed on the Stat parameter

Line 156-166: These lines handle the QueryInstances switch

Line 167-181: Check if a valid Instance was passed

Line 187-207: Construct the PerfQuerySpec object

Line 208: The actual call of the QueryPerf method

Line 230-235: Handle the presence of the MaxSamples parameter

Sample runs

Let’s start by investigating which metrics are available for a datastore.

$ds = (Get-Datastore MyDS).Extensiondata
Get-Stat2 -entity $ds -interval "HI2" -QueryMetrics | ft -AutoSize

The result of this call is a list with the available metrics.

Next let’s see what instances are available.

Get-Stat2 -entity $ds -interval "HI2" -stat "disk.capacity.latest" -QueryInstances

As we already learned from the table above, this metric only has a “” instance.

But the disk.used.latest metric has 3 different types of instances.

Note that the VMid numbers that you see in this list do not necessarily mean that these virtual machines currently have files on the specific datastore. Remember that we are looking at historical data! The only way to know what is currently on the datastore, is to get the metrics and check if there are actual values returned for a specific VMid. If the value is -1, there were no files from that virtual machine on the datastore at that specific time.

Retrieving the actual statistical data is easy now.

Get-Stat2 -entity $ds -stat "disk.used.latest" -interval "HI2" -Instance "49364" -MaxSamples 2

As we expected, this gives us the statistical data for a specific virtual machine on the datastore.

The original question, how to reproduce the specific graph that is available in the vSphere Client, becomes quite straightforward.

$metrics = "disk.capacity.latest","disk.provisioned.latest",
            "disk.used.latest"
$report = Get-Stat2 -entity $ds -stat $metrics -interval "HI2" | `
Sort-Object -Property Timestamp | `
Group-Object -Property Timestamp | %{
  New-Object PSObject -Property @{
    Timestamp = $_.Name
    "Capacity (GB)" = [Math]::Round(($_.Group | `
	    where {$_.CounterName -eq "disk.capacity.latest"}).Value/1MB,2)
    "Allocated (GB)" = [Math]::Round(($_.Group | `
	    where {$_.CounterName -eq "disk.provisioned.latest"}).Value/1MB,2)
    "Used (GB)" = [Math]::Round(($_.Group | `
	    where {$_.CounterName -eq "disk.used.latest"}).Value/1MB,2)
  }
}
$report | Export-Csv "C:\DS-stats.csv" -NoTypeInformation -UseCulture

This produces a nice CSV file with the values from the graph.

Notice how you can clearly see the addition of an 8 GB virtual disk, which we also saw in the screenshot at the top of this post.