Get the maximum IOPS

A quick post that is triggered by a tweet from @GernotNusshall I saw passing today. He wanted to know how to find the maximum IOPS values over the last 5 minutes for a number of VMs. The IOPS values are readily available from the vSphere statistics but the problem is that the values are returned as summation values over the measuring interval and that you have a read and a write value.

An ideal job for PowerShell to get the values Gernot was after.


Update April 26th 2011:  following changes are made

  • The output will also show the datastorename
  • A suggestion from Glenn Sizemore, the script uses the New-Object cmdlet to produce the output

Update June 29th 2011: A version that retrieves the average read and write IOPS was added.

Update August 6th 2011: Yet another version was added, now one that supports NFS datastores.

The script

Annotations

Line 1: The 2 metrics that measure IOPS. The metrics return the number of read and write operations over the measurement interval.

Line 5: The script stores all powered on guests in an array. The Get-VM statement can be adapted to return just the guests for which you want the IOPS numbers.

Line 6: The script uses only 1 Get-Stat cmdlet for all VMs and all requested metrics. This will optimise the use of resources on the vCenter Server to retrieve the metrics.

Line 7: To avoid hard-coding the duration of the interval the script retrieves the value from the first returned measurement.

Line 9-14: The script creates a hash table that contains all the canonical names of the LUNs used for the datastores on which the virtual machine are stored.

Line 16: The Group-Object cmdlet does all the hard work. It will group the statistical measurements just the way we would like them

Line 17: The New-Object cmdlet is used to create the output object.

Line 20-23: Calculates the maximum IOPS number by adding the read and write operations together. Since the IOPS value is ‘per second’, the script divides the summation value by the length, in seconds, of the interval.

Line 24: The datastorename is retrieved from the hash table with the canonical name of the LUN as the key.

A sample run

The script produces output similar to this

The sample output shows the only disadvantage, in my opinion, of using the New-Object cmdlet, you have no control over the order of the properties in the object.

The Read/Write Average version

A reader asked if it was possible to retrieve the average read and write IOPS as well. The following script should do the trick

The NFS datastore version

As one of my readers noticed, the original script didn’t handle NFS based datastores.
The following version remediates that shortcoming.

60 Comments

    Jerry

    Luc, what’s the real difference between…?

    disk.numberwriteaveraged.average
    virtualdisk.numberwriteaveraged.average
    datastore.numberwriteaveraged.average

    Check this out:

    (get-vm VM01 | Get-Stat -Stat datastore.numberwriteaveraged.average -Start (Get-Date).AddMinutes(-30) -IntervalMins 1 | Measure-Object Value -Average).Average
    122.188888888889

    (get-vm VM01 | Get-Stat -Stat disk.numberwriteaveraged.average -Start (Get-Date).AddMinutes(-30) -IntervalMins 1 | Measure-Object Value -Average).Average
    121.644444444444

    (get-vm VM01 | Get-Stat -Stat virtualdisk.numberwriteaveraged.average -Start (Get-Date).AddMinutes(-30) -IntervalMins 1 | Measure-Object Value -Average).Average
    10.9333333333333

    I want the average write IOPS of some VM in the last 30 minutes.
    Datastore? Disk? VirtualDisk?

      LucD

      These 3 metrics produce the average IO/sec over the interval corresponding with the Historical Interval, determined by the Start and Finish parameters.
      The Entity property shows which component is involved.

      – datastore.numberwriteaveraged.average : the metric for a Datastore. Entity is a Datastore.
      – disk.numberwriteaveraged.average: the metric for a LUN. Should be equal to the previous one, provided the Datastore is composed of only 1 LUN. Entity is the LUN’s CanonicalName.
      – virtualdisk.numberwriteaveraged.average: the metric per VMDK. Entity is the VMDK

      If you are interested in meaningfull numbers, specific for a VM, I would use the virtualdisk.numberwriteaveraged.average.

      If you want to see if a datastore is over-utilised, I would use the datastore.numberwriteaveraged.average.
      If the Datastore is spread over more than 1 LUN, I would also look at disk.numberwriteaveraged.average to check if the issue is due to a specific LUN.

      If you want to see the numbers for a 30 min interval, use the Start and Finish parameters on the Get-Stat cmdlet.
      Note that due to aggregation the Interval becomes longer, and thus the returned numbers less meaningfull to see/find outliers.

      Hope this helps.

    Gianfranco

    Hello,
    is it possible to avoid to truncate the of the disk name and datastore ?
    script is Get max IOPS. Below the output.
    IOPSMax Disk Datastore VM
    ——- —- ——— —
    695.4 naa.60002ac00000… DS10-WIN-CLUS… VIRTNAME01
    0.8 naa.60002ac00000… RO-PRD-ODB-RO-3P… VIRTNAME02
    thanks
    GB

      LucD

      Hi Gianfranco,
      That truncation is the result of PowerShell trying to get the objects in an optimal way on screen.
      The problem is that this screen space (line length) is limited, and not all information might fit on one line. Hence the truncation.
      There are a number of options to get more on a line:

      -) Use a Format-List -AutoSize cmdlet
      -) Use a Format-Table to get the properties on separate lines
      -) Save the results to a file, with for example an Export-Csv

      I hope this helps to get all the info.

    Markus Kraus

    Still a great article. Thanks!

    chris

    Just brilliant!

    Matt S.

    Luc, first off, thank you so much for this, and all, scripts you provide! Virtual high-five to you!

    Nearly 5 years after the last comment, I have a new question:
    One thing I was thinking of within the NFS version of this script is that the numbers given to us by vCenter are already the average of those reads/writes over that interval, right? So, if you want to actually get the real Max number of averaged reads/writes for that vmdk device, removing the part from Line 23 where it divides by $interval would be required, yes?

      LucD

      That’s correct, the values we get in the “average” counters are already the average over the interval we are looking at.
      Depending on the Statistics Level for the interval, you can also ask for the maximum and minimum values, but again these will be the average over maxima/minima of the interval.
      The division I do in line 23 is to get an IOPS value, which means “per second”. Hence the division by the duration of the interval.

        Matt S.

        How I understand that value is you’re already getting IOPS but it’s already averaged it out for you over that interval since it’s not a summation of the total amount of write operations executed in that interval.
        Ok, let’s say that in a 20 second interval you get a value of 100 writes/sec. That means that over that 20 seconds, you had writes above and below 100 and the average of that is 100. You can assume that your maximum write IOPS over that 20 seconds is going to be well above 100 in order to give you an average of 100. Dividing that number by the interval length will only reduce your average further.

        What would be nice is if the virtualdisk counters on NFS were the same as they are for disk counters on vmfs. I’m not finding anything in the virtualdisk counters list that actually shows a single max IO/sec value. It’s all averages over the interval.

          LucD

          You could be right, let me check that.

          And yes, would be nice to have those same metrics for NFS as well

    Sudheer

    IOPSWriteAvg / IOPSReadAvg

    are referring to what values in the CSV-report.
    Is it % or seconds

    what is threshold limit for both

    Fabio

    rewriting this to work with 5.5 …

    Found out this script:

    https://www.linkedin.com/grp/post/3992597-5937675088978534402

    It gets the statics even if the statistic levels are set to default in my 5.5 vcenters.

    Do you feel these are good numbers?

    Thank you 🙂

    F.

      LucD

      Hi Fabio,
      Both scripts fetch the metrics from the Realtime interval. In that interval all metrics (except for the ones created through aggregation of course) are available, the vCenter statistic levels do not come into play.
      The difference with my script is that I use the actual number of read and writes per interval, not the average (except for the NFS version).
      But both scripts would have to produce approximately the same results.
      My script should be a bit faster since I limit it to 1 call of Get-Stat, and it provides the name of the datastore on which the vDisk resides.

    Rutger

    Great script! What would the “NFS datastore with average IOPS” version look like? Tried to merge the two together without much look.

    Cheers!

    Sam

    LucD can you add the min – max field (with time Stamp) for a period of say to weeks to the Script?

      Nivendran Nair

      Hi LucD,

      Please can you assist, I need a script the monitors 6 different datastores on VMWare 5.5 for a period of 30 days, reporting on 8am-5pm. Report must show Max and Average IOPS. e.g
      Day 1 Monitored Time AVG & Max READ IOPS AVG & MAX Write IOPS
      Day 2 Monitored Time AVG & Max READ IOPS AVG & MAX Write IOPS

      and so on…

      I will appreciate your assistance.

      Thanks

    Daniel Otte

    Hi LucD,

    was wondering if this thread is still monitored, as it is quite old 🙂

    I got this script to work, the NFS version that is… now I would like it to only display scsi0:0 information….can you help me?

    Thanks
    Daniel

      LucD

      Hi Daniel, yes I still see the incoming comments.
      The easiest way to do that would be to use the Instance parameter on the Get-Stat line. That way you will only get back metrics for that specific instance.

    Mandeep

    Hello Luc,

    i am using the below script for disk IOPs calculation, but it is not working if i remove “-Realtime” switch, lets say i am calculating it for last week, please have a look where its going wrong.

    $metrics = “disk.numberwrite.summation”,”disk.numberread.summation”
    $start = (Get-Date).AddDays(-7)

    $report = @()

    $vms = Get-VM THASCCMSRV
    $stats = Get-Stat -Realtime -Stat $metrics -Entity $vms -Start $start
    $interval = $stats[0].IntervalSecs

    $lunTab = @{}
    foreach($ds in (Get-Datastore -VM $vms | where {$_.Type -eq “VMFS”})){
    $ds.ExtensionData.Info.Vmfs.Extent | %{
    $lunTab[$_.DiskName] = $ds.Name
    }
    }

    $report = $stats | Group-Object -Property {$_.Entity.Name},Instance | %{
    New-Object PSObject -Property @{
    VM = $_.Values[0]
    Disk = $_.Values[1]
    IOPSMax = ($_.Group |
    Group-Object -Property Timestamp |

    %{$_.Group[0].Value + $_.Group[1].Value} |
    Measure-Object -Maximum).Maximum / $interval
    IOPSAvg = ($_.Group |
    Group-Object -Property Timestamp |

    %{$_.Group[0].Value + $_.Group[1].Value} | `
    Measure-Object -Average).Average / $interval
    Datastore = $lunTab[$_.Values[1]]
    }
    }

    $report | Export-Csv “C:\IOPSMax-report.csv” -NoTypeInformation -UseCulture -ErrorAction SilentlyContinue

    Matt M

    I’m having a hard time with the script. I’m using the vSphere PowerCLI, I connect to a vSphere host and execute the script. It repeats this line over and over:

    [vSphere PowerCLI] C:\Scripts\vmware\PowerCLI\VC02> C:\Scripts\vmware\PowerCLI\V
    C02\get_iops.ps1
    New-Object : A parameter cannot be found that matches parameter name ‘Property’
    .
    At C:\Scripts\vmware\PowerCLI\VC02\get_iops.ps1:17 char:34
    + New-Object PSObject -Property <<<< @{
    New-Object : A parameter cannot be found that matches parameter name 'Property'
    .

    I copied the code using the widget and it looks right to my eye. What could be missing?

      LucD

      Hi Matt, I suspect you might be using PowerShell v1.
      The Property parameter was introduced in PowerShell v2.
      Do a

      $PSVersionTable

      The PSVersion property should be 2.0 or 3.0.

    sandesh

    Hi LucD,

    How can I get disk size details in same script

    KennyF

    Hi LucD,

    Thank you so much for the script. It’s exactly what I need to get the necessary information for our DR Site requirements. I’m currently battling to get the same information, but for a specific date/time range. I added the following, but I receive a “Cannot index into a null array” error after I run the script and the csv file is empty. I also removed the “-Realtime” as that would ignore the start/finish options.

    $start = “06 Aug 2012 08:00”
    $finish = “06 Aug 2012 17:00”
    $stats = Get-Stat -Stat $metrics -Entity $vms -Start $start -Finish $finish

    Any help would be greatly appreciated.

    Many thanks,
    Kenny

    Jebaz

    I would like to use The NFS datastore version to get the report for the period of last 30 days. Could you please help me to append the script.

    Sandesh

    @ LucD

    How can I get the disk size information in this script.

    MrG

    Hi Luc,

    In the NFS version of this script is it possible to have another field with the disk capacity? like this:

    IOPSMaxDisk VM Datastore TotalSize

      LucD

      Hi MrG, yes it is.
      See the code in VM Correct DISK IOPS.

    BerreB

    Hi Luc,

    Thx a lot for these scripts. I do have a question to ask you: I’m not sure how to interpret the values that are returned by the NFS script. For instance for one of my VM’s I get 11,5 IOPS Max for one of its vdisks whilst in esxtop CMD/s for this vdisk is constantly between 250 and 450?

    I’m using the NFS version of the script exactly like you posted on this page.

    Thx!
    B.

    Harry

    Hi Lucd, Excellent script and very helpful.
    Need help i wanted to run as schedule task every hour and append the info with time stamp in to an file. How do i achieve that if you could help???

    Many Thanks

      LucD

      Hi Harry, you can use the Windows Scheduler.
      Have a look at Alan’s post called Running a PowerCLI Scheduled task.
      You can add a Timestamp property to the New-Object structure, something like this

      $report = $stats | Group-Object -Property {$_.Entity.Name},Instance | %{
      New-Object PSObject -Property @{
      Timestamp = $start
      VM = $_.Values[0]
      Disk = $_.Values[1]
      IOPSMax = ($_.Group |

      Group-Object -Property Timestamp |

      %{$_.Group[0].Value + $_.Group[1].Value} | `
      Measure-Object -Maximum).Maximum / $interval
      Datastore = $lunTab[$_.Values[1]]
      }
      }

      You can store the results in a CSV file and append the new data.
      You can do something like this

      $oldReport = $null
      if(Test-Path .\report.csv){
      $oldReport += Import-Csv .\report.csv -UseCulture
      }

      $oldReport + $report | Export-Csv .\report.csv -NoTypeInformation -UseCulture

      In PowerShell v3 the Export-Csv cmdlet will have an Append switch, and you don’t have to do tricks like the one above anymore.

    Roderick

    Hey Luc,
    Getting an error I can’t explain:

    Cannot index into a null array.
    At C:\Users\roderick\Desktop\iops.ps1:7 char:20

    Cheers
    Roderick

      LucD

      @Roderick, it looks as if the Get-Stat didn’t return anything.
      Are there any poweredon VMs present, in other words are there any objects in the variable $vms ?

    Doug Youd

    Great script, saved me a bunch of time today….. now time to go investigate some ‘top talkers’. 🙂

    Cheers,
    Doug

    Jbuddy

    @LucD

    Hi LucD,

    Hope you had a nice weekend. I increased the level to 3 and was still getting an error before I realized I needed to add the following switch

    $stats = Get-Stat -Stat $metrics -Entity $vms -Start $start -Finish -Disk

    Now I am getting results.

    Thanks again

      LucD

      @Jbuddy, great. Glad you have it working now.

    Jbuddy

    @LucD

    Hi LucD,

    They were set to 2. So I set it to 3 in the 5min interval duration saving for 1 day. So the original script worked because it had the realtime switch.

    Thanks again
    Thanks again

      LucD

      @Jbuddy, you should now wait till the aggregation jobs have propagated the change. Should take 1 day for the values of the day before to be available.
      But remember that increasing the statistics levels, will also increase the size of your VC database. Monitor the VC database closely in the coming days. And make sure the aggregation jobs complete successfully.

    Jbuddy

    @LucD

    Ah ok thanks again

    Jbuddy

    @LucD
    Hi LucD,

    Thanks again for assisting me with this. I heard at your talk at vmworld you were quick to answer questions on your site :). So I tried finish before and was getting a null result. The actual error message is as follows. The metric counter “disk.numberread.summation” doesn’t exist for entity” My Start and end times are within 3 hours so

    $start = (Get-Date).AddHours(-14) #Friday, October 28, 2011 12:48:50 AM
    $finish = (Get-Date).AddHours(-12) #Friday, October 28, 2011 2:48:50 AM

    Any help is appreciated and again these are great scripts thanks

      LucD

      @Jbuddy, I suspect this is because your statistics level is not set to at least level 3.
      If you look at the disk metrics, you’ll see that numberRead and numberWrite require level 3.
      Check your current settings from the vSphere Client.

    Jbuddy

    Hi LucD,

    Hopefully you are still monitoring this thread. I am using both of your disk stats script regarding IO average and maximums. We have an issue that happens at 1:00am on and off for about an hour. Could we adapt the script to just monitor that time period instead of an entire day? Is this possible with get-stat? I am about average with powershell and I am still wrapping my head around your script.

    Thanks again

      LucD

      @Jbuddy, yes, I still look at comments for this thread 🙂
      You can change the script to use statistical data from a specific time range with the Start and Finish parameters. You can’t use the Realtime parameter in this case, when the specified time interval is further back in time than approx 1 hour.
      The script could look something like this

      $metrics = "disk.numberWrite.summation","disk.numberRead.summation"
      $start = Get-Date -Hour 1 -Minute 0 -Second 0
      $finish = $start.AddHours(1)
      $report = @()

      $vms = Get-VM mmmstd010 | where {$_.PowerState -eq "PoweredOn"}
      $stats = Get-Stat -Stat $metrics -Entity $vms -Start $start -Finish $finish
      $interval = $stats[0].IntervalSecs

      $lunTab = @{}
      foreach($ds in (Get-Datastore -VM $vms | where {$_.Type -eq "VMFS"})){
      $ds.ExtensionData.Info.Vmfs.Extent | %{
      $lunTab[$_.DiskName] = $ds.Name
      }
      }

      $report = $stats | Group-Object -Property {$_.Entity.Name},Instance | %{
      New-Object PSObject -Property @{
      VM = $_.Values[0]
      Disk = $_.Values[1]
      IOPSMax = ($_.Group |

      Group-Object -Property Timestamp |

      %{$_.Group[0].Value + $_.Group[1].Value} | `
      Measure-Object -Maximum).Maximum / $interval
      Datastore = $lunTab[$_.Values[1]]
      }
      }

      $report

      Note that the statistical data you will get this way is aggregated over longer intervals than 20 seconds. So the results will be more flattened out than the results you get from the Realtime interval.

      You can adapt the 2nd script in a similar way.

    Darren Phillips

    Hi Luc

    Would possible to amend this script to get the IOPS per VMDK file per VM.

    Thanks
    Darren

    Happy

    This is great stuff but I need it to work with VMs in a NFS Datastore. Can this be done?

      LucD

      @Happy, sorry for the delay, I completely forgot about this question.
      The script in it’s current form can’t handle NFS datastores.
      I’ll try to come up with something that also does NFS datastores.

    Pawel

    @LucD
    That’s exactly what I was looking for. Thanks.

    Pawel

    Is there a way to get the average read and write ratios? Thanks.

      LucD

      @Pawel, today I added a new version of the script that returns the average read and write IOPS.
      I hope this produces what you were looking for.

    Derek

    @LucD

    Sorry I was unclear. Using your max IOPS script, what if I wanted to add together two NoteProperty values such as MaxIOPS and IOPSAve to create a third noteproperty value called IOPSNumber for each VM (lines 18-24). Of course adding those two specific values makes no sense, but that’s the general concept I can’t quite master in PS. Is that clearer?

    Derek

    @LucD

    LucD,

    How would you perform a math function on existing group-object NoteProperty values and create a new NoteProperty based on that calculated value?

      LucD

      @Derek, I’m not sure that I understand 100% what you want, but this is my interpretation.
      A small test array with a property called Value.
      I then group the elements based on the Value being odd or even.
      This is added as a new property, called newValue, to each element.


      $array = @()
      1..10 | %{
      $array += New-Object -TypeName PSObject -Property @{
      Value = $_
      }
      }

      $newArray = $array | Group-Object -Property {$_.Value % 2} | %{
      $groupValue = if($_.Name -eq 0){"even"}else{"odd"}
      $_.Group | %{
      Add-Member -InputObject $_ -Name NewValue -Value $groupValue `
      -MemberType NoteProperty -PassThru
      }
      }
      $newArray | Sort-Object -Property Value

    Derek

    Thanks for the great script! I used it as a basis for a more VDI oriented report that shows total read/write IOs, read/write ratios, etc.

    https://derek858.blogspot.com/2011/06/powercli-script-to-dump-vm-io-stats.html

      LucD

      You’re welcome. Thanks for sharing your version in your post.

    Paul

    For some reason I’m getting this error message… Any suggestion?

    vSphere PowerCLI] C:\> foreach($ds in (Get-Datastore -VM $vms | where {$_.Type -eq “VMFS”})){
    $ds.ExtensionData.Info.Vmfs.Extent | %{
    $lunTab[$_.DiskName] = $ds.Name
    }
    }
    $report = $stats | Group-Object -Property {$_.Entity.Name},Instance | %{
    New-Object PSObject -Property @{
    VM = $_.Values[0]
    Disk = $_.Values[1]
    IOPSMax = ($_.Group |
    Group-Object -Property Timestamp |

    %{$_.Group[0].Value + $_.Group[1].Value} |
    Measure-Object -Maximum).Maximum / $interval
    Datastore = $lunTab[$_.Values[1]]
    }
    }
    $report | Export-Csv "C:\IOPSMax-report.csv" -NoTypeInformation -UseCulture
    Missing closing ')' in expression.
    At line:11 char:1
    + <<<< Group-Object -Property Timestamp |

    + CategoryInfo : ParserError: (CloseParenToken:TokenId) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : MissingEndParenthesisInExpression

    tuensel2k

    Awsome script!

    Is it possible to also calculate the average iops?

    This would be great addition for my daily storage reporting!

      LucD

      Sure, you can add the following lines to the PSObject properties hash table.


      IOPSAvg = ($_.Group |

      Group-Object -Property Timestamp |

      %{$_.Group[0].Value + $_.Group[1].Value} | `
      Measure-Object -Average).Average / $interval

    Gernot Nusshall

    As i already said on Twitter, thank you! 🙂
    The only thing i don´t get is the convertion from canonical names to datastore names. i “studied” the other post you mentioned but as i already said, i don´t get it. can you point me out in which section of the lun report script you are doing the conversion?

    thanks!

    Gernot

      LucD

      @Gernot, I updated the script. It now shows the datastorename as well.

    Troy Clavell

    Great Stuff! Is there a way to export-csv the three columns?

      LucD

      Thanks Troy.
      Sure, change the last line (20) into something like this

      $report | Export-Csv "C:\IOPSMax-report.csv" -NoTypeInformation -UseCulture

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*

This site uses Akismet to reduce spam. Learn how your comment data is processed.