PowerCLI & vSphere statistics – Part 3 – Instances

In PowerCLI & vSphere statistics – Part 1 – The basics I briefly mentioned instances. In this post I’ll go a bit deeper into that subject.

And to demonstrate it all I will use part of the esxtop post on Yellow Bricks. In that post Duncan compiled, from various sources, a number of “common sense” thresholds that you can use in esxtop to show you possible problems with your hosts and/or guests.

Since I’m not sitting 24/7 behind an ESX/ESXi console, I looked for a way to let PowerCLI/PowerShell do that for me 😉

First some “watch out” remarks for the solution I came up with.

• esxtop uses some metrics that are not available via the Get-Stat cmdlet. The script doesn’t aim to replace esxtop to pinpoint potential problems.
• esxtop uses a 5 second interval. The Realtime interval with 20 seconds is the shortest available and Historical Interval 1 is even 5 minutes. Consequentially short-lived spikes will hardly be visible in the report the script produces.

Does this mean the script has no real value ?

Not really, it will

• show you possible problem areas in your environment with a rough indication of the time when they took place. This will help you in deciding when to look at the details with esxtop.

• It will also come in handy when someone drops by your office and claims their guest had a problem half an hour ago 😉

From the tresholds Duncan published in his article I will, in this post, only look at the disk-related metrics.

First check, can we get all the metrics that esxtop shows ?

Yes, we can get most of them.

To start with I compiled a worksheet with the esxtop fields and the corresponding metrics. As you can notice, some esxtop fields require calculations when we try to emulate them with performance metrics.

Luckily the disk-related values all have corresponding metrics.

As you probably know most of the disk metrics come with an instance that is displayed in the vSphere client as the Identifier field.

If your have multi-pathing, your LUNs/disks can be reached over more than one path. Which path is taken depends on what you defined in the Path selection policy. It can be Fixed, Round robin or Most recently used.

In esxtop you can  see the values per adapter

but also per disk device

In this script I will try to emulate the adapter view. Since the values returned by the Get-Stat cmdlet are per instance, the script will have to add the values of all instances together to emulate the esxtop adapter view.

The user should be able to specify the adapter(s) (vmhba2 for example in the screenshot) he is interested in. It is up to the script to find out which LUN instances pass through the adapter(s).

Unfortunately the Get-ScsiLun and Get-ScsiLunPath cmdlets, as far as I know, don’t allow you to make the link between an adapter  and its LUN instances. That means we will have to dive into the SDK.

For an ESX host the HostStorageDeviceInfo object allows you to find these links. It’s a matter of following the links, using the keys and placing all the instances in one or more arrays.

As a side-remark, in the test setup I used a “Fixed” Path Selection policy. This is done to show that different instances (devices) can have a path over different adapters. In reality I normally use the “Round robin” policy.

The problem with the “Round robin” policy is that all paths will be marked as “active“. That means that the “adapter” concept becomes irrelevant. For example if you run the script with two adapter selected, all the instances will be mapped to the first adapter you specified.

Annotations

Line 1-2: the input, specify a host and the adapters you’re interested in

Line 3-5: the metrics that are needed to map the esxtop fields

Line 10-11: to have any practical value the metrics should be requested from the Realtime interval.

Line 13-32: gets the “active” instances for each adapter that was selected

Line 38: the reporting array is composed of two nested hash tables. I used hash tables because that makes it easy to find back the entries (by key).

Line 39: the outer hash table uses the adaptername as the key

Line 53: the inner hash table uses the timestamp as the key

Line 71-77: all the returned metrics for a specific timestamp are stored (as a summation) in the $row variable

Line 83-85: post processing of some of the acquired numbers

Line 88: a CSV file per adapter

The script produces one CSV file per adapter you specified. In the CSV file you see the values for the metrics (same headers as esxtop). Unfortunately I wasn’t able to stress my SAN components sufficiently to show some higher numbers for the latencies and to have some non-zero number in the QUED column.

Note that you most probably will have to change the format of the Time column in Excel to see the seconds.

To conclude, do not use this to replace esxtop ! But use it to see some trends, repeating patterns….