Thick to Thin with PowerCLI and the SDK

One of the interesting new features in vSPhere is vStorage Thin Provisioning. I’m not going to explain what Thin Provisioning is all about. For that you can consult several knowledgeable blogs (for example this entry on Virtual Geek) and/or books (for example Scott Lowe’s excellent Mastering VMware vSphere 4.0).

The problem I have with Thin Provisioning, is that there are a lot of existing guests and templates out there that were created with Thick VMDKs in the past. The only documented way I could find to convert these Thick VMDKs to Thin VMDKs was to use svMotion from the vSphere client.

Now that is not a solution I want to (or can) use in an automated environment.I could of course ’emulate’ the svMotion process in a PowerShell script with the RelocateVM method, but that is in my opinion not really an ideal solution.

Luckily there is the VirtualDiskManager with some handy methods like the CopyVirtualDisk_Task method.

This method combined with the ReconfigVM_Task method ultimately gave me a working solution. With the following script I succeeded in converting a Thick VMDK into a Thin VMDK in place. My “in place” statement is of course relative, you have to have enough free space in the datastore to create a Thin version of the VMDK besides the Tick version of the VMDK. Once the reconfiguration is done you can safely remove the Thick VMDK file(s).

The guest has to be powered off for the following script to work !

$vmName = "guest-name"
$vCenter = "vCenter-hostname"
$esxAccount = "ESX-account"
$esxPasswd = "ESX-password"
$vcHost = Connect-VIServer -Server $vCenter
$vmImpl = Get-VM $vmName
if($vmImpl.PowerState -ne "PoweredOff"){
Write-Host "Guest must be powered off to use this script !" -ForegroundColor red
exit
}
$vm = $vmImpl | Get-View
$esxName = (Get-View $vm.Runtime.Host).Name
# For Virtual Disk Manager we need to connect to the ESX server
$esxHost = Connect-VIServer -Server $esxName -User $esxAccount -Password $esxPasswd
$vDiskMgr = Get-View -Id (Get-View ServiceInstance).Content.VirtualDiskManager
$firstHD = $true
$vm.Config.Hardware.Device | where {$_.GetType().Name -eq "VirtualDisk"} | % {
if(!$_.Backing.ThinProvisioned){
$dev = $_
$srcName = $dev.Backing.FileName
$dstName = $srcName.Replace("/","/thin_")
$esx = Get-View $vm.Runtime.Host
$srcDC = Get-Datacenter | Get-View
$spec = New-Object VMware.Vim.VirtualDiskSpec
$controller = $vm.Config.Hardware.Device | where {$_.Key -eq $dev.ControllerKey}
switch($controller.GetType().Name){
"VirtualBusLogicController" {$adapter = "busLogic"}
"VirtualIDEController" {$adapter = "ide"}
"VirtualLsiLogicController" {$adapter = "lsiLogic"}
"VirtualLsiLogicSASController" {$adapter = "lsiLogic"}
"ParaVirtualSCSIController" {$adapter = ""}
"Default"{
Write-Host "Unknown controller type"
exit
}
}
$spec.adapterType = $adapter
$spec.diskType = "thin"
$taskMoRef = $vDiskMgr.CopyVirtualDisk_Task($srcName, $srcDC.MoRef, $dstName, $srcDC.MoRef, $spec, $false)
$task = Get-View $taskMoRef
while("running","queued" -contains $task.Info.State){
$task.UpdateViewData("Info")
}
if($firstHD){
$specHD = New-Object VMware.Vim.VirtualMachineConfigSpec
$firstHD = $false
}
$deviceMod = New-Object VMware.Vim.VirtualDeviceConfigSpec
$deviceMod.device = $dev
$deviceMod.device.Backing.FileName = $dstName
$deviceMod.Operation = "edit"
$specHD.deviceChange += $deviceMod
}
}
Disconnect-VIServer -Server $esxHost -Confirm:$false
# For the reconfiguration of the VM we connect to the vCenter
Connect-VIServer -Server $vCenter
$taskMoRef = $vm.ReconfigVM_Task($specHD)
$task = Get-View $taskMoRef
while("running","queued" -contains $task.Info.State){
$task.UpdateViewData("Info")
}

$vmName = "guest-name"

$vCenter = "vCenter-hostname"

$esxAccount = "ESX-account"

$esxPasswd = "ESX-password"

$vcHost = Connect-VIServer -Server $vCenter

$vmImpl = Get-VM $vmName

if($vmImpl.PowerState -ne "PoweredOff"){

Write-Host "Guest must be powered off to use this script !" -ForegroundColor red

exit

}

$vm = $vmImpl | Get-View

$esxName = (Get-View $vm.Runtime.Host).Name

# For Virtual Disk Manager we need to connect to the ESX server

$esxHost = Connect-VIServer -Server $esxName -User $esxAccount -Password $esxPasswd

$vDiskMgr = Get-View -Id (Get-View ServiceInstance).Content.VirtualDiskManager

$firstHD = $true

$vm.Config.Hardware.Device | where {$_.GetType().Name -eq "VirtualDisk"} | % {

if(!$_.Backing.ThinProvisioned){

$dev = $_

$srcName = $dev.Backing.FileName

$dstName = $srcName.Replace("/","/thin_")

$esx = Get-View $vm.Runtime.Host

$srcDC = Get-Datacenter | Get-View

$spec = New-Object VMware.Vim.VirtualDiskSpec

$controller = $vm.Config.Hardware.Device | where {$_.Key -eq $dev.ControllerKey}

switch($controller.GetType().Name){

"VirtualBusLogicController" {$adapter = "busLogic"}

"VirtualIDEController" {$adapter = "ide"}

"VirtualLsiLogicController" {$adapter = "lsiLogic"}

"VirtualLsiLogicSASController" {$adapter = "lsiLogic"}

"ParaVirtualSCSIController" {$adapter = ""}

"Default"{

Write-Host "Unknown controller type"

exit

}

$spec.adapterType = $adapter

$spec.diskType = "thin"

$taskMoRef = $vDiskMgr.CopyVirtualDisk_Task($srcName, $srcDC.MoRef, $dstName, $srcDC.MoRef, $spec, $false)

$task = Get-View $taskMoRef

while("running","queued" -contains $task.Info.State){

$task.UpdateViewData("Info")

}

if($firstHD){

$specHD = New-Object VMware.Vim.VirtualMachineConfigSpec

$firstHD = $false

}

$deviceMod = New-Object VMware.Vim.VirtualDeviceConfigSpec

$deviceMod.device = $dev

$deviceMod.device.Backing.FileName = $dstName

$deviceMod.Operation = "edit"

$specHD.deviceChange += $deviceMod

}

Disconnect-VIServer -Server $esxHost -Confirm:$false

# For the reconfiguration of the VM we connect to the vCenter

Connect-VIServer -Server $vCenter

$taskMoRef = $vm.ReconfigVM_Task($specHD)

$task = Get-View $taskMoRef

while("running","queued" -contains $task.Info.State){

$task.UpdateViewData("Info")

}

Annotation:

Line 3-4: provide an ESX account and password with root access on the ESX server

Line 8-11: the guest must be powered off for this script to work

Line 16: the VirtualDiskManager methods can only be used when connected to an ESX

Line 20: this loops through all devices and finds the Thick VMDKs

Line 24: the backing filename of the Thin VMDK is the same as the backing filename of the Thick VMDK with a “thin_” prefix

Line 29-39: this Switch construct provides the correct adapter name. Take note that the adapter names are case-sensitive

Line 41: the diskType property allows to specify what VMDK format the copy should have

Line 43: the CopyVirtualDisk_Task method creates a Thin copy of the Thick VMDK

Line 49-59: this construct creates the specification for the ReconfigVM_Task method

Line 64: the new Thin VMDKs are now connected (as backing file(s)) to the VM

Note that the script doesn’t remove the Thick VMDK files from the VM’s directory. This is a safety precaution should something go wrong with the conversion.

Note that this procedure was tested and tried in my environment and seems to work correctly. If you are going to use this script, first try it out in your test environment and make sure the guests with the Thin VMDKs are working correctly !

15 Comments

Ruben

June 15, 2011 at 21:05

@antize

I have created the script that you were looking for. It was specifically designed for LabManager but can be used in any ESX/VM/vmware environment in general.

We achieved huge savings using it in all our centers worldwide.

The script is called makeThin

https://vmutils.blogspot.com/2011/06/automatic-thinning-of-virtual-disks.html

LucD

June 15, 2011 at 21:34

Hi Ruben, thanks for the link. Impressive script and documentation.

antize

September 23, 2010 at 08:15

Hi Luc,

This solution works for VMs that vcenter knows about, however is there any way via API to search through datastore directories looking for VMDKs that are thick?

We have a Lab Manager system that has a ton of folders created with VMs on the datastores but most of which are not deployed on vcenter. It would be great to recurse through the datastores directly looking for VMDKs that are thick provisioned via API. Is this possible?

LucD

September 23, 2010 at 14:55

@antize That is possible.
We make a function of the script and then use the Get-Datastore, Get-VM and Get-Harddisk cmdlets to find guests that have “Thick” hard disks.
For those guests we call the function, which I called Set-ThinDisk.
In the sample all datastores are examined, but you can use the -Name parameter on the Get-Datastore or a Where-Object construction to select specific datastores.
Let me know how it works out ?
function Set-ThinDisk{ param($vmName,$vCenter,$esxUser,$esxPassword)


	$vcHost = Connect-VIServer -Server $vCenter

$vmImpl = Get-VM $vmName

if($vmImpl.PowerState -ne "PoweredOff"){

Write-Host "Guest must be powered off to use this script !" -ForegroundColor red

exit

}

$vm = $vmImpl | Get-View

$esxName = (Get-View $vm.Runtime.Host).Name
# For Virtual Disk Manager we need to connect to the ESX server

$esxHost = Connect-VIServer -Server $esxName -User $esxAccount -Password $esxPasswd
	$vDiskMgr = Get-View -Id (Get-View ServiceInstance).Content.VirtualDiskManager

$firstHD = $true

$vm.Config.Hardware.Device | where {$_.GetType().Name -eq "VirtualDisk"} | % {

if(!$_.Backing.ThinProvisioned){

$dev = $_

$srcName = $dev.Backing.FileName

$dstName = $srcName.Replace("/","/thin_")

$esx = Get-View $vm.Runtime.Host

$srcDC = Get-Datacenter | Get-View

$spec = New-Object VMware.Vim.VirtualDiskSpec

$controller = $vm.Config.Hardware.Device | where {$_.Key -eq $dev.ControllerKey}

switch($controller.GetType().Name){

"VirtualBusLogicController" {$adapter = "busLogic"}

"VirtualIDEController" {$adapter = "ide"}

"VirtualLsiLogicController" {$adapter = "lsiLogic"}

"VirtualLsiLogicSASController" {$adapter = "lsiLogic"}

"ParaVirtualSCSIController" {$adapter = ""}

"Default"{

Write-Host "Unknown controller type"

exit

}

}

$spec.adapterType = $adapter

$spec.diskType = "thin"
			$taskMoRef = $vDiskMgr.CopyVirtualDisk_Task($srcName, $srcDC.MoRef, $dstName, $srcDC.MoRef, $spec, $false)

$task = Get-View $taskMoRef

while("running","queued" -contains $task.Info.State){

$task.UpdateViewData("Info")

}
			if($firstHD){

$specHD = New-Object VMware.Vim.VirtualMachineConfigSpec

$firstHD = $false

}

$deviceMod = New-Object VMware.Vim.VirtualDeviceConfigSpec

$deviceMod.device = $dev

$deviceMod.device.Backing.FileName = $dstName

$deviceMod.Operation = "edit"

$specHD.deviceChange += $deviceMod

}

}

Disconnect-VIServer -Server $esxHost -Confirm:$false
# For the reconfiguration of the VM we connect to the vCenter

Connect-VIServer -Server $vCenter

$taskMoRef = $vm.ReconfigVM_Task($specHD)

$task = Get-View $taskMoRef

while("running","queued" -contains $task.Info.State){

$task.UpdateViewData("Info")

}

}
$vmName = "guest-name"

$vCenter = "vCenter-hostname"

$esxAccount = "ESX-account"

$esxPasswd = "ESX-password"
$serverMode = (Get-PowerCLIConfiguration )."Default Server Mode"

Set-PowerCLIConfiguration -DefaultVIServerMode "Single"
foreach($ds in Get-Datastore){

foreach($vm in (Get-VM -Datastore $ds)){

if(Get-HardDisk -VM $vm | where{$_.StorageFormat -eq "Thick"}){

Set-ThinDisk $vm $vCenter $esxAccount $esxPasswd

}

}

}

Set-PowerCLIConfiguration -DefaultVIServerMode $serverMode

Alasdair

August 10, 2010 at 22:42

Hi Luc,

Did you ever have any luck with this? I’ve had a trawl through hte new version of the toolkit, but it doesn’t look like VMware have added a thin disk option.

Cheers….

Steve Ochry

March 8, 2010 at 16:35

@LucD
I was provided a workaround in C#. Essentially, the workaround is to create a propertyCollector and retrieve the properties on the httpNfcLeaseMor object. Looping through the properties, picking out the ones you need. Code pasted below in C#, for what it is worth.

private void RetrieveHttpNfcLeaseProperties(ManagedObjectReference httpNfcLeaseMor,
out HttpNfcLeaseInfo httpNfcLeaseInfo,
out HttpNfcLeaseState httpNfcLeaseState,
out int httpNfcLeaseInitializeProgress,
out LocalizedMethodFault httpNfcLeaseError)
{
httpNfcLeaseInfo = null;
httpNfcLeaseState = HttpNfcLeaseState.error;
httpNfcLeaseInitializeProgress = 0;
httpNfcLeaseError = null;

PropertyCollector propertyCollector =
new PropertyCollector(vimClient, vimClient.ServiceContent.PropertyCollector);

ObjectContent[] objectContent = propertyCollector.RetrieveProperties(
new[] {ViewBase.GetPropertyFilterSpec(httpNfcLeaseMor)});

if ((objectContent != null) && (objectContent.Length > 0))
{
foreach (DynamicProperty property in objectContent[0].PropSet)
{
switch (property.Name)
{
case “info”:
httpNfcLeaseInfo = (HttpNfcLeaseInfo) property.Val;
break;

case “state”:
string state;

if (property.Val is XmlNode[])
{
state = ((XmlNode[]) property.Val)[1].Value;
}
else
{
state = property.Val.ToString();
}

httpNfcLeaseState = (HttpNfcLeaseState)Enum.Parse(typeof(HttpNfcLeaseState), state);
break;

case “initializeProgress”:
httpNfcLeaseInitializeProgress = (int) property.Val;
break;

case “error”:
httpNfcLeaseError = (LocalizedMethodFault) property.Val;
break;
}
}
}
}

LucD

March 8, 2010 at 17:27

Thanks for sharing that Steve.
I’ll have a look if I can package that solution in a PowerShell script.

Steve Ochry

March 1, 2010 at 22:43

@LucD
I am bumping into this very issue on the C# side of things. Did you ever get anywhere with a work around?

LucD

March 1, 2010 at 23:00

Hi Steve, no I’m afraid I didn’t find a workaround for this yet 🙁
But it’s still on my list of things to investigate further.

Aladair Carnie

January 31, 2010 at 12:55

Thanks for the update.

Aladair Carnie

January 31, 2010 at 10:04

Hi Luc,

Any luck getting this to work?

Al…….

LucD

January 31, 2010 at 10:42

I encountered a problem while testing this. The HttpNfcLease MoRef the ImportVApp method returns has a problem.
One of the PowerCLI developers confirmed that there is a “…bug in xml deserialisation of NfcLeaseState object.”
I’m still looking if there is a way around the problem.

Aladair Carnie

January 18, 2010 at 22:11

This would be useful for restoring VMs without the need of third party systems or using the DR Appliance. The only issue I’ve had is the network adapter which defaults to a standard vmxnet adapter, but as long as it’s removed and replaced with a vmxnet3 card before powering it on (which can easily be scripted), the VM boots up and picks up it’s original IP info and works perfectly.

Aladair Carnie

January 18, 2010 at 17:20

Hi Luc,

Using the powershell toolkit, is there a way of importing an OVF and specifying the disk type as Thin? I’ve had a dig around in the kit and can only find the vApp cmdlet in the main toolkit. I assume that it will import an OVF, but there is no option for disk provisioning, however I did notive that the OVF Tool has those options.

Any guidence will as always be much appreciated.

Ali………

LucD

January 18, 2010 at 20:48

Very intriguing idea.
A quick look at the VirtualApp methods in the SDK makes me think it should be possible.
Let me play with the concept in my lab. Might even produce a new post when I get it to work.
Luc.

Thick to Thin with PowerCLI and the SDK

15 Comments

antize

Steve Ochry

Steve Ochry

Leave a Reply Cancel reply