PowerCLI Collection: Setting RDM PereniallyReserved flag on RDM LUNS used by MSCS Nodes that take a long time to scan or boot

Yeah that’s a lot information in the title, but this is actually what this PowerCLI script will do. Set the PereniallyReserved flag of RDMs storage devices that are shared between multi-server nodes. This happens with Microsoft Fail over cluster across boxes shared RDM’s.

I returned from my vacation and one of the first questions / remarks I got when a team member tried to reboot a maintenanced vSphere 5.1 host is, why does this boot takes long…. While discussing the situation and monitoring the VMkernel log while booting (ALT-F12 on the console), I learned that a VM with a physical shared scsi bus physical RDM setup was shutdown before the maintenance (as online vMotion is not allowed in this situation) and that the boot is hanging on the storage device discovery. It continues on to the next device after a few retries, and hangs when encountering a same sort of storage device. I remember this situation from a previous encounter and redirected the customer to Knowledge Base article http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1016106&src=vmw_so_vex_pheld_277 and the vSphere MSCS Setup Checklist from http://pubs.vmware.com/vsphere-55/index.jsp?src=vmw_so_vex_pheld_277&topic=%2Fcom.vmware.vsphere.mscs.doc%2FGUID-E794B860-E9AB-43B9-A6D0-F7DE222695A1.html. And yes, the storage devices where part of a MSCS cluster and all the symptoms matched.

The Symptoms

In a scenario where you have to reboot or rescan a host that is participating in a Microsoft cluster (MSCS or Microsoft Failover Cluster), you will notice slow boot up or rescan times for the host. This situation includes all shared RDM’s where an active node owns a scsi reservation, so this will not be only MSCS but others can be effected as well.
The delay exists because the active node (the hypervisor host with the other cluster node VM with the active cluster resources) will still have scsi reservations on the RDM LUN. This will inadvertently cause the “other” hypervisor host (the one with the passive node) to slow down during boot or rescans, as it tries to interrogate each of the devices it is presented. This including the MSCS RDM’s (which are actually active on other node) LUNs during storage path claiming and device discovery. It will fail the reserved/active RDM’s and retries until ESXi gives up and moves along.
With hosts that have a lot of RDM’s this can take a while to finish, from somewhere in the range of 10 minutes to times up to 2 hours that I have seen.

The solution

From the knowledge base article 1016106 and the vSphere checklist we can learn that there is an option that needs to be set on the MSCS LUNs, set the perennially reserved flag on the shared storage device(s). This perennially reserved flag is false (unselected) by default. 

You will have to indentify the RDM Luns that are part of MSCS cluster(s). Take note of the NAA id’s of the RDM shared devices.

We have several options to set this flag in 5.1 (and 5.5) using:

  • Host Profiles; Using the Storage Configuration – Pluggable Storage Architecture (PSA) configuration – PSA Device Setting – <Storage Device NAA id> – Setting – Configuration Details – Flag Device Perennially reserved status. Ofcourse this situation included editions without host profiles. 

Host Profile

  • Use esxcli command on all the participating hosts for all of the shared RDM NAA id’s. Use esxcli storage core device setconfig -d naa.id –perennially-reserved=true command.
  • Use PowerCLI toghter with Get-ESXcli to set the storage.core.device.setconfig on the involved NAA id’s.

Putting it together in a PowerCLI script

I have created a PowerCLI to connect to vCenter and connect to a target cluster. At this time a vSphere cluster target is used as the MSCS clusters are setup to support HA on the VM’s. All hosts in a cluster have the RDM’s visible, and the vSphere cluster is made up with different clustered VM’s. HA is supported for MSCS cluster node VM’s, but can be discussed if necessary on VM’s that participate in application/service clustering. The vCenter and Cluster are defined in $vCenter and $Cluster variables, set these to your environment.

Next the script takes the cluster location and scans for VM’s that have the scsicontroller BusSharingMode equalled to Physical (as per required Physical mode in MSCS deployments, and not getting any none shared RDM’s in the enviroment) using the Get-ScsiController cmdlet and using the Where-Object equal to Physical BusSharingMode. Each VM found with one or more shared scsi controllers are looped through in a foreach. The VM name is captured and the VM is scanned for virtual or physical RDMS with Get-HardDisk – DiskType. I’m using this twice to save the information to a CSV (Export-CSV cmdlet) for documentation purposes and the second time to get the ScsiCanonicalName (the NAA id). If you don’t need the export because the information is not needed or it takes a long time due to a lot of these RDM’s remove the line with the Export-CSV cmdlet.

Each host is looped and connected with Get-EsxCLI cmdlet to run on the vSphere cluster hosts. After that the RDM naa.id are looped and the esxcli storage.core.device.setconfig is used on the RDM naa.id’s to set to true.  To set the device as perennially reserved, the command runned is: $esxcli.storage.core.device.setconfig($false, “naa.id”, $true), which in the script is created as $esxcli.storage.core.device.setconfig($false, ($RDM.ScsiCanonicalName), $true) where $RDM is the found disktype RawPhysical/RawVirtual Selected Object ScsiCanonicalName.

You can check if the device is set by opening PowerCLI and connect to vCenter. Use $esxcli=Get-EsxCli -VMHost <participating host> and list the device $esxcli.storage.core.device.list(“RDM naa.id”). IsPerenniallyReserved parameter should now read : true.

Lastly the VM name is saved in the OldVM variable to check if (in a if statement) there are no double VM names as the loop is created on the scsicontroller check. VM’s can have multiple scsicontrollers and will else be double scanned.

After running this script all subsequent rescans/boots will be at normal speed.

# This script sets the PereniallyReserved flag for RDM datastores that are part of a Microsoft cluster.
# Main reason is to lower the boot time or rescan times of the hosts as these are waiting for the datastore when booting/rescan.

# In a scenario where you have to reboot a host that is participating in a Microsoft cluster, you will notice slow boot up or rescan times for the host.
# This is because the active node will still have scsi reservations on the rdm lun which will inadvertently cause the hypervisor to slow down during boot or rescans as it tries
# to interrogate each of the devices it is presented, including the MSCS RDM’s (which are actually active on other node) LUNs during storage discovery. It will fail the reserved/active RDM’s and retries until ESXi gives up and moves along.
# With hosts that have a lot of RDM’s this can take a while to finish. Seen times up to 2 hours.

# This script connects to the defined cluster and get’s VM’s configured with a shared physical scsi controller and that vm’s RAW disks canonical names.
# It then connects to all hosts in that cluster and set’s the canonical devices to PereniallyReserved (does not make sense, but is configured in this environment. DRS no, HA is for discussion)

# See KB 1016106 for details (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1016106&src=vmw_so_vex_pheld_277)

# Settings
$vCenter = “<vcenter fqdn>” # vCenter hosts name, the user that is running needs access to the vCenter instance.

$Cluster = “<clustername as shown in inventory>” # Cluster to connect to

# Connect to vCenter
Connect-VIServer -Server $vCenter

# Get the hosts from the cluster
$VMhosts = Get-Cluster $Cluster | Get-VMHost

# We need VM’s with RDM’s with SCSI Bus Sharing on Physical (as per vSphere MSCS guide/checklist)
# Check with OldVM and VM’s is added to filter out VM’s with more than one vscsi controller.
Get-VM -Location $Cluster | Get-ScsiController | Where-Object {$_.BusSharingMode -eq “Physical”} | ForEach {
$VMs=$_.Parent.Name
# Get RDM Disk of these VM’s
# RDM’s can be either Virtual or Physical, look for both types.
# We want an export of the disk information found, comment out the line with Export-CSV when you don’t need or takes a long time.
If ($VMs -ne $OldVM){
Get-VM -Name $VMs | Get-HardDisk -DiskType “RawPhysical”,”RawVirtual” | Select-Object -Property Parent, Name, DiskType, FileName, CapacityGB, ScsiCanonicalName | Export-CSV D:\Scripts\MSCSRDMs\RDM-List-$VMs.csv
$RDMs = Get-VM -Name $VMs | Get-HardDisk -DiskType “RawPhysical”,”RawVirtual” | Select-Object -Property ScsiCanonicalName

# Get EsxCli for each host in the cluster
ForEach($hostName in $VMhosts) {
$esxcli=Get-EsxCli -VMHost $hostName
# And set each RDM disk found to PereniallyReserved
ForEach($RDM in $RDMs) {
# Set the configuration to “PereniallyReserved”.
$esxcli.storage.core.device.setconfig($false, ($RDM.ScsiCanonicalName), $true)
}
}
}
$OldVM= $VMs
}

# Disconnect all the connection objects as we are finished.
Disconnect-VIServer * -Confirm:$false

– Hopefully will help you too.

PowerCLI Collection: Back up the ESXi configuration (and restore it)

I happen to run in to vSphere environments without Enterprise plus, or need to implement vSphere environments where for example budget is a constrain, and one or more other limiting factors are there for having Enterprise Plus.

When we don’t have a vSphere edition supporting Host Profiles, Auto Deploy and Distributed switches, we need to have to create a way to (multiple) host back up and restoring for a disaster recovery or such. It can be easy with PowerCLI…

Back-up

In this post I want to write about the PowerCLI cmdlet to back-up a host configuration. We can use the Get-VMHostFirmware cmdlet to back-up the local configuration to a tarbal and save this on a disk location. The Get-VMHostFirmware takes the following options:

Get-VMHostFirmware -VMHost <host> -BackupConfiguration -DestinationPath <directorywithouttrailing\>

This creates a tarbal (in other words tar archive zipped in .tgz) bundle with the the filename configBundle-hostname.tgz. In this tarbal there is a manifest text file with the release level of the host, UUID, kernel options and user options.  There is also a state.tgz tarbal with the host state of the moment the tarbal is created. In this state tarbal is the local tarbal (getting lost a bit here ;-)) with local dump of the configuration of the host. Here you will find the files for creating local settings, vSwitch configurations, Resource Pools, Authentication, NTP, SSH etc. etc. etc.

Configuration Tarbal

Script

We need to build a script around this to occasionally take the dumps from hosts, for example with a schedule task. For the current customer I’m doing this for clusters. An side effect to doing the configuration dump, is checking the configuration is the same across all hosts in the cluster. If you happen to have a tarbal with a different size there is probably something different about this host. This can be with a reason or not. But something worth to check.

The following script takes input in variables for the vCenter, cluster, Hostroot, Hostpassword (note: plain text!), Destinationpath (without trailing \). It furthermore creates a log file name with the cluster and date. In this logfile we put the hostname, buildnumber to get the exact patchlevel and output of the Get-VMHostFirmware.

# This script dumps the config of hosts in cluster
# to be used with restoring via Set-VMHostFirmware (and Set-VMHost -State ‘maintenance’)
# To be runned with a user with access to the environment
# Must be runned against hosts with same build, when needed use -f force to override

# Log is created where name and build is written with time stamp

#Settings
$vCenter = “<vcenter server>”
$Cluster = “<clustername>”
$HostRoot = “root”
$HostPassword = “<password>”
$DestinationPath = “<driveletter:\pathname>”

# connect to vCenter
Connect-VIServer $vCenter

# Create LogFileName
$LogFile = “$DestinationPath\Dump-$Cluster-$(Get-Date -Format ddMMyyyy).log”

# Open Logfile with timestamp
Get-Date | Out-File $LogFile

Foreach($Hostname in (Get-VMHost -Location $Cluster | Select-Object name, build)){
“Dump and build for $Hostname” | Out-File $LogFile -Append
Get-VMHostFirmware -VMHost $Hostname.name -BackupConfiguration -DestinationPath $DestinationPath | Out-File $LogFile -Append
“Dump created” | Out-File $LogFile -Append
}

Disconnect-VIServer -Confirm:$False

Restore

Restoring the configuration when something goes amiss? Take a host at the same build number, either the one where some configuration change went wrong or reinstall a host in case of a complete disaster.

For restoring we need two cmdlets, one to put the host in maintenance and one to restore.

Set-VMHost -VMHost <host> -State ‘Maintenance’

And to restore:

Set-VMHostFirmware -VMHost <host> -Restore -SourcePath <backupfile complete pathname.tgz> -HostUser <user> -HostPassword <password>

The Sourcepath must include the driveletter\directory-tree\dumpfilename.extension.

Like said when restoring configuration the build number of the host must match the build number of the host that created the backup file. If this not possible, use the -force option to override this requirement. Recommended is having the same build number.

– Enjoy!

Sources: vmware.com

PowerCLI Collection: Reporting to HTML

Another PowerCLI for the collection, this time a returning job of creating reports of the environment with specific information requested by Service Delivery or other IT management report. Basic PowerCLI (and with that basic powershell) outputs to the screen, but there are several cmdlets to help and export to a different output; in this case the ConvertTo-HTML cmdlet for export to a nice HTML output. This output can be send to an output file or send via e-mail.

We want a report of VM’s and their power state, vCPU count and vRAM. And specifically resources in a defined cluster. In the final report this shows the cluster resources in the top part and a breakdown of the VM’s in a lower part.

ConvertTo-HTML

ConvertTo-HTML takes the output of an inventory cmdlet (Get-VM for example) and converts the output to an HTML table. When using no formatting it actually creates a ready to use information HTML table. But with the management reports we also want to add some markup to show the information in a nice report.

Let see if we can get some of the wanted information. After we connect to a vCenter server:

$ClusterName = “Testlab”

Get-VM -Location (Get-Cluster -Name $ClusterName) | Select-Object Name,PowerState,NumCPU, MemoryGB | Sort-Object Name | ConvertTo-HTML | Out-File “testlab.html”

We set the clustername in a variable, and use the Get-VM cmdlet to get information of the VM’s in this cluster. With Select-Object we choose the VM Name, the power state, the number of vCPU’s and the configured Memory. We sort on the name of the VM’s and pipe to Convert-HTML to get a nice HTML markup. We pipe this yet again to Out-File for saving the HTML that we can view from disk in a HTML browser.

What output do we have:

image

Wasn’t this what was asked? Yes and no, the report needs to be displayed in a nicer format (you know management ;-)) and we were asked some totals as well.

Get more results and a nicer view

More results? Yes part of the job was to get the cumulative vCPU and vRAM of the cluster.

For this we check the cluster for the specific counts and add this to a total variable (that has been declared at 0 and added with a +=).

#Get VMHost info for CPU and Memory totals
$countcpu = 0
$countmem = 0
$MemoryGBtot = 0
$cpu=Get-VM -Location (Get-Cluster -Name $ClusterName) | Select-Object NumCPU
Foreach ($itemcpu in $cpu) { $countcpu += $itemcpu.NumCpu }

$memory=Get-VM -Location (Get-Cluster -Name $ClusterName) | Select-Object @{Name = ‘MemoryGBtot’; Expression = {“{0:N2}” -f ($_.MemoryGB)}}
Foreach ($itemmem in $memory) { $countmem += $itemmem.MemoryGBtot }
# Now we have $countcpu and $countmem for cluster totals
# Memory should be formatted to two decimals which is #done via Expression N2.

Next up creating a HTML style and markup. We declare a CSS stylesheet with colors in the organization colors. And place box objects where we put the information, this we use in the HTML line to add HTML code and information into the ConvertTo-HTML. A small snippet, in the total script you can see it all together (this part does not work on it’s own).

# Style of the Report in Css
$Css=”<style>
body {
font-family: Verdana, sans-serif;
font-size: 14px;
color: #666666;
background: #FEFEFE;
}
#title{
color:#FF0000;
font-size: 30px;
font-weight: bold;
padding-top:25px;
margin-left:35px;
height: 50px;
}

</style>”

And for some div boxes:

# HTML Markup
$PageBoxOpener=”<div id=’box1’>”
$ReportClusterStats=”<div id=’boxheader’>ClusterTotal $ClusterName</div>”
$ReportClusterHeader=”<table><tr><th>vCPU Count</th><th>Memory GB Total</th></tr>”
$BoxContentOpener=”<div id=’boxcontent’>”
$PageBoxCloser=”</div>”
$br=”<br>”
$ReportGetVmCluster=”<div id=’boxheader’>VM Details $ClusterName</div>”

Create some Div boxes with the styles as we defined and markup in the CSS.

We use these declarations in the report making:

# Create HTML report
$MyReport = ConvertTo-Html -Title “VM Details Report” -Head “<div id=’title’>VM CPU and Memory Reporting</div>$br<div id=’subtitle’>Report generated on: $(Get-Date)</div>” -Body ” $Css $PageBoxOpener $ReportClusterStats $BoxContentOpener $ReportClusterHeader <tr><td>$countcpu</td> <td>$countmem</td></tr> $PageBoxCloser </table> $br $ReportGetVmCluster $BoxContentOpener $GetVmCluster $PageBoxCloser”

The report is created as $MyReport with a ConvertTo-HTML to create a HTML format and with – title to give a title instead of HTML Table. in the Body we put the $Css style, the box parts and the collected information.

This creates the report in a nice markup as shown in the following screenshot:

image

To save or to send, that’s the question

Or both of course, if you want. We can use the earlier mentioned Out-File cmdlet to send pipelined output directly to a text file rather than displaying that output on screen. Well, as we will be directing the output to a variable we don’t see the output on screen, but we still can use this cmdlet to save to a file. A nice declaration of the file name and pipe the HTML report to Out-File.

$fileNameReport = “$OutputPathReportVMDetails-$ClusterName-$(Get-Date -Format o | foreach {$_ -replace “:”, “.”}).html”

and add | Out-File $fileNameReport to the report $MyReport line (add at the end after closing “) when wanting to save

FileNameReport marks up the filename with a clustername and date stamp.

We can also use an e-mail to inform our whomever it is that is interested in the report. For this will need some declaration of from and to adresses, mail title, a SMTP server. We send our HTML marked up report as body.

function Send-SMTPmail($to, $from, $subject, $smtpserver, $body) {
$mailer = new-object Net.Mail.SMTPclient($smtpserver)
$msg = new-object Net.Mail.MailMessage($from,$to,$subject,$body)
$msg.IsBodyHTML = $true
$mailer.send($msg)
}

Write-Host “Sending e-mail” -foregroundcolor “red”
# Sending the report
Send-SMTPmail $EmailTo $EmailFrom “$ClusterName Capacity Report $(Get-Date)” $SMTPSRV $MyReport

We create a Send-SMTPmail function where we need input in the form of a to, from and subject, a SMTP server where the host where the script is run is allowed to relay and a $body. When we call this function we add this values via declared variables. The last is the Report that we put in an HTML markup.

Putting it all together

And to conclude we put this all together in the following script. I left out the saving as I want the report to the e-mail. The code is still in there. Here we go:

# Settings
# vCenter Connection details
$vCenterFQDN=”<vcenter>”
$vCenterUser=”<user>”
$vCenterPass=”<password>”
# Name of the Cluster on which you need to run the report
$ClusterName=”<cluster>”
#Location where you want to place generated report
$OutputPath=”<output path>”
# Set the SMTP Server address
$SMTPSRV = “<SMTP Server>”
# Set the Email address to recieve from
$EmailFrom = “<from email address>”
# Set the Email address to send the email to, comma separate when using multiple reciptients
$EmailTo = “<to e-mail address>”

Write-Host “Connecting to vCenter” -foregroundcolor “red”
Connect-VIServer -Server $vCenterFQDN -User $vCenterUser -Password $vCenterPass

# Style of the Report in Css
$Css=”<style>
body {
font-family: Verdana, sans-serif;
font-size: 14px;
color: #666666;
background: #FEFEFE;
}
#title{
color:#FF0000;
font-size: 30px;
font-weight: bold;
padding-top:25px;
margin-left:35px;
height: 50px;
}
#subtitle{
font-size: 11px;
margin-left:35px;
}
#main {
position:relative;
padding-top:10px;
padding-left:10px;
padding-bottom:10px;
padding-right:10px;
}
#box1{
position:absolute;
background: #F8F8F8;
border: 1px solid #DCDCDC;
margin-left:10px;
padding-top:10px;
padding-left:10px;
padding-bottom:10px;
padding-right:10px;
}
#boxheader{
font-family: Arial, sans-serif;
padding: 5px 20px;
position: relative;
z-index: 20;
display: block;
height: 30px;
color: #777;
text-shadow: 1px 1px 1px rgba(255,255,255,0.8);
line-height: 33px;
font-size: 19px;
background: #fff;
background: -moz-linear-gradient(top, #ffffff 1%, #eaeaea 100%);
background: -webkit-gradient(linear, left top, left bottom, color-stop(1%,#ffffff), color-stop(100%,#eaeaea));
background: -webkit-linear-gradient(top, #ffffff 1%,#eaeaea 100%);
background: -o-linear-gradient(top, #ffffff 1%,#eaeaea 100%);
background: -ms-linear-gradient(top, #ffffff 1%,#eaeaea 100%);
background: linear-gradient(top, #ffffff 1%,#eaeaea 100%);
filter: progid:DXImageTransform.Microsoft.gradient( startColorstr=’#ffffff’, endColorstr=’#eaeaea’,GradientType=0 );
box-shadow:
0px 0px 0px 1px rgba(155,155,155,0.3),
1px 0px 0px 0px rgba(255,255,255,0.9) inset,
0px 2px 2px rgba(0,0,0,0.1);
}

table{
width:100%;
border-collapse:collapse;
}
table td, table th {
border:1px solid #FA5858;
padding:3px 7px 2px 7px;
}
table th {
text-align:left;
padding-top:5px;
padding-bottom:4px;
background-color:#FA5858;
color:#fff;
}
table tr.alt td {
color:#000;
background-color:#F5A9A9;
}
</style>”
# End the Style.

# HTML Markup
$PageBoxOpener=”<div id=’box1’>”
$ReportClusterStats=”<div id=’boxheader’>ClusterTotal $ClusterName</div>”
$ReportClusterHeader=”<table><tr><th>vCPU Count</th><th>Memory GB Total</th></tr>”
$BoxContentOpener=”<div id=’boxcontent’>”
$PageBoxCloser=”</div>”
$br=”<br>”
$ReportGetVmCluster=”<div id=’boxheader’>VM Details $ClusterName</div>”

Write-Host “Getting Cluster Details for CPU and Memory totals” -foregroundcolor “red”
#Get VMHost info for CPU and Memory totals
$countcpu = 0
$countmem = 0
$MemoryGBtot = 0
$cpu=Get-VM -Location (Get-Cluster -Name $ClusterName) | Select-Object NumCPU
Foreach ($itemcpu in $cpu) { $countcpu += $itemcpu.NumCpu }

$memory=Get-VM -Location (Get-Cluster -Name $ClusterName) | Select-Object @{Name = ‘MemoryGBtot’; Expression = {“{0:N2}” -f ($_.MemoryGB)}}
Foreach ($itemmem in $memory) { $countmem += $itemmem.MemoryGBtot }
# Now we have $countcpu and $countmem for cluster totals
# Memory should be formatted to two decimals

Write-Host “Getting VM information for $ClusterName” -foregroundcolor “red”
#Get VM infos
$GetVmCluster=Get-VM -Location (Get-Cluster -Name $ClusterName) | Select-Object Name,PowerState,NumCPU, MemoryGB | Sort-Object Name | ConvertTo-HTML -Fragment

# Create HTML report
# and export to HTML out file
# Use $fileNameReport = “$OutputPathReportVMDetails-$ClusterName-$(Get-Date -Format o | foreach {$_ -replace “:”, “.”}).html” and | Out-File $fileNameReport when wanting to save
$MyReport = ConvertTo-Html -Title “VM Details Report” -Head “<div id=’title’>VM CPU and Memory Reporting</div>$br<div id=’subtitle’>Report generated on: $(Get-Date)</div>” -Body ” $Css $PageBoxOpener $ReportClusterStats $BoxContentOpener $ReportClusterHeader <tr><td>$countcpu</td> <td>$countmem</td></tr> $PageBoxCloser </table> $br $ReportGetVmCluster $BoxContentOpener $GetVmCluster $PageBoxCloser”

# Disconnect
Write-Host “Closing vCenter connection” -foregroundcolor “red”
Disconnect-VIServer -Confirm:$false

function Send-SMTPmail($to, $from, $subject, $smtpserver, $body) {
$mailer = new-object Net.Mail.SMTPclient($smtpserver)
$msg = new-object Net.Mail.MailMessage($from,$to,$subject,$body)
$msg.IsBodyHTML = $true
$mailer.send($msg)
}

Write-Host “Sending e-mail” -foregroundcolor “red”
# Sending the report
Send-SMTPmail $EmailTo $EmailFrom “$ClusterName Capacity Report $(Get-Date)” $SMTPSRV $MyReport

Notice: at this time the script takes a plaintext password.

Add your environment entries to the settings variables and your good to go.

More

You can add more or specific required cmdlets and information to the report to get what you want. Or you could go on the community and find an already made reporting scripts out there. Maybe not the best way to learn, but you can check in the source what these guys and gals have been up to. A great one for example, vCheck is highly recommended to get an overview report of your environment. This script can be found at: http://www.virtu-al.net/vcheck-pluginsheaders/vcheck/.

– Enjoy!

Sources: Microsoft.com powershell cmdlet library, vmware.com PowerCLI reference, community.vmware.com for examples.

PowerCLI Collection – Adding SNMP settings to ESXi hosts

It’s been a while since I posted a PowerCLI script, but since I started a new project one of my first tasks was to help out and setup SNMP to monitor ESXi hosts from Nagios (which is a great product BTW).  So a new script for the collection.
O yeah, a little notice as I’m not always working with scripts there are probably better scripts or better scripters out there that will be more efficient. I have chosen to use a script and not a killer one liner, just to make clear what is happening and for others to use and understand what is happening.
If you have tips and leave a comment to help me, no problem I would appreciate that and probably any reader out there as well.

SNMP Scripts

To change the SNMP settings we need some information about what need to be set. For this example I will set the readonly community, trap destination and trap community name.  I have set up the script to take input from the user for these values.

I’m using two scripts, one to change a single hosts and one to change hosts connected to vCenter in a specific cluster. The main difference is that the single host script takes a input for the ESXi hostname, while the cluster script takes vCenter and cluster name (which can be easily changed to another object location for example replacing Get-cluster with nothing or Get-datacenter) and loops through the hosts. There is a host connection needed to change the SNMP values for that host, this is what is done first in the loop. If not direct connected to the host, the SNMP settings will not be changed. This goes for the single host script as well.

For SNMP there are two cmdlet’s relevant Get-VMHostSNMP and Set-VMHostSNMP, respectively getting the SNMP values from the host and setting the values. First enable SNMP with Set-VMHostSNMP to set value –Enabled to true. The SNMP settings value are done via two more Set-VMHostSNMP cmdlets, first setting the Read only community with –ReadOnlyCommunity option and the other to set a SNMP trap target and communityname (-Target and -TrapCommunity).

The rest of the comments in the script should tell what action is done at that time. If not, you know where to find me….

I have chosen not  to hardcode or use files that store passwords, again this is user input with the Get-Credential cmdlet. For the ESXi hosts in the cluster script this is stored to a variable to be used in the host loop. The condition is that the hosts use the same credentials.

The Singlehost script:

#Settings

$ESXHost = Read-Host “ESXi hostname to change:”

$trapDestination = Read-Host “SNMP Trap Mgmt server hostname: “

$trapCommunity = Read-Host “Trap Community Name: “

$ReadOnlyCommunity = Read-Host “Read Only Community Name: “

# Running no need to change under this line

# Connect to the ESX server

$viServer = Connect-VIServer -Server $ESXHost -Credential (Get-Credential)

 

# Get snmp

$snmpConn = Get-VMHostSnmp -Server $defaultViServer

 

# Enable snmp

Set-VMHostSnmp -HostSnmp $snmpConn -Enabled:$true

 

# Set read-only community

Set-VMHostSnmp -HostSnmp $snmpConn -ReadOnlyCommunity $ReadOnlyCommunity

 

# Set trap target and community name

Set-VMHostSnmp -HostSnmp $snmpConn -AddTarget -TargetCommunity $trapCommunity -TargetHost $trapDestination

 

#disconnect from ESX server

Disconnect-VIServer -Server $viServer -Confirm:$false

 

The cluster script with a host loop:

#Settings

$vCenterFQDN = Read-Host “Enter vCenter FQDN: “

$ClusterName = Read-Host “Enter Cluster name of hosts that will need to be changed: “

$trapDestination = Read-Host “SNMP Trap Mgmt server hostname: “

$trapCommunity = Read-Host “Trap Community Name: “

$ReadOnlyCommunity = Read-Host “Read Only Community Name: “

 

# Running no need to change under this line

# Connect to the vCenter server

Write-Host “vCenter credentials”

$viServer = Connect-VIServer -Server $vCenterFQDN -Credential (Get-Credential)

 

# Get all hosts in vCenter managed Cluster so we can cycle thru them

$Hosts = Get-Cluster -Name $ClusterName | Get-VMHost

 

# Get ESXi Host credentials

Write-Host “ESXi host credentials”

$EsxCred = Get-Credential

 

# Cycle through each host

ForEach ($VMHost in $Hosts)

{

                # Need to connect to ESXi itself

                $esxconnect = Connect-VIServer -Server $VMHost -Credential $EsxCred

 

                # Get snmp object

                $snmpConn = Get-VMHostSnmp

 

                # Enable snmp

                Set-VMHostSnmp -HostSnmp $snmpConn -Enabled:$true

 

                # Set read-only community

                Set-VMHostSnmp -HostSnmp $snmpConn -ReadOnlyCommunity $ReadOnlyCommunity

 

                # Set trap target host and trap community

                Set-VMHostSnmp -HostSnmp $snmpConn -AddTarget -TargetCommunity $trapCommunity -TargetHost $trapDestination

 

                # Disconnect-VIServer

                Disconnect-VIServer -Server $esxconnect -Confirm:$false

}

 

#disconnect from ESX server

Disconnect-VIServer -Server $viServer -Confirm:$false

Just copy and save the scripts. And when your execution policy allows it, execute them.

–          Enjoy!

WebCommander – VMware Lab Flings

An other post in the VMware Lab Flings series. This time a fling that I was re-introduced to when following the PowerCLI session at VMworld Europe.

So what’s is WebCommander?

Webcommander is a GUI framework around PowerShell and PowerCLI scripts. It gives an easy to use web interface. This can be used to provide users with scripts without those users knowing the PowerCLI commands, or to give users access only to specific prepared tasks without giving them access to the web client (they still need to have permissions to do there operations). A great way in delegating specific tasks!

What’s needed, WebCommander Architecture?

WebCommander is a package that uses or needs the following components:

– Windows Server 2008 or 2012.
– PowerShell v3 or v4.
– vSphere PowerCLI.
– IIS (minimal v8).
– PHP 5.

The installer also needs .Net Framework 3.5 to be installed on the system. And of course the WebCommander package from the VMware Fling site. This can be found at http://labs.vmware.com/flings/web-commander. At that site there is also a install instruction manual.. Setup of WebCommander is done by installing above apps and running the Setup powerscript supplied with the WebCommander installer. Here you add the default password to connect to ESXi hosts. Default user is set to root, default guest user is set to administrator (you can select them in several of the actions).

Before running the setup script you have to change the Execution policy to unrestricted first…

image

Change the default user to Adminstrator in IIS Manager after installation is finished. 

Interface

After installing and checking your installation you can open the default interface.

image

There are several categories of PowerShell/PowerCLI included with the installation. You have the option to do Active Directory, View Broker, Guest, VM and vSphere actions. Actions for adding, removing, start/stop/restart operations, renaming and taking snapshots are included.
Of course you can add (or remove) you own specific organisation tasks. The how to is also included in the installation manual.

When running a command a screen is shown where you can add the VC or ESXi host FQDN/IP, and select the users.

image

After running the results also show an URL with the command just run. You can give this URL to your user when you want only access to this command. Great feature!. Unfortunately the supplied password is in plain text in the URL (in this case root……).

image

– Enjoy commanding your environment! Thank to the developers of this fling.

vCenter Collectors – Syslog and Dump collector (with some PowerCLI)

With stateless VMware Auto deploy hosts, or hosts with SD or USB Stick installation, these hosts typically do not have local storage attached. There will be no space assigned to the scratch partition for dumps and logs. Defaulted they will be stored in RAM (/tmp/scratch). After a reboot the /tmp will be cleared.

We need to redirect the scratch partition to a shared VMFS and a central log / dump location. Scratch redirection is performed with the following steps:

  • Decide on a shared data store where preferably all hosts in a physical location connect, beware of physical location Bias. The volume is like /vmfs/volumes/UUID
  • Create a locker directory on that data store, for example .locker-Hostname
  • Connect to the ESXi host with Connect-ViServer -Server <Hostname>
  • Set the scratch partition with Set-VMHostAdvancedConfiguration -Name “ScratchConfig.ConfiguredScratchLocation” -Value “/vmfs/volumes/UUID/.locker-Hostname”
  • Maintenance mode (Set-VMHost -VMHost <host> -State ‘Maintenance’) and reboot (Restart-VMHost <host>).

With central management of an environment, a central log/dump location is recommended (also for stateful deployments this should be a standard). For a central log we need some services added to the vCenter:

– dump collector; collecting ESXi dumps of the infamous PSOD.

– syslog collector; collecting the host logs or syslogs.

On Windows we use the vCenter installer to install both services. With the vCenter Server Appliance (VCSA) these service are bundled with the appliance.

image

Dump Collector installation


Installation of the dump collector is straightforward. Determine in advance where your dumps will be saved, how much space (2GB default is sufficient for most environments) and to do a standalone or integrate with vCenter (last option is preferred for single management).

image

(if you got a data disk, change the default C: parameters)

image

And fill in the vCenter and account details on the next screen. The account will be used to register the extension.

Syslog Collector installation

Just like the dump collector the installation is straightforward. Determine the syslog rotation and log location. And start the installation to the VMware vCenter Server.

image

(if you got a data disk change the default C:).

image

Change the ports when needed for your environment, standard default is sufficient for most environments.

And finish the installation.

If you still use the vSphere client, then install both service add-ins.

Configuring the hosts

The ESXi hosts need to be configured to use the remote syslog and network dump location. You can use ESXi CLI and the advanced system settings or you can use PowerCLI. If we use PowerCLI for syslog we can use the Set-VMHostAdvancedConfiguration cmdlet and for the dump collector we need the Get-EsxCli cmdlet to use esxcli.

When we do this for a list of hosts we can use the following script. Just replace the vcenter servername at $vCenterFQDN and vCenter IP at the $esxcli.system.coredump.network.set line.

#set vCenter IP/FQDN

$vCenterFQDN = “<vcenter server>”

#connect to vCenter

$viServer = Connect-VIServer -Server $vCenterFQDN -Credential (Get-Credential)

#get all hosts in vCenter so we can cycle thru them

$Hosts = Get-VMHost

#cycle through each host

ForEach ($VMHost in $Hosts)

{

   #Add Syslog to Syslog Collector

   Set-VMHostAdvancedConfiguration -VMHost $VMHost -Name Syslog.global.logHost -Value $vCenterFQDN

    # Add DUmp to collector

    $esxcli = Get-EsxCli -vmhost $VMHost

    $esxcli.system.coredump.network.set($null, “vmk0”, “<vcenter ip>”, “6500”)

    $esxcli.system.coredump.network.set($true)

    #write out host name that was changed:

    write “Updated host: $VMHost”

}

#disconnect from ESX server

Disconnect-VIServer -Server $viServer -Confirm:$false

Checking the configuration

A host can be checked by going to the dump directories and checking the contents, just like the following for a host’s syslog directory.

image

I don’t have a crashed ESXi host at my disposal, so it is easier to check on the hosts via esxcli:

esxcli system coredump network get.

The IP and port should be in the output at:

Network Server IP:

Network Server Port: 6500

PowerCLI collection – Changing PSP

On storage controllers in combination vSphere 5.x multi path IO, path selecting policy should be Round Robin (always check if your storage supports this). When receiving the LUN it happens that those PSP are defaulted to Fixed. You can change the PSP policies of all attached storage with this PowerCLI oneliner (Round Robin in the example):

get-cluster -Name “Cluster” | Get-VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.MultipathPolicy -ne “RoundRobin”} | Set-ScsiLun -MultipathPolicy “RoundRobin”

A little caution when you have storage from mixed storage providers (mixed storage controllers without a SVC,Vplex or such layer).

What does this one do:

– get-cluster -Name ”Cluster”; Retreives the cluster information for Cluster. This command can be replaced with get-datacenter. And pipe’s this information to:

– Get-VMHost; retreives the hosts information (one by one in from the cluster). The output is piped to:

– Get-ScsiLun – Luntype disk | Where-Object {$_.MultipathPolicy -ne “RoundRobin}

Get de Scsi Lun of type disk en checks if this object current policy is not equal with Rond Robin. The output is piped to:

– Set-ScsiLun -MultipathPolicy “RoundRobin. The Lun found with previous part is changed to a RoundRobin policy.

The policies can be set to:

RoundRobin, MostRecentlyUsed, Fixed or Unknown.

PowerCLI collection – Adding hosts to inventory

With several projects at several sites I have come along a nice list of powerCLI oneliners or scripts. I wanted a place to have these scripts in a handy catalogue online, maybe share and learn on some blogging skills (or be a little louder than only 140 characters). I know there are probably already place online, but..how can I learn the bloggin. So here goes…

This example is a script that adds 24 hosts to a vCenter inventory. ESXi is installed, vCenter and PowerCLI are setup. PoweCLI has a connection setup to the vCenter (Connect-VIServer -Server <vc> -Protocol HTTPS). No other automation or scripting is in place.

To add 24 hosts to a inventory:

1..24 | Foreach-Object { Add-VMHost hostesxi$_.example.nl -Location (Get-Datacenter DataCenter) -User root -Password “<replace>” -RunAsync -force:$true}

So what does this one do:

  •  1..24; Amount of hosts. This is piped to the command and used by the $_ variabel. Foreach-Object let’s us loop within the 1..24 range so the hosts number will change every loop.
  • Add-VMHost is actually the PowerCli cmdlet to add the host.
  • hostesxi$_.example.nl. Creates hostnames of hostesxi1_.example.nl till hostesxi24.example.nl in the datacenter DateCenter
  • -User and Password are the user and password for access to the ESXi hosts.
  • -RunAsync option is to let the tasks run parallel.