Powershell: Patching XenServer

And now for something different…..
Last week I was tasked with the question to be able to automate applying multiple patches to a XenServer instead of admins doing the patching manually. Sure no problem, we can do this with Powershell.

Requirements.

I am using XE.exe from the XenCenter installer to control the XenServers. xe.exe needs to be in the path environment variabel. If you installed XenCenter to D:\Program Files (x86)\XenCenter you need to add this to the current path.

Path XenCenter

Next up is Powershell. As XenCenter needs to be on a Windows server, be sure to install Powershell if you are not running a version already. Should be in the management server by default.

Download the required patches to a location on the server. This location can be added to the script via the $UpdatePath variable for example download and set it to “E:\Patches\NEW-XEN-6.1\xen-6.1”.

Patches

Next up set your pool master ip (or IP of the server). Currently the script is set to get two values, one for the IP hard coded and one for the server you want to patch via user input. They can be the same, but the IP cannot be a slave. Xe won’t connect to a slave in a pool setup. At the environment this script was needed the hosts did not have a DNS record therefor IP and name is used. Hosts without DNS records is not advisable!
The script also needs the root user and password. At this time this is hard coded.

What does the script do?

The script takes a server name to patch as user input (with Read-Host), this is the name of the server as it is known in XenCenter. Next the script finds the uuid via the host-list command.

The patching procedure is to disable the host (aka Maintenance mode via host-disable), Evacuate running VM’s via host-evacuate. With the evacuate command there is an error checking to be sure the script can evacuate. If not the script exits here. We won’t apply patches and reboot with running VM’s. If you happen to run this on a non pooled server, you will need to power off the VM’s yourself. If the scrip exits here, you can find the reason in the evacuate txt file in the running directory. Most of the time this is a memory related error, but this has more to do with networking not available on an other host, storage, mounted iso’s or other items that fail a evacuation.

If all is well the patch directory is executed trough (Get-ChildItem of $UpdatePath), the patch is uploaded to the server. The UUID of the patch is saved and the update is installed. And on to the next one.

Last is the server reboot to finish the updates. The script sleeps for five minutes to let the host start-up, and then enables the host again. From this point you can distribute workloads back to this server and optionally run it at the next one.

Status of the patching is shown in the execution Window, and two files are saved during the execution: Hostname_evacuate.txt and Hostname_uuid.txt. The first saves the host-evacuate output (if there is nothing in the file all went okay), the second is used to save the UUID’s of the patches.

XenServer Patch script

Note: Citrix recommends rebooting the server before a patch. Currently this is not in the script as we where doing some other actions before the patch that included reboot. But this is easily fixed by copying the host-reboot line and pasting that one just after the evacuation. Be sure to check if the reboot enables the host or not.

Putting it all together

The script:

# This script connects to the Defined Pool master, Get server to patch, enters maintenance, patches the server, reboots and enables again.
# uses xe.exe from the XenCenter installation, requirement is to have this in the path of the system where this is run.
# Patches must be downloaded and placed in UpdatePath location.

# The script leaves txt files for evacuate and patch uuid’s actions. When evacuate fails script will stop, you can find the error in the text file.
# Mostly to do with networking or storage, or mounted tools on VM’s….

# Citrix Highly recommends rebooting XenServers prior to installing an update, this is not implemented in this script.

# Settings
$XenServerMst = “<pool master Ip>” # Pool Master IP and name of to update server must be filled in! XE cannot connect to slave.
$XenServer_Name = (Read-Host “XenServer Name to update”) # Server to update.

$XenServer_username = “<user>”
$XenServer_password = “<passwd>”
$UpdatePath = “E:\Patches\NEW-XEN-6.1\xen-6.1”

# nothing should be changed below this comment

# Loop through Updates
# Every Patch should be applied in the same procedure.
# Maintenance, reboot, upload, patch, reboot, end maintenance
# Evacuation of VM’s that are running.
#
Write-Host “Getting host uuid” -foreground green
$SrvUUID=(xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password host-list name-label=$XenServer_Name –minimal).split(‘,’) | % {$_.trim()}

# Maintenance, reboot, upload, patch, reboot, end maintenance
# Evacuation of VM’s that are running.
Write-Host “$SrvUUID will be disabled….” -foregroundcolor green
xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password host-disable host=$SrvUUID

Write-Host “$SrvUUID will be evacuated…(will take a long time)” -foregroundcolor green
xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password host-evacuate host=$SrvUUID | out-file $XenServer_Name”_evacuate.txt”

# We should exit the script when this fails
# $?
# Contains the execution status of the last operation. It contains
# TRUE if the last operation succeeded and FALSE if it failed.
if($?){
Write-Host “Migrated. Continuing with update” -foregroundcolor green
}
Else
{
Write-Host “Error with migration. Exit script at this point. Updates are not installed” -foregroundcolor red
exit
}

Foreach ($Update in Get-ChildItem $UpdatePath){
# Loop through Updates
# Every Patch should be applied in the same procedure.

Write-Host “$SrvUUID will be pathed with $Update. Preparing to upload….” -foregroundcolor green

Write-Host “$Update will be uploaded” -foregroundcolor “green”
xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password patch-upload file-name=$Updatepath\$Update\$Update.xsupdate | out-file $XenServer_Name”_uuid.txt”

$uuid_patch = (xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password patch-list name-label=$Update –minimal)

Write-Host “$Update will be installed to this server” -foregroundcolor green
xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password patch-apply uuid=$uuid_patch host-uuid=$SrvUUID

Write-Host “After Patching, remove patch files to protect against running out of disk space” -foregroundcolor green
xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password patch-clean uuid=$uuid_patch

Write-Host “=======================================================” -foregroundcolor green
# End Update Loop
}

Write-Host “Post Patch reboot” -foregroundcolor green
xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password host-reboot host=$SrvUUID

# Wait for reboot seconds 300 (5 minutes)
Write-Host “Waiting five minutes for reboot to continue” -foregroundcolor green
start-sleep -s 300

Write-Host “Enable $SrvUUID again” -foregroundcolor green
xe.exe -s $XenServerMst -u $XenServer_username -pw $XenServer_password host-enable host=$SrvUUID

Write-Host “=======================================================” -foregroundcolor green

# Need to redistribute VM’s yourself

 

– Happy patching

 

 

vCenter Operations Manager Part 3: Nice, but yeah I’ve also got some other stuff to monitor

While consulting on using vCenter Operations Manager (vCOPS) I am often (or is this always) confronted with the current organizations monitoring product in place and/or the multiple components in the infrastructure. It is unknown that vCOPS can also be leveraged to `other than that VMware hypervisor’ infrastructure components. Or let it work together with other specialistic monitoring solutions. With the latter it is important to define who works with what solution as a point of entry, where the specific solution is used for detailed information , and where a single monitoring dashboard can be created for a single monitoring pane.

vCOPS provides a complete VMware hypervisor monitoring, analysis and reporting tool from the box. It’s required to have an license /configuration investment for the analysis / reporting, and for example the additional vCenter Hyperic installations to include other than VMware hypervisor monitoring and application landscape. vCenter Hyperic uses a Hyperic Server (can be deployed as a virtual appliance) and agent architecture, and uses a management pack from VMware solution Exchange (https://solutionexchange.vmware.com/) to provide with collectors and dashboards. Next to Hyper-V this adds SQL and Exchange to your custom dashboard.

Hyperic Dashboard

The agent is installed on the to monitor hosts where there are Windows agents for your Hyper-V or Windows based workloads, and Linux agents for XenServer and Linux based workloads.

Further more there is a SCOM management Pack to leverage the SCOM monitoring that is usialy found with Microsoft deployments. The following model shows them both.

vCOPS-Hyperic

Other devices such as networking can be integrated by vendor specific adapters/management packs, SNMP and/or leveraging Log Insight integration. Not all vendors have a vCOPS adapter/management pack, for these SNMP will need to be leveraged. Traps can be catched with a vCenter Orchestrator workflow to add an alert to the monitoring solution. Plus side on vCOPS is the addon to the vCenter Management interface providing a single pane of glass for management, monitoring and reporting. Yes, it needs some more management packs to be a replacement of all the monitoring solutions.

Getting more with adding VMware vCenter Multi-Hypervisor Manager 1.1 to the vCenter management layer. This provides means to manage Hyper-V from vCenter. vCenter Orchestrator can be leveraged to manage XenServer tasks, with SSH workflows to run xe actions for example. Putting the vCenter, vCenter Orchestrator, vCOPS and Multi-Hypervisor support provides a management, monitoring and control plane of a software defined data center.

Collectors, adapters and Management Packs

Adapters work closely with the collectors. The collector is the gateway between the adapters and vCOPS. The adapters connect to and collect data from data sources, transforming the data into a vCOPS format. The analytics VM houses the collector or in a distributed architectures remote collectors gather metrics locally and feed the central vCOPS to offload the central install. Depending on the data source and the adapter implementation, an adapter might collect data by making API calls, using a command-line interface, or sending database queries.

Management Packs have specific configuration/information to connect to specific data sources with it’s specific connection and gathering method. They also offer ready to use custom dashboards or automatic discovery information adding the adapters to the environment.

vCOPS Architecture

Custom collectors and management packs can be downloaded from solution exchange. These are in *.pak format.

Monitoring other hypervisor or workloads with Hyperic

Hyperic server component can be downloaded from my.vmware.com (that is if you have a valid download account). There are the installable downloads, but also the Hyperic vApp. I am deploying this one as the server. You need the install bundle for the agent installation (that is the easy one). The appliance creates two VM’s in a Hyperic vApp, the Hyperic server and the vFabric vPostgres database server. Deploy the OVF in your ESXi environement. Fill in the values as required (location, size, space and networking) and fill in the hqadmin details. Or choose a new user account. Start up the Hyperic vApp when finished. You can manage the Hyperic environment by using: https://<fqdn or IP>:5480/ for the appliance, http://<fqdn or ip>:7880, https://<fqdn or ip>:7443 or logging on to the console. Logon with root and the password supplied for the hqadmin user in the OVF deployment. Here you can set the time zone and view networking (managed by VAMI).

Hyperic Agent

Deploy a Hyperic Agent. For this demo I am using the Windows agent on a Hyper-V host (demo/test environment that is).
Create a destination directory, for example program files\Hyperic. Start by extracting the zip and command prompting to that directory (run as….). Run setup.bat -full. Accept EULA and choose 2. for the HQ Agent installation. 

HQ Agent

Type the destination path created. For the example C:\Program Files\Hyperic.
When entering the setup this unpacks the agent to this directory. Go to the Hyperic Agent installation at with the c:\Program Files\Hyperic\agent-5.8.1-EE/bin/. Do a hq-agent.bat install to install as a Windows service. Do a hq-agent.bat start to start the service for the first time and do an initial configuration. Here you define, communication method, IP of the HQ server, login credentials, port numbers et al. Accept self signed when trusted. Don’t forget firewalls when they are in the transport.

Configuration Agent (a yes failed to type in the password on the first go ;-) )

Creating the match made in heaven…

Next up is configuring vCenter Hyperic and vCOPS to find each other (going from one interface back to the other and back again warning).

The Hyperic management PAK can be installed via the Update tab in the admin console of vCOPS. Browse to the https://<UI fqdn (or IP)>/admin. Log in and go to the Update tab. Browse the pak file and update. Notice the warning that updates are irreversible, you should have a back-up or this is the time.

vcops updateConfirm Update

Login to the vCOPS custom dashboard (/vcops-custom/).  Select Admin > Support. On the Info tab, find the Adapters Info pane. Check if MP for Hyperic in the pane. If not, click the Describe icon. The Describe icon is located at the top right of the Adapters Info pane.Click Yes to start the describe process and click OK. And be patience….

Log in to the vCenter Hyperic site, at http://<hyperic fqdn or IP>:7080/ or https at port 7443. Here you can check the autodiscovery. My hyper-v.testlab.local is found and can be added to the inventory. Yeah!

Hyperic Found Hyper-V

Go to the Hyperic  Administration tab. Select the HQ Server Settings link and configure the vCenter Server settings. Click ok on the bottom to save. This will find vCenter as a resource. Sometimes a little discovery will get it from stuck state (this can also be just my lab environment).

Management Pack for vCenter Hyperic instance requires a user name and password to connect to vCenter Hyperic server. vCenter Hyperic server requires a user name and password to connect to the vCOPS adapter. You must provide this information in a credential in vCOPS. Go to the custom vCOPS and browse to Environment > Configuration > Credentials. In the manage credentials select the MP Hyperic and Hyperic credentials type. Click add. Fill in the required values.

 Credentials

Are we there yet? Almost. Add an adapter instance to vCOPS. Go to Environment > Configuration > Adapter Instances. You will only find the default adapters if you haven’t added any. Select the vCOPS collector and adapter kind MP for Hyperic and click on the add icon (the first). Fill in the required values in the Add window. Important are the url’s, the certificate checking and the auto discovery. I select no checking for the certificates, but if you want you have to import the required certificates in the appropriate trust stores. For the URL of the vCenter Hyperic server in the Hyperic Server URL text box.For example: https://<fqdn IP Hyperic server>:7443. The URL of your vCenter Operations Manager server in the vCenter Operations Manager URL text box.For example: https://<fqdn or IP vCOPS>. Select the credential name you created in the previous step.

Add Adapter

Click test to test the connections. Ok to add the adapter. When auto discovery is finished the Hyper-V VM’s pop up in the vCOPS custom dashboard. Just take a look at Hyperic Hyper-V VM utilzation tab. My lab isn’t much but you will get it.

HYper-V VM's

– Happy Monitoring your other hypervisors and workloads!

Sources: vmware.com

Design Choice: vCenter Windows or vCenter Server appliance?

A question that is again popping up lately is whether customers must go for the vCenter Server Appliance (VCSA) or a Windows based vCenter system. Yes I have my favorite, but I will often go for the consultant answer: it depends. But on what does this depend? I will try and sum up some of the arguments to take in to account.

But first, what is this VCSA your talking about?

You may or may not know vCenter can be deployed in two ways, the commonly known vCenter installation on a Windows based host(s) or deploying vCenter as a appliance. The appliance is a 64-bit Linux based (SUSE Linux Enterprise for the distro interest) virtual machine with an embedded database (vPostgres) or the option to use an external Oracle database. The appliance has been around since it’s introduction with version 5.0, but has not been really production worthy since the 5.5 version. With the 5.5 version the vCSA supports managing 100 hosts and 3000 VM’s with the embedded database versus the 5 host and 50 VM’s with the previous version. The 5.5 limit for the external Oracle database is even higher (100 hosts and 10000 VM’s), but to be honest I haven’t seen that combination around.

VCSA

The appliance is build with the following vCenter services: vCenter services (vCenter, Inventory, SSO, Web Client) and services such as syslog collector, dump collector, AutoDeploy (and capable of doing DHCP/TFTP).

vCSA services

What are the arguments to consider?

  •  For organizations that are having a DBMS install base of MSSQL there is currently no support of using this in combination with the vCSA where the Windows based happily works with MSSQL (via ODBC DSN). If there is no experience, no business investment or no DBA’s wanting to manage an additional DB flavor you can stop right here (or wait when a next version of VCSA will hopefully support MSSQL as an external database).
  • Windows vs Linux install base and license. You got your Windows based organizations and you got your Linux based organizations. More often they are mixed. Got Windows data center licenses sorted out for your organization, another Windows VM is often not an issue. Need to save some licensing or just don’t want to invest in new, VCSA will probably be more of your taste. Beware that you can’t install Update Manager on VCSA, this still needs to be set up on a Windows host which can be an existing. Same goes for your trusty toolset, PowerCLI needs Windows PowerShell, RVTools need .Net. These tools can be easily put on a management server or admin workstation (where the toolset are supposed to be on the first place), they connect to VCSA just as vCenter on Windows. Need remote CLI? Added by deploying the vSphere Management Assistant (vMA) appliance. For Windows only based IT departments a knowledge investment is needed to support Linux based appliances.
  • vCenter Orchestrator. Windows vCenter installers installs vCenter Orchestrator on your Windows system or you can use the Windows installer to install Orchestrator on another designated Windows server. vCenter Orchestrator is not installed on the VCSA, but there is the vCenter Orchestrator appliance you can deploy next to it.
  • Linked mode. Want a single management plane, or in other definition a Single Pane of Glass, for multiple vCenter instances linked mode is required. VCSA does not support Linked mode where vCenter on Windows does. There is an other option, leveraging the built-in SSO Replication by joining all (or some you want to share) SSO instances under a common domain. A benefit next to the automatic synchronization, is that you will be able to see the inventory of all SSO instances from whatever Web client.
  • Experience with appliances. I think appliances add simplicity for deployment of the vCenter systems, but can be a hassle to (re)configure and for example do database back-up dumps when you don’t have any experience with Linux. Don’t worry, VCSA has an admin Web interface or the vSphere Web client for configuration tasks and when you need the console commands are documented. The procedures are on Documentation Center, Knowledge Base and widely blogged about in the community. Go ahead and learn appliances as there are a lot of good ones.
  • Monitoring/Back-up. Monitoring of vCenter server services and system parameters. If you use a Windows based or agent type of monitoring, the Windows based monitoring does not work for a Linux appliance. Check if your monitoring solution supports Linux. Same goes with thinking about your back-up strategy and how to back-up the VCSA.
  • Physical vCenter. When there are reasons to install the management of your virtual infrastructure on a physical server outside the virtual infrastructure, than Windows vCenter is the only option. VCSA is a virtual appliance which needs to be deployed on virtual environment. Depending on the requirements, you could also go with an VCSA deployment in an other vSphere cluster than the one vCenter is managing.
  • IPv6. IPv6 is not supported on the VCSA and is supported on Windows. If you happen to use IP addresses in the vCenter configuration or connecting to vCenter these are IPv4 only. To connect to vCenter system in an IPv6 environment you must use the fully qualified domain name (FQDN) or host name of the vCenter Server. FQDN is preferred to be used in any case.
  • vCenter Server Heartbeat. Use this solution to protect your vCenters, you cannot use this with the VCSA. But on the other hand, you should be working on a replacement solution (you have some time) as VMware announced EOA of all vCenter Server Heartbeat versions. See details here: http://www.vmware.com/nl/products/vcenter-server-heartbeat. Leveraging vSphere HA instead for example.

Note: These arguments are not ordered by importance, that’s is up to the organization in question.

And what do you think?

I have a favorite, but do keep in mind that there is no one size fit’s all. I happen to like the appliance way of deployment, simplicity at your fingertips and no abuse from Windows admins logging on and hogging the vCenter resources (sizing) with OS / RDS footprint. Or worst destroying to system with all those I needs this and this tool for my work.

Yes you need Windows for Update Manager, which can be combined with other update management tools. And yes you need your Windows based tool set somewhere, are those supposed to be on the vCenter server in the first place?
When a design decision is made to use the Windows based vCenter, please use it as an infrastructure server. Don’t abuse it as an administrator playground (aka RDS server, aka steppingstone, aka PowerCLI test heaven). I like to remove the GUI aka use core when Windows is the design choice (PowerCLI Get-WindowsFeature *gui* | Uninstall-WindowsFeature –Restart). From Windows version 2012 switching from GUI to core does not require a reboot.

What do you think? I’m interested in your views. Let me know in the comments.

Sources: vmware.com, microsoft.com