Dissecting vSphere 5.5 Enhancements – vCenter Server Appliance ready for lift off

A real concern for implementing the vCenter Server appliance (vCSA) 5.1 in production environments is the environment limits with the use of the embedded database. The embedded database with 5.1 has a limit of 5 managed hosts and 50 managed VM’s. That is not the size of production in a lot of setups.
The maximums could be lifted, but needed an external database service that was limited to only Oracle (and with the 5.5 version still is limited to Oracle as an external database). Not a lot of organizations use Oracle for infrastructure services, so that option is not widely used (or at least haven’t seen that around much). I’ve only used the vCSA in lab / testing environments.

With the introduction of vCSA 5.5 one of the most important parts is a reengineered embedded database (vPostgres). This lifts the configuration maximums of the vCSA to (editted, still production worth though) 100 managed hosts and 3000 managed VM’s. Well that’s more like it, this is production worth.

Table with specification vCSA 5.1 vs vCSA 5.5.

*) Specifications of the minimum requirements are depending on your environment. Don’t expect a difference approach, 2vCPU and memory according to your environment size. Scale down or scale up the standards.

The 5.5 release makes vCSA a production appliance. It is deployed fast and in comparison to a vCenter server does not need a Windows license.

– I’m gonna see a lot more of the vCSA 5.5 in customer environments.

— This post has been edited to change the number of managed hosts to 100 and managed VM’s to 3000. Earlier numbers were unofficial.

Evaluation – VMware vCenter Log Insight – Part one the what, why and installation

A few posts back I wrote about the vCenter Collector services to centrally collect logs and dumps. There is also the VMware vCenter Log Analyzer appliance, a collector and an analyzer (with the collectors you have to do the analyzing part yourself). The appliance is an OVF/OVA download, that you can add to your environment.

What does it gives you:

  • Log file collection and analysis.
  • Alert and events collection and analysis.
  • vCenter en vCenter Operation Management integration.
  • Connect to everything. Everything? Well everything that’s able to generate log data. Several partners have content packs for their logs that you can import that gives you an additional layer for analyzing.

So you were writing about the vCenter collectors, why will we not use them? Well you can for you virtual environment. And you will if you are budget constrained. You will have to do the analysis all by yourself, with your own expertise.

How do I get it?

What does it cost?

VMware vCenter Log Insight is licensed on a per operating system instance (OSI) basis, which is defined as any server, virtual or physical, with an IP address that generates logs, including network devices and storage arrays. You can analyze an unlimited amount of log data per OSI. vCenter Log Insight price is currently announced as $200 per OSI.

And it really depends on the amount of log generating devices you want collected or analyzed (not only VMware related).


Installation of the appliance is straightforward, just like any OVF: Right click datacenter or cluster for Deploy OVF Template. Select your source location, and don’t forget to change the file type in the browse window (else it defaults looks for *.ovf and not the ova extension the Log Analyzer has) and select the downloaded version.

Accept license agreement (O yes, you want to read it first ;) ), choose your hostname and location, disk layout, datastore location and network. If you want you can customize by setting a GW, DNS and IP for log insight. Default or blanks will give you DHCP. And let it fly.

Start your engines when it is all finished. Log on to the console and press CRTL ALT F1.


Login with root and blank password. This enables you to set a new password for root.

The vCenter Log Insight Web interface is available at http://log_insight-host/. The HTTPS-based secure Web interface is available at https://log_insight-host/.


When you access the vCenter Log Insight Web interface for the first time after the deployment, you must complete the initial configuration steps:

  • set the admin password and optionally a e-mail address.
  • set up a permanent or evaluation license key,
  • type in the e-mail address of the mailbox to receive notifications (some notifications about Log Insight are only send via e-mail notifications),
  • if you want you can participate in the Customer Experience Improvement Program select the thickbox of the send weekly option,
  • save and continue.
  • On the time page setup a NTP server, when none is available you can optionally sync with the ESXi host.
  • save and continue.
  • Setup the SMTP server details.
  • save and continue.
  • You can now setup the optional retrieval of vCenter events/tasks/alerts or send alert notifications to vOPS. (well optional, if you want central management set those options up. Leave out ops if you don’t have this.)
  • save and continue.
  • Set up an optional NFS archival location. You can also add more vmdk’s to your system for online data locations. But you will want to have some archiving in the future. For the evaluation I’m skipping this one.
  • save and continue.
  • Restart to complete the initial setup.


After the restart open a browser and viola the vCenter Log Insight home screen is shown. That was a smooth install.

This concludes the first part of the vCenter Log Insight evaluation. In the next part we will handle the following:

What to do next?

We need to configure some hosts to send syslogs to vCenter Log Insight. We can use to provide script configure-esx or we can use PowerCLI to setup a syslog host at the ESXi host advanced settings. We will use Log Insight to query log messages, set up alert notifications, import content packs and more. When I got my lab a bit more setup (I have a little resource issue) I will follow-up in a second post.

-Enjoy for now.

VMworld Defy Convention – VMware Now Live Notes.

Managing multiple devices. Accessing Private cloud from behind firewalls and demands from mobile users with multiple devices (personal and corporate) is growing. Is IT ready for the mobile world? Budget has not grown, investments are in the past. Datacenters are of the past. Adapt or die, how do we do that?

The time is now! Software defined datacenter. Virtualization of the complete datacenter. Hybrid cloud moves the datacenter from behind the corporate firewalls to the public cloud. Mobility platforms are there to connect mobile users and their mobile devices to the corporate data and applications.

Software Defined datacenter, transfer workloads from private to hybrid cloud and mobile workforces. Applications for the future. We build for the applications. One size does not fit!


I will try to find the time to blog more about some of the above (I have some other subjects). I also want to be prepared for VMworld EU in october.

Network, Storage and Management (automation) need to be further software defined and virtualized.

We are not done yet. Awaiting an exciting future!

Early access and beta’s are available.

Installing vCenter – CA Signed Certificates with the Certificate Automation Tool

With the installation of vCenter 5.1 components (just like vSphere hosts) the components are installed with self signed certificates. No problem when you trust your own systems, but securing your environment when you have different interactions with the services (for example the corporate users of the vSphere web clients) will lead you to implement signed certificates. Especially when the most environments have their own CA infrastructure, use it!

There are seven separate components in vCenter Server 5.1 that utilize certificates to encrypt communication. We want don’t want to spend time generating and installing certificates for each component, we want automation! And what do we have that, the VMware SSL Certificate Automation Tool. Well…..not a complete automation, but automated enough to help updating the vCenter related services (and rollbacking in case something is going wrong). That beats doing it all on yourself.

The VMware SSL Certificate Automation Tool can be used to generate certificate signing requests for the vCenter services, that can be send to the CA to be signed. The Automation Tool can then be used to implement the certificates for the services. Use the planner option to be presented with the correct order of steps. Possibly in a second screen.


The procedure is as follow:

– Download the VMware SSL Certificate Automation Tool.
– Generate CSR’s for the vCenter Service with the automation tool.
– Sign the CSR’s in the CA.
– Creating the PEM chain files.
– Update the vCenter services in the correct order. (Replacing one service and not re-trusting the interacting services will have some errors as result)


The installation starts with downloading the Automation Tool at https://my.vmware.com/group/vmware/get-download?downloadGroup=SSL-TOOL-101. Download and unzip to a preferred location. For this example I have unzipped to D:Scripts, a the folder structure starts from SSLAutomationTool1.0.1.

Predefining the default values in the tool helps prevent typing errors and save time, you can edit ssl-environment.bat. This lets the tool automatically include specific information that you have defined as the default instead of prompting you for it. Details for the serveral services we will fill in after generating the CSR (plus key’s) and PEM chain files.

For now we add the parameters to: Common parameters (SSO and VC users) and parameters that will be used to generate a CSR (fqdn, ips, country, state, organization and so on).

Now we start the tool: ssl-updater.bat.


We first generate the CSR by choosing action 2. At that menu we choose every service we have to update. With the common parameters we only have to take the default entries (be sure to check them). The CSR’s are saved to the requests directory. There you will also find the keys. Repeat until you have selected all options.

With all the necessary services done, we have a collection of CSR’s to be signed by the CA. The process of signing differs from what product is used as CA. From every directory the rui.csr is the signing request we have to upload or open and copy the contents.

For example Microsoft CA works with a web site you can use to request the certificate:


Choose the request a certificate and an advanced request. Request an BASE64-encoded web server certificate, with the option Submit a certificate request by using a base-64-encoded CMC or PKCS #10 file, or submit a renewal request by using a base-64-encoded PKCS #7 file.

You will have to repeat the request for every service you wish to update.

We continue when we have received the signed requests. Save the .crt file in a appropriate location and open the file to verify the proper names.

After repeating this we have to create PEM files with the complete certificate chain in one file. Depending on the hierarchy used in the certificate infrastructure you will need the root and all the intermediate (if any) Base64-encoded certificates. Get them from your trusted certificate store or from the CA signing service. Also save these files to the directory. Open a text editor and create a PEM file (name including the service). Open the signed certificate and copy and paste to the PEM, open any intermediate certifcate and past the contents to the PEM (after the —-END Certificate — of the certificate), repeat for any other intermediate. Finish by copying the root certificate in the PEM. This should look like something like this (truncated):

——-BEGIN CERTIFICATE——- <——-Certificate
——-BEGIN CERTIFICATE——- <——-Intermediate Certificate
——-BEGIN CERTIFICATE——- <——-Root Certificate

Close and repeat. And we now have the PEM chain’s for all (or you choice) of vCenter services.

Now we need to implement them at the appropriate services. First edit the ssl-environment.bat for the services and include location for keys and pem files. When you are finished the menu options Update will handle the updates for you.

Here is a screenshot of the the updating of the SSO service. It handles the update and restarting of the appropriate services.


Type 3 to return to the main menu and repeat for the other actions the planner .

In case something goes amiss rollback and you can find the service-update-ssl.log in the logs directory of the automation tool.

One more thing, when updating the vCenter certificate one of the wanted information is “Enter vCenter server original database password”. This is a somewhat confusing line. But it is the vCenter database user password (ODBC System DSN Password for connection to VCDB) that is requested. That is either a Windows user or a SQL user.

—- Enjoy updating!

Learned Lessons – Nexus 1000V and the manual VEM installation

At an implementation project we implemented 24 ESXi hosts and used the Nexus 1000V to have a consistent network configuration, feature set and provisioning throughout the physical and the virtual infrastructure. When I tried to add the last host to the Nexus it failed on me with the InstallerApp (Cisco provided java installer that adds the VEM and adds the ESXi host to the configured DVS and groups). Another option is to use Update Manager (the Nexus is a appliance that can be updated by update manager), but that one threw an error code at me. I will describe the symptoms a little bit later, first some quick Nexus product architecture so you will have a bit understanding how the components work, where they are and how they interact.

Nexus 1000V Switch Product Architecture

Cisco Nexus 1000V Series Switches have two major components: the Virtual Ethernet Module (VEM), which runs inside the hypervisor (or with other words on the ESXi host), and the external Virtual Supervisor Module (VSM), which manages the VEMs (see figure below). The VSM can be either a virtual appliance or an appliance in the hardware device (for example in the physical Nexus switch).
The Nexus 1000V replaces the VMware virtual switches and adds a Nexus Distributed Switch (Enterprise Plus). Uplink and portgroup definitions are bound to the Cisco ethernet profile configurations. The VEM and VSM use a control and data link to exchange configuration items.

Configuration is performed through the VSM and is automatically propagated to the VEMs. Virtualization admins can pick up these configuration to select the portgroups at VM provisioning.


Symptoms to a failing installation 

The problem occurred as follows:

– Tried to install the VEM with the InstallerApp. The installer app finds the host and when the deployment is done, it stops when adding the hosts to the existing Nexus DVS. This happens somewhere from moving existing vSwitches to the DVS. Error presented is: got (vim.fault.PlatformConfigFault) exception.

– Checked the status of the host in update manager and this showed a green compliant Cisco Nexus Appliance. This probably delayed me a bit, because it really wasn’t.

– Tried to manually add a host to the Nexus DVS in the vSphere Webclient. This gave an error in the task. Further investigation let me to the line in the vmkernel.log: invalid Net_Create: class cisco_nexus_1000v not supported. Say what?

– With the Cisco support site and a network engineer I tried some VEM cli on the host. But hey wait vem isn’t there. An esxcli software vib list | grep cisco doesn’t show anything either (duh). While on an VEM installed ESXi host this shows the installed VEM software version. So Update Manager is screwing with me.

Manual Installation

That leaves me with trying to manually install the VEM. With the Cisco Nexus 1000 installation guides the working sollution is as follows:

  • The preferred Update Managers does not work. It fails with an error 99.
  • copy the vib file containing the VEM software from the VSM homepage using the following url: http://Your_VSM_IP_Address/. Check an ESXi host that is installed for the running version (esxcli software vib list | grep cisco). Download this file (save as).
  • Upload this file to a location where the host can access this. On the host or a datastore accessible from the host. I did the latter as the host did not have direct storage. Used WinSCP to transfer the files to a datastore directory ManualCiscoNexus.
  • On the hosts I added this vib by issuing the following command:

esxcli software vib install -v /vmfs/volumes/<datastore>/ManualCiscoNexus/Cisco_bootbank_cisco-vem-v160-esx_4.

  • vem status -v now gives output. Look for VEM Agent is running in the output of the vem status command.
  • vemcmd show port vlans only shows the standard switches. Communication with the VSM is not yet there.
  • I added the host manually to the Nexus DVS and success. When migrating the standard vmkernel management port to the DVS groups the hosts is also visible on the VSM. Communication is flowing and the host is part of the Nexus 1000v.

I hope this post will help when you experience the same problem, and also learns you a little about the Nexus 1000V Product Architecture.

vCenter Collectors – Syslog and Dump collector (with some PowerCLI)

With stateless VMware Auto deploy hosts, or hosts with SD or USB Stick installation, these hosts typically do not have local storage attached. There will be no space assigned to the scratch partition for dumps and logs. Defaulted they will be stored in RAM (/tmp/scratch). After a reboot the /tmp will be cleared.

We need to redirect the scratch partition to a shared VMFS and a central log / dump location. Scratch redirection is performed with the following steps:

  • Decide on a shared data store where preferably all hosts in a physical location connect, beware of physical location Bias. The volume is like /vmfs/volumes/UUID
  • Create a locker directory on that data store, for example .locker-Hostname
  • Connect to the ESXi host with Connect-ViServer -Server <Hostname>
  • Set the scratch partition with Set-VMHostAdvancedConfiguration -Name “ScratchConfig.ConfiguredScratchLocation” -Value “/vmfs/volumes/UUID/.locker-Hostname”
  • Maintenance mode (Set-VMHost -VMHost <host> -State ‘Maintenance’) and reboot (Restart-VMHost <host>).

With central management of an environment, a central log/dump location is recommended (also for stateful deployments this should be a standard). For a central log we need some services added to the vCenter:

– dump collector; collecting ESXi dumps of the infamous PSOD.

– syslog collector; collecting the host logs or syslogs.

On Windows we use the vCenter installer to install both services. With the vCenter Server Appliance (VCSA) these service are bundled with the appliance.


Dump Collector installation

Installation of the dump collector is straightforward. Determine in advance where your dumps will be saved, how much space (2GB default is sufficient for most environments) and to do a standalone or integrate with vCenter (last option is preferred for single management).


(if you got a data disk, change the default C: parameters)


And fill in the vCenter and account details on the next screen. The account will be used to register the extension.

Syslog Collector installation

Just like the dump collector the installation is straightforward. Determine the syslog rotation and log location. And start the installation to the VMware vCenter Server.


(if you got a data disk change the default C:).


Change the ports when needed for your environment, standard default is sufficient for most environments.

And finish the installation.

If you still use the vSphere client, then install both service add-ins.

Configuring the hosts

The ESXi hosts need to be configured to use the remote syslog and network dump location. You can use ESXi CLI and the advanced system settings or you can use PowerCLI. If we use PowerCLI for syslog we can use the Set-VMHostAdvancedConfiguration cmdlet and for the dump collector we need the Get-EsxCli cmdlet to use esxcli.

When we do this for a list of hosts we can use the following script. Just replace the vcenter servername at $vCenterFQDN and vCenter IP at the $esxcli.system.coredump.network.set line.

#set vCenter IP/FQDN

$vCenterFQDN = “<vcenter server>”

#connect to vCenter

$viServer = Connect-VIServer -Server $vCenterFQDN -Credential (Get-Credential)

#get all hosts in vCenter so we can cycle thru them

$Hosts = Get-VMHost

#cycle through each host

ForEach ($VMHost in $Hosts)


   #Add Syslog to Syslog Collector

   Set-VMHostAdvancedConfiguration -VMHost $VMHost -Name Syslog.global.logHost -Value $vCenterFQDN

    # Add DUmp to collector

    $esxcli = Get-EsxCli -vmhost $VMHost

    $esxcli.system.coredump.network.set($null, “vmk0”, “<vcenter ip>”, “6500”)


    #write out host name that was changed:

    write “Updated host: $VMHost”


#disconnect from ESX server

Disconnect-VIServer -Server $viServer -Confirm:$false

Checking the configuration

A host can be checked by going to the dump directories and checking the contents, just like the following for a host’s syslog directory.


I don’t have a crashed ESXi host at my disposal, so it is easier to check on the hosts via esxcli:

esxcli system coredump network get.

The IP and port should be in the output at:

Network Server IP:

Network Server Port: 6500

VMware IO Analyzer – lab Flings

Flings in VMware labs is a great place for (very) useful tools or applications. This time I want to blog about a fling I often use in a test phase for implementation projects or in health assessments, see what synthetic load an environment can handle and if your vSphere design is up to the right io charactics and capacity.

Important in these kinds of test is your test methodology and plan: Assess, Filter test, plan, collect, analyse and report. With several of these steps IO Analyzer can be the player.

IO analyzer can configure, schedule and run several IOmeter workloads or replay vSCSI traces.

Download at:


Import ovf to your environment. Start with more then one, so you have some dedicated workers thoughout your environment.

After deployment change the second vmdk for the defaulted “small” configuration to approx 4GB plus (and Thick Eager). Why? Because the small amount of disk is used as disk test and fits in most storage cache. We need to get out of that and hit some real scenario’s.

One (or yes two) more things, logon to the consoles of all the appliances. Open a console, choose first option or press enter and login with root and password vmware. *ssst a very secret vmware user*. An other usage of the console is checking or monitoring the IOmeter tests i the console when they are running.


Type down one of the ip’s or hostnames of the appliances, and will use that one as the controller.

Open a browser (chrome or firefox) and type http:// and you will reach IO analyzer in your environment.

There we have the following options


For this I will use the workload configuration to add two tests to two appliances and check the results. Test scheduler is not used, will run immediately.

In this screen we first add the hosts where our test machines are, use the root password to connect to the hosts. When a connection is established the VM’s on that host are visible in the Add Workload Entry. Here you can find all kinds of IOmeter tests.


I have created two workloads, one Exchange 2007 on our first appliance and SQL 64K blocks on the other appliance. The duration is changed from the default 120 seconds to a 5 minute (300 seconds) schedule. This configuration is saved as Demo config.

Click on Run Now to let the test run. After a initialization you can see the progress in the console of one of appliances.


After completion of the tests you can view the test results in View Test Results (so it is not just a clever name :)).

Here you can check the two tests and the different VM and host metrics saved from IOmeter and esxtop (if there are any, you will get as a bonus the metrics of other VM’s on the hosts). More detailed information about these metrics, see the following URL: https://communities.vmware.com/docs/DOC-9279. Duncan Epping also has a good article about esxtop metrics and more. Go see his site when waiting for the test to finish: http://www.yellow-bricks.com/esxtop/

Here see the results of our Demo tests (I’m not going over in detail in this post).


Enjoy stress testing.

PowerCLI collection – Changing PSP

On storage controllers in combination vSphere 5.x multi path IO, path selecting policy should be Round Robin (always check if your storage supports this). When receiving the LUN it happens that those PSP are defaulted to Fixed. You can change the PSP policies of all attached storage with this PowerCLI oneliner (Round Robin in the example):

get-cluster -Name “Cluster” | Get-VMHost | Get-ScsiLun -LunType disk | Where-Object {$_.MultipathPolicy -ne “RoundRobin”} | Set-ScsiLun -MultipathPolicy “RoundRobin”

A little caution when you have storage from mixed storage providers (mixed storage controllers without a SVC,Vplex or such layer).

What does this one do:

– get-cluster -Name ”Cluster”; Retreives the cluster information for Cluster. This command can be replaced with get-datacenter. And pipe’s this information to:

– Get-VMHost; retreives the hosts information (one by one in from the cluster). The output is piped to:

– Get-ScsiLun – Luntype disk | Where-Object {$_.MultipathPolicy -ne “RoundRobin}

Get de Scsi Lun of type disk en checks if this object current policy is not equal with Rond Robin. The output is piped to:

– Set-ScsiLun -MultipathPolicy “RoundRobin. The Lun found with previous part is changed to a RoundRobin policy.

The policies can be set to:

RoundRobin, MostRecentlyUsed, Fixed or Unknown.

PowerCLI collection – Adding hosts to inventory

With several projects at several sites I have come along a nice list of powerCLI oneliners or scripts. I wanted a place to have these scripts in a handy catalogue online, maybe share and learn on some blogging skills (or be a little louder than only 140 characters). I know there are probably already place online, but..how can I learn the bloggin. So here goes…

This example is a script that adds 24 hosts to a vCenter inventory. ESXi is installed, vCenter and PowerCLI are setup. PoweCLI has a connection setup to the vCenter (Connect-VIServer -Server <vc> -Protocol HTTPS). No other automation or scripting is in place.

To add 24 hosts to a inventory:

1..24 | Foreach-Object { Add-VMHost hostesxi$_.example.nl -Location (Get-Datacenter DataCenter) -User root -Password “<replace>” -RunAsync -force:$true}

So what does this one do:

  •  1..24; Amount of hosts. This is piped to the command and used by the $_ variabel. Foreach-Object let’s us loop within the 1..24 range so the hosts number will change every loop.
  • Add-VMHost is actually the PowerCli cmdlet to add the host.
  • hostesxi$_.example.nl. Creates hostnames of hostesxi1_.example.nl till hostesxi24.example.nl in the datacenter DateCenter
  • -User and Password are the user and password for access to the ESXi hosts.
  • -RunAsync option is to let the tasks run parallel.