VMware Utility Belt must have tools – RVTools 3.6 released

This februari the 22nd RVTools version 3.6 was released. As I use this tool very often, and I notice on some occasions not all consultants are familiar with this tool, I wanted to write a post about RVTools to further spread the word.

This in my opinion is the tool each VMware specialist must have in his VMware utility belt together with the other standard presented tools. At this time RVTool is free and lightweight, very simple in usage, and budget is small (just a donation!) and will not have to be a constrain to use this tool.

What is RVTools?

RVTools is a Windows .NET application which used the VI SDK to display information about your VMware infrastructure.
Connection can be made to vCenter or a single host to get information about hosts, VM’s, VM Tools information, Data stores, Clusters, networking, CPU, health and more. This information is displayed in a tab view. Each tab represents a type of information for when clicking on the tab you will be displayed that specific kind of information from your environment.

RVTools can currently interact with Virtual Center 2.5, ESX Server 3.5, ESX Server 3i, Virtual Center 4.x, ESX(i) Server 4.x, Virtual Center 5.0, Virtual Center Appliance, ESXi Server 5.0, Virtual Center 5.1, ESXi Server 5.1, Virtua lCenter 5.5, ESXi Server 5.5.

RVTools can export the inventory to Excel and CSV for further analysis. The same tab from the GUI will be visible in Excel.



There is also a command line option to have (for example) a inventory schedule and let the results be send via e-mail to an administrative address.

Use Cases?

On site Assessment / Analysis; Get a simple and fast overview of a VMware infrastructure. The presented information is easy to browse through, where in the vSphere Web Client you would go clicking through screens. When there is something interesting in the presented data you can go deeper with the standard vSphere and ESXi tools. Perfect for fast analysis and health checks.

– Off site Assessment / Analysis; Get the information and save the Excel or CSV dump to get a fast overview and dump for later analysis. You will have the complete dump (a point in time reference that is) which you can easily browse through when writing up an analysis/health check report.

– Documentation; The dumped information can be used on or offline to write up documentation. Excel tabs are easily copied in to the documentation.

– (Administrator) reporting; Via the command tool get a daily overview of your VMware infrastructure. Compare your status of today with the point in time overview of the day before or last week (depending on your schedule and/or retention). Use this information in the daily tasks of adding/changing documentation, analysis, reporting and such.

Release 3.6 Notes

From the version information page, at 3.6 the following has been added:

  • New tab page with cluster information
  • New tab page with multi-path information
  • On vInfo tab page new fields HA Isolation response and HA restart priority
  • On vInfo tab page new fields Cluster affinity rule information
  • On vInfo tab page new fields connection state and suspend time
  • On vInfo tab page new field The vSphere HA protection state for a virtual machine (DAS Protection)
  • On vInfo tab page new field quest state.
  • On vCPU tab page new fields Hot Add and Hot Remove information
  • On vCPU tab page cpu/socket/cores information adapted
  • On vHost tab page new fields VMotion support and storage VMotion support
  • On vMemory tab page new field Hot Add
  • On vNetwork tab page new field VM folder.
  • On vSC_VMK tab page new field MTU
  • RVToolsSendMail: you can now also set the mail subject
  • Fixed a data store bug for ESX version 3.5
  • Fixed a vmFolder bug when started from the command line
  • Improved documentation for the command line options

Pretty fly…


RVTools is written by Rob de Veij aka Robware. You can find Rob on twitter (@rvtools) and via his website http://robware.net.
Big thank you Rob for letting this excellent tool in the cyberspace!

As the tool is currently free please donate if you find the application useful to help and support Rob in further developing and maintaining RVTools.

Let’s get ready to rumble – voting for the 2014 top VMware & virtualization blogs is open!

Eric Siebert (@ericsiebert) from vsphere-land.com yesterday opened this years voting for the 2014 top VMware & virtualization blogs.
There are some plus 300 blogs participating on this years ballot. And new for this year is some great prize giveaways courtesy of Veeam.

A lot changed from the 2008 beginning to this years voting. You can read all about the history in Eric’s http://vsphere-land.com/news/the-history-of-the-top-blog-voting-over-the-years.html
post. A big thanks you is in order to Eric for setting up this contest every year.

I am honoured to be part of the list of great VMware and virtualization blogs. I started a half year ago and this blogging stuff is hard work.
But I’m not ready yet. If you liked any of my content, please remember my blog for the voting. Yes there are a lot of good blogs on vLaunchpad (http://planet.vsphere-land.com/),
I will not be disapointed that much if I’m not in this years top lists. There is always next year(s). My content (and hopefuly my skills) will grow.

For this year voting can be done at this linkie : http://www.surveygizmo.com/s3/1553027/Top-VMware-virtualization-blogs-2014.
Voting can be done until 17 March 2014.


The voting procedure in short:

  • Step 1, Vote for your ten blogs for the top 50, when you reach ten the checkboxes will be greyed out. The current top 50 (that is the 2013) are in bold in the list. You can find Pascal’s Wereld on the 2014 ballot. But please vote accordant to your own taste.
  • Step 2, Drag the ten votes in the appropiate order. Rank 1 gets 10 points, Rank 10 get 1 point.
  • Step 3, Choose the favorites in the presented categories. If you didn’t pick blogs in the top 50, you can still vote for them in the categories. The top 50 and category lists are independant. Or you can go for the none if you accencidental never visit those blogs. Don’t forget Pascal’s Wereld in the New Blogger and independent blogger categories.
  • Step 4, Enter your name and e-mail address to be included in the price give a way. And fill in the nice Captcha.
  • Step 5, Submit.
  • Step 6, Tell others that you voted and let them vote as well.

Be honest in your voting and your selection of blogs. A little warning, there is some protection like Captcha, one vote per IP adress, location etc. If there is any indication of fraud these votes will be removed. If your in a organization/company network, be sure to be the first to cast the vote over your colleagues (and direct them to their home network to cast their vote)

– Please join the voting!

Being part of Nutanix blogger session and opening of EMEA HQ

On februari the 18th I, together with other bloggers from the Netherlands (the netherlands as a whole, not just one country), was invited to an informal Nutanix blogger session. One plus one is that I should make a blog about that.
At this session we were updated on the Nutanix EMEA HQ opening and strategy with the community and Nutanix. The update session was at the new EMEA HQ of Nutanix in Hoofddorp the Netherlands that was opened on the same day. At the blogger session founder / CEO Dheeraj Pandey (@trailsfootmarks) and Howard Ting (@howardting),VP Marketing/Product Management were also present. Great meeting these guys, and all the others from the Nutanix family. And also very nice to meet some of the community in person.


For the bloggers this session was all about how Nutanix needs the community and how the community can be an important part for Nutanix.

And we got a hard copy of the Nutanix special edition Software-Defined Storage for Dummies. You can grab your free e-version at http://go.nutanix.com/DummiesBook_SDSD.html.


(liked I tweeted a good combination bubbles, bloggers, software defined storage and opening of Nutanix EMEA )

But first a little about the Nutanix solution as I have not yet written about this subject (only named them in some IO articles).

Small introduction to Nutanix

Nutanix is a (hyper)converged infrastructure solution that consolidates the computing (server) tier and the storage tier into a single, integrated appliance. Nutanix calls this the Nutanix Virtual Computing Platform. The integrated appliances work in a modular building block design, what makes it a simple and scalable solution. This makes it easy to start with a small deployment and scale incremental to a large cluster solution. Because of the consolidation of the computing and storage tier, Nutanix eliminates the requirement for a complex storage infrastructure. This introduces simplicity to your infrastructure, and cuts down in deployment en engineering/maintenance costs. This furthermore keeps to IO within the Nutanix building blocks, at which your workloads gets it’s IO without having to move downstream and upstream to storage infrastructures. The simplicity goes even further with this solution by delivering rapid deployment without disruptions, and easy management from a single management layer.

Each Nutanix instance or node runs a hypervisor (VMware ESXi, KVM, Hyper-V are currently supported) and a Nutanix controller VM (CVM). This controller VM handles all IO operations for the nodes local hypervisor and VM’s.

An architecture model for a Nutanix node taken from the Nutanix web site.


When the Nutanix nodes are grouped together they form a distributed platform which is called the Nutanix Distributed File system (NDFS). To the hypervisor the NDFS appears to be a centralized storage layer, but from within the Nutanix solution the IO is handled local on the nodes to provide the highest performance to the required workload/User VM’s. And by forming a group, NDFS leverages techniques including striping, replication, auto-tiering, error detection, fail over and automatic recovery. By presenting centralized storage pools, hypervisor features like high availability, vMotion and DRS can be leveraged as well.

A perfect match for VD, for virtualizing business critical applications like MSSQL and Exchange and elastic big data.

The community

Nutanix acknowledges the importance of the community and is willing to be part of the community, and let the community be part of the Nutanix magic. As discussed at this blogger session, the community is needed to keep the company (and others in the digital space) honest and real. The company needs feedback from the community about technical and even strategical choices. It’s great of a company to reach out and acknowledge the importance of a community for a product, with a added bonus of an in person session. It takes two to tango….
Most bloggers will not be one subject biased, but we all have our favorite subjects and knowledge background were we can be of a specific assistance. Great solutions will be broader adopted in the (enough with borders) community. That is a step one, great for doing some more steps and match the community to some of the parts in the Nutanix solution.

Thanks Nutanix

A big thank you is in order for the invitation as I am just trying to get used to the community and practicing my blogging skills. Now let’s see how I can be part of this Nutanix community magic…..let’s see what will be `next’…..

vSphere the statistics gathering

For VMware vSphere infrastructures and the how and why my environment is doing that, it is helpful to understand how vSphere and vCenter standardly collect and store statistics, and how these are displayed. The point here is that there is an awful lot of assumption in performance reports or troubleshooting done. When does there assumptions come in to play? When looking at several counters and the way the data is collected (and the what), storage and graphs are made. Especially when selecting intervals how to display (or gather), peaks can become less because they’ve been averaged out over the displayed graph time of the historical (historical vs realtime) or metrics are missing when needed.

How does the statitics gathering work?

Each host stores statistics data for up to a hour via the local performance manager. The performance manager receives realtime instance data from for example the CPU instances. Within the vCenter data collection interval, vCenter performance manager queries each host (that is the hosts that are managed by this vCenter) and vCenter will retrieve a subset of the host statistics data, and stores it in the vCenter database. When, what and how much are configurable in your VMware infrastructure. We have two values for that, interval and levels that can be set on the vCenter.

These collected historical metrics can then be displayed via the vSphere client.

A small model taken from the VMware Documentation Center.


Statistics rollups and intervals

As the ESXi host collects the statistics realtime (that is every 20 seconds) these are rolled up to the vCenter database for historical purposes. The vCenter collects data from all of the hosts that the vCenter Server manages. The PerformanceManager defines performance intervals that specify time periods for performance data rollup, a methodology for combining data values. The server stores the rolled up performance counter data in the vCenter database. This is done in four performance intervals that determine how collected instance data is aggregated and stored. The aggregated data is a set of instance data values collected for a performance counter. These intervals can be modified to a limited extent via the collection intervals. These determine the duration for which statistics are aggregated, calculated, rolled up, and archived. Together, the collection interval and collection level determine how much statistical data is gathered and stored in your vCenter Server database.


Are those rollups evil? Yes they can be. Peaks can become less because they’ve been averaged out over the graph time. But you must not forget the fact these are historical data, graphs made for a month with 12 data points can show different peak values than week graphs. Know what you are looking for, for which periode and don’t base your conclusion on just one graph.

Statistics Levels

To reduce traffic to the vCenter database vCenter uses a technique to limit which metrics are archived in the database. Certain statistics might be deemed more valuable for your like then others. The statistics levels varies from one to four, with one being the least-detailed statistics level and four being the most detailed.

  • Level one; statistics cluster Services, CPU, Disk, Memory, Network, System, and Virtual Machine Operations counters. Default level.
  • Level two; level one plus all disk, memory and VM operations metrics. Use for long term performance monitoring when device statistics are not required but you want to monitor more than the basic statistics.
  • Level three incorporates level one and two plus per-device statistics, such as CPU usage of a host on a per-CPU basis, or per-virtual machine statistics .Use for short-term performance monitoring after encountering problems or when device statistics are required.
  • Level four. All possible metrics. Use for short-term performance monitoring after encountering problems or when device statistics are required. Only to be used in the shortest amount of time due to the large quantity of data.

The statistics level is used to dictate whether or not a statistic is stored in the vCenter database. If a metric is a level two statistic, but vCenter is configured to level one, this metric is not stored in the database. Not stored means users are not able to query its historical values. Not a problem all the time, but also not good to have if you are looking for just these counters.

These levels sure have their benefits (information at minimal database costs) and drawbacks (possible missing metrics), but the ability to have and change the statistics levels gives something back. We can gather basic information at minimal database cost for the normal running environment with the level one counters. When needed at a troubleshooting scenario we can temporarily increase the statistics level to get more detailed information.

– Happy statistics gathering!

Should I hot add CPU/Mem or go now?

I am doing my VCAP5-DCA prep and using the unofficial official VCAP5 DCA Study Guide to guide me true the subjects (great resource! Check it out at http://www.virtuallanger.com/vcap-dca-5/). And apparently I’m easily distracted, this time with the subject of resource optimization/management and in particular the hot plugging CPU and memory options.

As you probably know there are several device options you can add to a running VM like virtual nics and virtual disks. vSphere 5.5 even has support for hot add/remove of PCIe SSD. These normally work hot out of the box.
For CPU and memory (the virtual ones vCPU and vRAM) this is not the case. They are disabled by default. Why? Well because support of the guest OS is limited, not all hypervisor features are operable (for example FT) and for applications running in the VM support is even more limited. Often you will have to recycle the application service to let it use the added resource or there is possibility of resource degradation/stability. That means downtime and that isn’t what hot add/plug is about. What is the point of using hot add/plug when there is downtime involved? Sure a guest OS restart vs an application service restart takes probably more time on the OS part, but applications work in chains you will know which part (or all) of the layers need attention. This tends to count more for the vCPU hot add, memory hot add/plug is longer around and is incorporated in more guest OS’ses/applications then hot plug vCPU is.

Secondly if you haven’t selected hot add/plug/remove vCPU and vRAM in the creation of your VM (or template), and enabled guest OS support, you will have to power down the VM before being able to change these options. Planning ahead (capacity management) is key here, but that goes for resource management as well. You might as well adjust the resource values you require before powering on the VM.

And what about with elastic resources, is there a hot remove option? For vSphere this is currently a no no, no matter what the guest OS support on this is (on some of the Linux distributions, Windows is also a no no). Memory remove is only done by powering down the VM.

What OS’ses support hot add/plug/remove CPU/Memory?


NB. I have left out Windows Server 2003. Yes there is support in there, but this product is bound for EOL. If you are thinking on hot add for this product, rather start thinking about doing a step up in life cycle.

NB. For Linux I just included some distributions. There are of course more supported on VMware.

What applications support hot add/plug/remove CPU/Memory?

This is the harder list to create. For a very few there is public information on support (or just it can be done) of hot add vCPU/vRAM. For applications it is harder to determine whether the resource is added to the OS for usage in the application or for other purposes. Applications must support multi processors to take the full advantage of a virtual socket that is added.

IIS? No. Needs a recycle of service or application pool.

SQL Server? Needs Enterprise edition. After adding vCPUs, execution of the RECONFIGURE statement is needed before it is used.

Exchange? A maybe, again a recycle. There are performance issues rumored when using these features, some have to do with paging depletion afterwards.

Be sure to test test test.

What are you prerequisites to hot add/plug/remove CPU/Memory a running VM?

  • The virtual machine has a guest operating system that supports hot functionality, and must be turned on in the OS.
  • The virtual machine is using hardware version 7 or later. With KB 2050800 there is mentioned an issue with 2012 and windows 8 on hardware version 9 (5.1).
  • VMware Tools is installed.
  • Hot add must be enabled per VM, at the creation time or by powering off.
  • The hardware must be able to support hot plug as well, else there must be resources available to evict the VM’s to an other host while doing maintenance on the physical host.
  • Does not need FT.
  • Does not need vNUMA.
  • Use vSphere Essentials, Essentials plus, Standard, Enterprise or Enterprise Plus. The are limitations per edition you will have to take in account, and versions (for example 5.0 standard edition will not let you use hot add)
  • Check application licensing, how will this effect a per core or per processor application license.

Edit settings of VM to change hot plug/add.




Do we need to enable hot plug by default on my VM’s? Depending on the usage, only set it on for VM’s that have a tendency to run out of resources quickly, are very in demand and cannot tolerate a little down time in re-configuring. For a small part of the private desktops of the VD environment this can also be a good option to configure before hand, this is normally is a small group of your users for example developers with some private type desktops. For the standard bunch of VM’s/VD’s leave it off, like the products default. You can normally plan ahead the resources for these kind of VM’s. vCenter Operations is great for resource capacity planning. So plan ahead and enable default on all your VM’s? No not yet, there is to little application support out there specially on the vCPU’s. Planning ahead means knowing some more variables and they are not around yet (on the application level that is). Planning ahead is having your resource configured and monitored correctly, when there is need to change, change the resource settings.

The limited OS support and very limited application support is the why to the non adoption of this hot plug features that have already been around a little while. Will we expect it to grow with all the cloud movements and software defined data centers? This depends, when there is a need it will be incorporated in more applications. This list is currently small.

– Back to studying.