Reducing IO @ the host

Workloads need IO resources to operate correctly and with good response in an information chain. No performance is many unhappy users and business processes grinding to a halt.

IO can be networking and storage in this case. As networking IO mostly is less of a problem (but the main line goes for networking IO also), main focus of this blog post is storage IO.

In my opinion it is important to handle or reduce the IO closest to the source before letting it get down to other infrastructure components. The effect should be that the IO downstream can be handled by a smaller simpler storage device or preferably none.
The first side effect of this is that it cuts in the complexity of the infrastructure, costs of handling IO at the storage layer and simplifying project tasks as they won’t need a over-tasked storage engineer or wait for parts to come in (and take the thing partly down and figuring that controllers where not failover tested that much). To put it simple, cuts cost and time.
The second side effect of this is that it improve VM to host consolidation ratio’s. Depending on the sort of VM workload this can be an advantage. With virtual desktops (VD’s) as they will be part of much to same IO workloads (Windows 7 images) to effectiveness of some of the host based solutions will increase.

Most important (unfortunately not seen that often) is knowing your organizations workloads and their IO requirements. VDI is commonly write intensive so trying to reducing read isn’t going to give that much and your storage will still be hit hard. Other application parts are more read heavier. Etc.

What are our options that I have come across?

  1. Reduce guest IO footprint – OS. Optimize your guest OS so that it needs less CPU, memory and IO. Less to a certain amount that is. If the workload is letting your OS swap to disk because of low memory, IO on the storage will be heavier then sizing the memory to include the workload (and stop swapping). Know the correct sizing of your organizations image. Optimization of Windows can be done via the several optimization guides available (eg. Windows 7 via VMware or Citrix or with tools like VMware OS Optimization Tool Fling
  2. Reducing guest IO footprint – Virus protection. Moving the virus protection from the guest to the Hypervisor layer by using McAfee MOVE or Trendmicro Deepsecurity. These products use a hosts based virtual appliance to plug directly to the hosts hypervisor (by using vShield for example). Seriously reducing CPU cycles, memory consumption and storage IO coming from the guest.
  3. Reducing VM IO by offloading swap. Benefits of offloading swap files to host local (flash) storage is the reduction of the space footprint and to offload read and write IO’s to shared storage on to local storage (which in turn must be able to handle the amount of IO, preferably via SSD).
  4. Reducing VD storage requirements by using composer linked clones or PVS vdisks. With View Composer you create desktop images that share virtual disks with a base image, so you can reduce the required storage capacity.
    View Composer uses a base image, or parent virtual machine, and creates a pool of up to 1,000 linked-clone virtual machines. Each linked clone acts like an independent desktop, yet the linked clone requires significantly less storage. When placing the clones disks on local accelarated storage even more response can be offered to the desktop.
  5. Host caching and deduplication. Several so called accelerators are available to cache IO on flash or RAM, and are able to do inline data deduplication. They work at the hypervisor level by introducing a virtual appliance or hypervisor module, which can be clustered for fault tolerance. These solutions give your workloads more IOPS at lower latency (milli to micro). They can be shared storage backed, but with lower requirements and mainly for capacity. I’m thinking about Atantis ILIO, Infinio and PernixData. Cost effective solutions for better responses.
  6. IO handling at host cluster – Virtual SAN or VSAN. Radical simple storage from pools (cluster) of vSphere hosts. VSAN uses flash disks (SSD) as a read cache and write buffer. The read cache keeps a list of commonly accessed disk blocks to reduce I/O read latency in the event of a cache hit (that is, the disk block is in cache). The write cache behaves as a non-volatile write buffer, reducing latency for write operations as well.
    In the event of a host failure a replicated copy of cache, as well as the replicated disk data are available on one or more VSAN cluster hosts. See more at: Unfortuanally in Beta, so not production worthy yet.
  7. IO accelerator cards – FusionIO or HP Accelarator cards. Accelerating flash with PCI Express Cards. Integrates with servers at the system bus and kernel level. Lot’s of IO performance with minimal power consumption. Gives more IO then SSD. Great for large data centers. Costly solutions but offers very much IOPS at the host level.
  8. (Hyper)converged architectures. All your Infrastructure or data center resources in a box with a single management layer. No need for complexity. Easily scalable (it is just like Lego with those blocks), great for starting small and grow incrementally when needed. Includes storage features (depending on system that is) such as flash acceleration on several layers (SSD, PCIe and such), deduplication, compression and replication all at the hosts/block level. Thinking about Nutanix or SimpliVity here.

Of course there will be other solutions out there. Like written before, these are just to ones I have come across. Know of some that are certainly be bound to be in this post, drop a comment and I will take a peek to include them.

– Happy reducing IO @ the host.

Time for 2013 recap – Starting a blog

As 2013 is coming to an end (and my holiday break is coming up) the time for looking back at 2013 and looking forward to 2014 is starting.

2013 is a productive year with starting my blog in august, getting some 35 37 blog posts out there (and maybe some more before the year ends), getting some notice on twitter, visiting VMworld Europe and successfully passing my VCAP5-DCD. Great goals with great finishes (if I may say so myself :-))!

But what will be 2014 like? I’m hoping to get my VCAP5-DCA and start on VCDX, and get VCAP-DTA. Hopefully I can keep up the blogging pace (not easy with work and family life, respect to all those that are doing this) and my content is somewhat interesting for the readers (and if not please do tell, also when you like it ;-) ).
And hopefully I will be able to be part of the VMworld magic again. Goals Goals goals. To be honest I have not yet been making that much targets, so maybe there will be some more on the list.

And yes, looking back is also a statistics thing, so to finish this post some of my blog 2013 statistics; let see what my top five best read posts are for 2013.

*drumroll* My top 5:

  1. Webcommander. Walkthrough of the Webcommander Lab Fling. Been at the top for a while now. Posted at:
  2. My VCAP5-DCD Experience to a succesful certification. Posted at:
  3. Veeam One v7. Evaluation of Veeam One v7. Posted at: 
  4. Veeam Backup and Replication v7. Evaluation of another release of the great Veeam product. Posted at
  5. Release of VMware Horizon View 5.3. Posted at:

Well If you would like to see the other posts please visit my blog at or for a faster service to the archive page.

To end this post, have a nice Christmas and a superb ending of 2013. I wish you all the best for 2014!

NTP and the hey didn’t I set it up woes

Once in a while I come across a production network where there are some unexplained issues in the environment. Some (or a lot depending on the environment) can be because of time synchronization woes. Mostly because of undocumented or partly unconfigured time services.

Why is this an issue?

Time is inherently important to the function of routers, networks, servers, clusters, applications, storage or name it. Without synchronized time, accurately information between hosts becomes difficult, if not impossible. When it comes to analyzing and security, if you cannot successfully compare logs between each of devices, you will find it very hard to develop a reliable picture of an incident. Some or part of application clustering will fail. Application services are waiting on data packets that will not be processed. Locks are not removed in time. If an authentication time stamp coming from the client differs with more than default 5 minutes from a Domain Controllers time, it will discard the packet as fake (or think what your two factor will do). Storage controllers providing CIFS access need the same access to the directory as the previous client example. Performance data cannot be accurately interpreted if the time stamps on the data are not synchronized between the managed and management components. When wanting to present logs as proof, even if you are able to put the pieces together, not synchronized times, may give an attacker with a good attorney enough wiggle room to escape prosecution. These are just some of the things that can be affected with a mis-configured or not in place time synchronization.

Often there isn’t anything in the standard installation which gives the engineer a chance to setup NTP. Proper NTP usage only occurs when the engineer is knowledgeable of NTP and its administration, and has as standard practice of configuring NTP as one of the post-installation tasks. This is often not the case or is partly done and documented.

And if you don’t have a blue box around, Wibbly Wobbly Timey Wimey stuff is not what you want!

But what is in a NTP architecture?

NTP is designed to synchronize the time on a network of hosts. NTP runs over the User Datagram Protocol (UDP), using port 123 as both the source and destination. 

A NTP network usually gets its time from an authoritative time source, such as a radio clock or an atomic clock attached to a time server (a stratum 1) or one or more trusted time servers in the hierarchy. NTP then distributes this time across the network. Depending on your network and available time sources several options of your NTP infrastructure can be done. But often you will see a hierarchical structure with external time source on the Internet ( for example). Core infrastructure components (core router, firewall or Active Directory Domain Controller) have a client-server relationship with external time sources (and are allowed through the firewall), the internal NTP services have a client-server relationship with the internal servers, devices et al, the internal workstations have there time services in a client-server relationship with the time synchronized server (via Windows services in this example), and so on down the tree. A hierarchical structure is the preferred technique because it provides consistency, stability, and scalability.


Relations and configurations

In the virtual world there will be varying resources. On busy systems, resources may be denied for short periods of time or during high workloads. Some may receive higher resources. This will result in something referred to as time drifting, the clock ticks will sometimes run in a faster or slower pace creating an offset. This really show to need for time synchronization.

NTP prefers to have access to several sources of time (preferable three) since it can then apply an agreement algorithm. Normally, when all servers are in agreement, NTP chooses the best server in terms of lowest stratum, closest (in terms of network delay), and claimed precision.

For external NTP sources a path must exist from the trusted device to the external sources, firewall rule must allow this traffic. When using fully qualified domain names in the configuration (not always possible but preferable) the NTP client relies on the DNS client to resolve. DNS must be set-up correctly.

With a Windows Domain your Windows Members are automatically configured to use Windows Time Service to their domain controllers (that is with the NT5DS setting). The domain controller (or the PDC emulator to be precise) needs to be manually changed to type NTP and setup with a peer list.

In other devices it is a configuration tab you can set from the web interface, other have CLI in a Linux or other OS. Linux can be setup to use the NTP daemon.

For VMware VM’s use the guest operating system time synchronization such as Windows w32tm or NTP, and not use VMware Tools time synchronization.

Be ware that all your devices, hosts, guests, database servers, application, services et al are working together and need to have the same reliable source of time. As a rule you will have to set up your time service, this is not done for you.

Set it up correctly and document for reference.

– Happy timey wimey!

vSphere Performance monitoring tools standard available

I am currently working on a project where we are optimizing the virtual infrastructure which consist of vSphere and XenServer hypervisors. In the project we want to measure and confirm some of the performance related counters. We got several standard tools at the infrastructure components to see what the environment is capable of and check if there are some bottlenecks regarding IO flow and processing. 

With any of the analyzing it is important to plan (or know) what to measure on what layer so this is repeatable when wanting to check what certain changes can do to your environment. This check can also be done from some of the tools available, such as earlier written in the blog post about VMware View planner (to be checked at this url or is a repeat of your plan (which then can be automated/orchestrated). Your measuring tools need to have similar counters/metric throughout the chain, or at least show what your putting/requesting from a start and at the end (but if there is a offset you got little grey spots in the chain).
A correct working time service (NTP) is next to correct working of for example clustering and logging, also necessary for monitoring. To get to right values at the right intervals. Slightly off will in some cases give you negative or values that are off at some components.

Some basics about measuring

You will have to know what the measuring metrics are at a point. Some are integers, some are floating, some are averages over periods or amounts used, some need a algorithm to calculate to human or a similar metric (Kb at one level and bytes on the other, some of them are not that easy). A value that is high in first view but consists of several components and is an average of a certain period, could be normal when devided by the amounts of worlds.

Next up know or decide on your period and data collection intervals. If you are measuring every second you probably get a lot of information and are a busy man (or woman) trying to analyze trough all the data. Measuring in December gives a less representative workload then measuring in a company’s peak February period (and for Santa it is the other way around ;-)). And measure the complete proces cycle, try to get a 4 weeks/month period to get the month opening and closing processes in there (well depending on the workload of course).

Most important is that you know what your workloads are, what the needs for IO is and what your facilitating networking and storage components are capable off. If you don’t know what your VD image is build of for a certain group of users and what is required for these, how will you know if a VD from this groups requesting 45 IOPS is good or bad. At the other hand if you put all your management, infrastructure and VD’s on the same storage how are you going separate the cumulative counters from the specific workload.

Hey you said something about vSphere in the title, let’s see what is standard available for the vSphere level.

VM monitoring. In guest Windows Perfmon counters or Linux guest statistics. The last is highly depending on what you put in your distribution, but think of top, htop, atop, vmstat, mpstat et al.
Windows Perfmon counters are supplemented with some VM insights with VMware tools. There are a lot of counters available, so know what you want to measure. And use the data collection sets to group them and have them for reference/repeatable sets (scheduling of the data collection). 

– Host level; esxtop or vscsistats. Esxtop is great tool for performance analysis of all types. Duncan Epping has an excellent post about esxtop metrics and usage, you can find it here Esxtop can be used in interactive or batch mode. With the batch mode you can load you output file in Windows Perf mon or in esxplot ( Use VisualESXtop ( for enhancements to the esxtop commandline and a nice GUI. On the VMA you can use resxtop to remotely get the esxtop stats. vscsistats is used when wanting to get scsi collections or get storage information that esxtop is not capable of showing. And ofcourse PowerCLI can be an enormous help here.

vCenter level; Statistics collection which depends on your statistics level. Graphs can be shown on several components in the vSphere Web Client, can be read via the vSphere API or again use PowerCLI to extract the wanted counters. To get an overview of metrics at the levels please read this document or check documentation center for your version.

– vCenter™ Operations Management Suite (vCOPS). Well standard, you still have to option to not include operations in your environment. But your missing out on some of the automated (interactive/proactive) performance monitoring, reporting and insight in your environment options. Root cause analysis is part of the suite, and not down to your own understanding and analytic skills. If you are working on the previous levels your life could have been simpler with vCOPS suite.

Next up

These standard tools need to be supplemented with specific application, networking (hops and other passed components) and storage (what are the storage processors up to is there latency build up in the device it self) counters.

– Happy measuring!

Managing multi-hypervisor environments, what is out there?

A little part of the virtualization world I visit are in the phase of doing multi-hypervisor environments. But I expect more and more organizations to be not one type only and are open to using a second line of hypervisors other then their current install base. Some will choose on specific features or on product lines for specific workloads or changing strategies to opensource for example.

Some providers of hypervisors are having or bringing multi support to their productlines. VMware NSX brings support for multi-hypervisor network environments via the Open vSwitch support in NSX (with a separate product choice that is), where XenServer leverages the Open vSwitch as an standard virtual switch option. Appliances are standard delivered in the OVF format. Several suites are out there that claim a single management for multi-hypervisors.

But how easily is this multi-hypervisor environment managed and for what perspective? Is there support in only a specific management plane? Is multi-hypervisor bound to multi-management products and thus adding extra complexity? Let’s try and find out what is currently available for the multi-hypervisor world.

What do we have?

Networking, Open vSwitch; a multi-layer virtual switch which is licensed under the open source Apache 2.0 license. Open vSwitch is designed to enable network automation through programmatic extension, and still supporting standard management protocols (e.g. NetFlow, sFlow, SPAN, RSPAN, CLI, LACP, 802.1ag). Furthermore it is designed to support distribution across multiple physical servers similar to VMware’s distributed vswitch concept. It is distributed standard in many Linux kernel’s and available for KVM, XenServer (default option), VirtualBox, OpenStack and VMware NSX for multi-hypervisor infrastructures. Hyper-V can use the Open vSwitch, but needs a third party extension (for example using OpenStack extension). Specifically for networking, but it is a real start for supporting true multi-hypervisors.

Transportation, Open format OVF/OVA; Possibly the oldest of the open standards in the virtual world. Open Virtualization Format (OVF) is an open standard for packaging and distributing virtual appliances or more generally software to be run in virtual machines. Used for offline transportation of VM’s. Wildly used for transporting appliances of all sorts. Supported by muiltiple hypervisor parties, but sometimes conversion are  needed especially for the disk types. OVF’s with a VHD disk needs to be converted to VMDK to be used on VMware (and vice versa). Supported by XenServer, VMware, Virtualbox and such. OVF is also supported for Hyper-V, but not in all versions of System Center Virtual Machine Manager support importing/exporting functionality. OVF allows a virtual appliance vendor to add items like a EULA, comments about the virtual machine, boot parameters, minimum requirements and a host of other features to the package. Specifically for offline transportation.

VMware vCenter Multi-Hypervisor Manager; Feature of vCenter to manage other hypervisors next to ESXi hosts from the vCenter management plane. Started as a VMware Lab fling, but now a VMware supported product (only support for the product, underlying Hyper-V issues are still for the Microsoft corporation) available as a free download with a standard license. Currently at version 1.1. Management of host and provisioning actions to third party hypervisors. Supported other then VMware hypervisors is limited to Hyper-V. And to be honest not primarily marketed as a management but more a conversion tool to vSphere.

vCloud Automation Center (vCAC);  vCloud Automation Center focuses on managing multiple infrastructure pools at the cloud level. You can define other then vSphere endpoints and collect information or add these computing resources to an enterprise group. For certain tasks (like destroying a VM) there still is manual discovery necessary for these endpoints to be updated accordantly. But you can leverage vCAC workflow capabilities to get over this. Uses vCAC agents to support vSphere, XenServer, Hyper-V or KVM hypervisors resource provisioning. Hypervisor management is limited to vSphere and Hyper-V (via SCVMM) only. vCAC does offer integration of different management applications for example server management (iLO, Drac, Blades, UCS), powerShell, VDI connection brokers (Citrix/VMware), provisioning (WinPE, PVS, SCCM, kickstart) and cloud platforms from VMware and Amazon (AWS) to one management tool. And thus providing a single interface for delivery of infrastructure pools. Support and management is limited as the product is focussed on workflows and automation for provisioning, and not management per se. But interested to see what the future holds for this product. Not primarily for organisations that are managing their own infrastructures and servicing only their own. Specifically for automated delivery of multi-tenant infrastructure pools but limited.

System Center Virtual Machine Manager (SCVMM); A management tool with the ability to manage VMware vSphere and Citrix XenServer hosts in addition to those running Hyper-V. But just as the product title says, it is primarily the management of your virtual machines. As SC VMM can be able to read and understand configurations, and do VM migrations leveraging vMotion. But need to do management tasks on networking, datastores, resource pools, VM templates (SCVMM only imports metadata to it’s library), host profile compliancy (and more) or fully use distributed cluster features you will need to switch to or rely on vCenter to do this tasks. Some actions can be done by extending SCVMM with a vCenter system, but that is again limited to managing VM tasks. Interesting that there is support to more then one other hypervisor with vSphere and XenServer support. And leveraging the system center suite gives you a data center broader management suite, but that is out of scope for this subject. Specifically for virtual machine management, and with another attempt to get you to convert to the primary hypervisor (in this case Hyper-V).

Other options?; Yes, automation! Not a single management solution but more of a close the gap between management tasks and support of management suites. Use automation and orchestration tools together with scripting extension to solve these management task gaps. Yes, you still have to have multiple management tools, but you can automate repetitive tasks (if you can repeat it, automate it) between them. PowerShell/CLI for example is a great way to script tasks in your vSphere, Hyper-V and XenServer environments. Use a interface like Webcommander (read at a previous blog post to present a single management interface to your users. But yes, here some work and effort is expected to solve the complexity issue.

– Third parties?; Are there any out there? Yes. They are providing ways to manage multi-hypervisor environment as add-ons/extensions that use already in place management. For example HOTLINK Supervisor adds management of Hyper-V, XenServer and KVM hosts from a single vCenter inventory. And Hotlink hybrid express adds Amazon cloud support to SCVMM or vCenter. Big advantage is that Hotlink is using the tools in place and integrate to those tools so there is just a minimal learning curve to worry about. But why choose a third party when the hypervisor vendors are moving there products to the same open scope, will an addon add extra troubleshooting complexity, how is support when using multiple products from multiple vendors where does one ends and the other starts? Well that’s up to you if these are pro’s or cons. And the maturity of the product of course.


With the growing number of organisations adopting a multi-hypervisor environment, these organisation still rely on multiple management interfaces/applications and thus bringing extra complexity to management of the virtual environments. Complexity adds extra time and extra costs, and that isn’t what the large portion of the organisations want. At this time, simply don’t expect a true single management experience if you bring in different hypervisors or be prepared to close the gaps yourself (the community can be of great help here) or use third party products like Hotlink.
We are getting closer with the adoption of open standards, hybrid clouds and a growing support of multiple hypervisors in the management suites of the hypervisor players. But a step at a time. Let’s see when we are there, at the true single management of multi-hypervisor environments.

Interested about telling your opinion, have a idea or party I missed? Leave a comment. I’m always interested in the view of the community.

– Happy (or happily) managing your environment!