Keep on IO’in in the VDI world

I have already done some posting about analyzers, optimization and IO offloading (https://pascalswereld.nl/post/72323420934/ram-based-storage-vdi and https://pascalswereld.nl/post/71309552232/reducing-io-at-host), but a question that rises often in VDI projects is what IO(PS) do we expect in what layers of the infrastructure and how can we measure them accordantly?

But first start at the start.

What kind of IO can we expect in VDI?

IO in the case of VDI, and other virtualization workloads, are mainly in the networking and storage. Networking IO is measured in the throughput of the network components and in the time a command takes in traveling through the components (up and downstream) measured in latency.

Storage IO is measured in the amount of operations (input output per second, IOPS), characteristics of the IO (sequential, random, read/write ratio), the time a command takes in traveling through the components (latency) and the throughput of the storage components.

Why is IO important in VDI?

Resources and IO these resources need is key in delivering a good desktop experience. It does not matter that the desktop is a physical or virtual desktop, it is a matter of ensuring that the desktop has sufficient hardware resources (the four horsemen CPU, memory, networking and storage) to run the OS and applications at the right time. Don’t have enough or slower then needed, the desktop experience will suffer immediately as will the users.
With physical desktops each of the resources are local to and dedicated for that desktop, with modern systems this is often not a problem, but here some of your user experience requirements are started. Most of the users are used to working with physical desktop environments and know what to expect in experience of those systems. They want the same or faster in VDI. With resources in VDI, there will be abstracting and pooling of those (hardware) resources. The resources are presented and pooled to the hypervisor host. For example the virtual hard drive is available on the hypervisor local or centralized storage. This storage is shared (or replicated) between hypervisor hosts. IO requirements and consequences of the IO load is spread across the virtual and facilitating resource (storage and network) infrastructures. In order to deliver a consistently high performance, virtual desktops require a constant access to preferably low latency and high throughput resource infrastructures.

What is normal IO in the VDI?

Normal? There is no normal. Every organization will have it’s own workload footprint, this is the case in server virtualization and this is also the case for virtual desktops. A Windows image will have a certain IO footprint that is more or less known out there (in averages), but depending on your build, sizing, own components and usage these differ. The IO footprint of a virtual desktop will be influenced by choices in your deployment, for example the optimization of your image, persistent or stateless, offloading of certain applications (like virus protection to the hypervisor), using application virtualization, optimized in IO characteristics (small random IO to large sequential IO) and de-duplication/compression throughout the infrastructure. With certain techniques you can get your IO down to and handled on the host.

How will you be able the know your specific IO? Well measure, test, baseline, report (or document) and repeat. Preferably start when planning (pilot or poc) for a VDI.
Sure, you can use one of the available calculators out there as a reference point (please do), but you still have to do some of the magic yourself.

How can we measure the IO of the VDI?

It is important here to know what kind of components are in your infrastructure and how to measure the metrics of these components.

Start from within your guest as there the user will feel it’s pain, oh wait….will get his warp speed desktop experience.

Tools as a complete solution:

  • vCenter Operations (VCOPS) and for VDI specifically, VCOPS for Horizon view. Complements the vCenter / ESXi monitoring and metrics already in place with excellent views and reporting states of your environment.
  • EdgeSight and Desktop Director for XenDesktop VDI.
  • XenServer monitoring, depending on your version you will possibly need to add a Performance Monitoring Enhancement pack.
  • Planning tools like the VMware View Planner (https://pascalswereld.nl/post/66369941380/vmware-view-planner), or analyzer tools like the VMware fling vBenchmark (https://pascalswereld.nl/post/62991166022/flings-vbenchmark) / IO Analyzer (https://pascalswereld.nl/post/58225706990/vmware-io-analyzer-fling) or for Citrix the XenServer PerformanceVM.
  • Login VSI an industry standard load testing tool for virtualized desktop environments. Login VSI can be used to test the performance and scalability of every Windows based virtual desktop solution. Can be used for VDI Horizon View, XenDesktop as well as heterogeneous solutions where SBC or RDS are in place. ( there is going to be a blog post on this subject a little later on).
  • Nagios / Cacti and the like. You will have to add your counters and checks to the configuration, which can be a little gruesome if you don’t have any experience. But these are excellent products to have your whole infrastructure monitored with all sorts of counters and have a chains with parents and child objects in there.

Tools at the layers:

This really depends on the solution in place. Know your infrastructure and how traffic flows is key here. The monitoring needs to be planned so that you get your counters in the same time window and with the same metrics. Get the disk and network counters from the guest OS itself thoughout the infrastructure. Keep an eye on the CPU and memory on all layers, and check for paging/swapping. Averages on layer x and downstream in real time can be a bother. Also like written in a previous post, time (and time zone) needs to be set correctly and synchronized with the same source.

But to give some pointers

  • Perfmon / Resource monitor.
  • iostat/vmstat/top and other Linux tools.
  • Hypervisor monitoring, like from vSphere webclient /vCenter/ESXi host or XenCenter.
  • Esxtop / XenTop / vscsistats. See some standards in my post (https://pascalswereld.nl/post/69976450322/vphere-monitoring-metrics)
  • Networking SNMP tools, or the own supplied tools.
  • Storage SNMP tools, or the own supplied tools. When you have switches and controllers in your storage infrastructure monitor them all the way.
  • Benchmarking tools like IOMeter, IOzone, Passmark or specific to a kind of workload (be ware, there are a lot out there).

– Happy IO’in in the VDI world (and cue Neil Young)

Virtual Infrastructure WAN Benchmarking with Dummynet

Ever had a customer environment where either a deployment is done over WAN links or the customers wants to change there WAN links, and they (or you in planning or testing phase) are interesting to see what happens in advance? Or you just want to know how latency or ping drops in a WAN link influences user experience in VDI or application deployments (well if the users aren’t complaining first ;-)…)?

Introduce some real world networking examples to your lab set-up?

You will have some opportunities in the virtual infrastructure, vSphere for example let’s you throttle a vSwitch/portgroup to a certain peak bandwidth (traffic shaping). But bandwidth is just a part of the deal, what about what is latency doing to your desktop or application experience, what will ping drops do to the communication?

For these sort of testing/benchmarking I often use a Dummynet setup to influence the traffic between infrastructure components. So how does this work? Time to find out in this blog post.

What is Dummynet?

Start at the start you dummy…..Dummynet is a traffic shaper, bandwidth manager and delay emulator and is included in FreeBSD. There are several ways to use this, deploying a VM with a FreeBSD image or with some of the live ISO’s out there (Frenzy for example http://frenzy.org.ua/en/index.shtml ). Dummynet was initially implemented as a testing tool for TCP congestion control by Luigi Rizzo <luigi@iet.unipi.it>. See Luigi’s site http://info.iet.unipi.it/~luigi/ip_dummynet/. Dummynet using ipfw to influence traffic flowing through the created network tunnel, 

How to use Dummynet in your virtual infrastructure?

Build a Dummynet VM with two networks connected. Connect a client VM and a server VM on one of the networks (client in network X and server in network Y). Let the dummynet route and pipe the network traffic between those VM and shape the network accordantly to your testing needs.

A model to help clarify.

image

When installing FreeBSD be sure to include src for the ability to include Dummynet firewall in the kernel. You can either use a kernel load or recompile a kernel with the dummynet option included.

Demo

I’m putting a client VM on the same network as one of the dummynet interfaces. The second dummynet interface is connected to an other VM network where the server VM is connected. The VM’s are given IP addresses in their respective IP subnets and the dummynet VM is given IP in these as well (on the em0 for the client VM subnet en on the em1 for the server VM subnet). I configure the client and server VM’s to use the Dummynet VM as their IP gateway (just a route add for there subnet pointing to the dummynet interface).

On the dummynet VM you can use the following commands or included them in the rc.conf and /etc/sysctl.conf files.

ifconfig em0 192.168.243.241 netmask 255.255.255.0
ifconfig em1 10.0.1.1 netmask 255.255.255.0
sysctl net.inet.ip.forwarding=1 (Tell FreeBSD to forward packets between the two IP addresses)

Check if your can reach the both VM’s by pinging there IP adresses from the dummynet host.

image

Yes? Okay move on. Check if you can reach from one to the other VM via the ip forwarding option. The freebsd-server is on the 10.0.1.0 subnet. So first add the route like said before.

image

This works. Now introduce some dummynet.

kldload dummynet
ipfw flush

Add a firewall rule to allow all traffic from and to the first vm to the second.

ipfw add 1000 allow all from any to any

Add a Dummynet pipe to check if ipfw works:
ipfw add 100 pipe 1 ip from any to any

And put some delay on that:

ipfw pipe 1 config delay 10ms

And check with ping:

image

You will see the delay multiplied by how many times to traffic passes the pipe. From 2.5 / 3ms in previous shot to 39/38 ms in current shot.

When this works you can add some other test for example:

Add a rule to delay selected packets by 50 ms, randomly drop 30% (0 is for no loss and 1 is for 100% packet loss) of the packets and limit the bandwidth to 1Mbps (bandwidth can be checked by a copy or such.)

ipfw pipe 1 config delay 50ms plr .03 bw 1024Kbits/s

image

Sequence 3 is dropped from the packets.

To see what is configured on the pipe use:

ipfw pipe 1 show and destroy with ipfw pipe 1 delete.

This concludes this blog post.

– Happy Dummynet network testing!

Benchmarking your VMware Horzion VDI infrastructure with VMware View Planner

So I got myself a VDI infrastructure based on VMware Horizon view. Planning or Design phase done, first implementing phase done. Depending on your why’s you are either in concept planning phase or testing before going to production. Good let’s introduce the next step in the project phases, planning for workload or testing the infrastructure for the right workload. But how do I measure if my architecture can be, or is, up to the right numbers….
For vSphere server testing we can use the IO analyzer fling or specific service/applications tools like sqlio, iometer, vscsistats, iozone, Citrix Edgesight for load testing (well till the end of the year as Citrix EOL is per 31 december 2013 ), website testing with Apache JMeter and so on. But VDI needs its own benchmarking.

For VMware Horizon View VDI infrastructure we have VMware View Planner as our designated tool for planning or benchmarking.

So what is VMware View Planner?

VMware View Planner is a tool designed to simulate a large-scale deployment of virtualized desktop systems. This is achieved by generating a workload to the infrastructure that is representative of (many) user-operations, from user or administrative point of view. Selection can be made from user actions that typically take place in a VDI environment or customs workloads can be added.

The VMware Planner can be downloaded as a virtual appliance (OVF template) from http://www.vmware.com/products/desktop_virtualization/view-planner/overview.html. There you will also find some documentation to get your environment up and running.

What testing can be done?

The test can be summarized in three categories:

  1. Workload generation; By configuring View Planner to simulate the desired number of users and configured applications, View Planner can accurately represent the load presented in a given VMware VDI deployment. Once the workload is running, resource usage can be measured at the servers, network, and storage infrastructure to determine if bottlenecks exist.
  2. Architectural comparisons; To determine the impact of a particular component of VDI architecture, you can configure a fixed load using View Planner and measure the latencies of administrative operations (provisioning, powering on virtual machines, and so on) and user operations (steady-state workload execution). Changing the component and measuring latencies again provides  a comparison of the different options. A note, you will have to do some comparison of other architecture components (eg. hosts, storage, networking) configuration decisions to the View Planner workload to completely have a view of the impact.
  3. Scalability testing; The system load is increased gradually until a selected resource is experiencing contention or is exhausted. Resources measured include CPU, memory, and storage bandwidth.

There is more, as VDI deployments are organization specific (at least organizations have specific application landscapes) VMware View Planner allows for Custom Applications to be added.

View of predefined workloads:

image

So what is need, VMware View Planner Architecture.

Basically you need a VMware Horizion view environment where you deploy the VMware View Planner appliance.

I have taken the following model from the VMware website

image

  • VMware View Planner Virtual Appliance. A Linux-based appliance virtual machine. This virtual appliance interacts. It runs a web server to present a web interface. The View Planner appliance interacts with a VMware vCenter View connection server or Virtual Center server to control desktop virtual machines. It also communicates with client virtual machines to initiate remote protocol connections.
  • Harness. Part of the appliance, but such an important piece it needs to be mentioned on it’s own. The central piece in the View Planner architecture. The harness controls everything, from the management of participating desktop virtual machines (which can be scaled in thousands, up to 4000), to starting the defined or selected workloads, and collecting results back from these workloads. Results are storage in a database in the appliance. The harness provides monitoring the state through a web user interface.
  • Web Interface; The graphic user interface to interact to the VMware View Planner appliance to setup the environment, set up the workloads, controlling the workloads and viewing the results.
  • Workload; A predefined set of actions that can be custom build for your organization. The workloads can be sequenced in any way desired. Workloads typically have two possible categories to operate in: user and admin operations. User operations can include typing documents, browsing the web, reading or printing PDF documents, checking email, etc, etc. Admin operations can include provisioning virtual machines, cloning operations, powering servers on and off, etc. etc. etc.
    Workloads are placed in sequences to better simulate a real user environment. An example of user workload sequence steps: starting with Excel (Open, compute, save, close, minimize, maximize, enter value), followed by Zip (Compress) and let’s play a Video (Open, Play, Close).

If your not going to stick to only local mode workloads, you will need interaction with Active Directory, vCenter, ESXi hosts and View Connection Server.

Test Driving

In my VMware Workstation 10 lab environment I have a Horizon View 5.2 environment with vSphere 5.1 hosts, and 5.1 Windows vCenter Server system. vCenter is running on Windows 2012 with a SQL Express DB. View components are running on Windows 2008 R2 server and also using the SQL Express instance. Not much is to expect from this environment but to walk through the deployment steps of VMware View Planner.

What will you need for resources? The appliance is configured with 1vCPU, 3GB vRAM and a circa 15GB for disk space. Can fit in any environment easily.

In this virtual infrastructure I add the VMware View Planner Appliance downloaded from the VMware web site. Deploying the OVF is straightforward. Just accept the agreement, select location and networking. Add the appliance to the current network and let it rip.

Connect to the console to setup use the provided scripts (see installation manual). The console can be accessed by the root vmware combo.

Change to the /root/ViewPlanner directory. Set the path for Python by running the command: source setup.sh
Configure the virtual machine’s static IP address and the corresponding settings by running the command: python ./harness_setup.pyc -i <ipaddr> -m <netmask> -g <gateway> -d <full-domainname> -n <dnsip1> [,<dnsip2>, …]

Replacing <ipaddr>, <netmask>, <gateway>, <full-domainname>, <dnsip1>, and (optionally) additional DNS addresses with appropriate values, including the static IP address and the fully-qualified domain name.

Fire up the view planner web interface (http://your fqdn you configured in the python harness setup)

image

and logon via user root and password: abc123.

image

Next setup view planner to your vCenter and AD DS.

Go back to Run & Reports to start setting up your profiles. First you will have to set some up, and when you have some saved you can return later and load them when you want to rerun them.

image

Conclusion

For VDI based on VMware products this is the tool to use in your planning phase to determine if your infrastructure is up to the challenge and can be easily scaled up when necessary. How are some architectural decisions going to influence your environment now and in the future, well find out with VMware View Planner.
You can also use it to test/benchmark current infrastructures or just implemented infrastructures, and test what they are capable of. With existing and operational systems (and concepts that are using shared components) stress testing influences your environment. So be careful of the consequences of your action. Main focus of this product is in the name: Plan!

As with projects, same goes with the planner, start small and scale when needed. Get the hang of it and try your use case scenario’s and see what results you can get before going out in an all war scenario (just fireup all sequences for 4K virtual desktops sergeant!).

– Enjoy your VDI VMware Horizon View planning!

VMware vBenchmark – Lab Flings

Last month I did a blog post of a fling I often use in project phases (read it at https://pascalswereld.nl/post/58225706990/vmware-io-analyzer-fling). I want to blog about another fling you can use in a project and also in normal management of virtual infrastructures.

This fling is called vBenchmark and can create reports of your virtual environment with measurements of the performance of your (just to state the obvious; VMware) virtualized infrastructure. These measurements can be used to report to IT management about the benefits brought with a implementation or migration project. You can also use this in a assessment phase to baseline report your existing environment prior to making (to be beneficial) changes.

The report is targeted at IT as measurement metrics are mostly technical of art, but hey that’s what a virtualization infrastructure is made off.

Measure metrics are categorized as follows:
– Configuration: for example, how much virtual vs physical RAM is registered.

image

– Efficiency: for example, how is you environment utilized.

image

– Agility: for example, how much time do you take on average to provision a VM, or how productive can your administrators be (do not get overexcited, this is only from the infrastructure prospective ;-) ).

image

– Quality of Service: for example, how much downtime do you avoid by using availability features, or how much downtime have you experienced.

image

The environment metrics can be uploaded to a VMware server to compare (based on your license type and organisation) your metrics with the outside worlds matched peers. Your metrics will be anonymously uploaded.

You can add one or more vCenters to vBenchmark. Be sure to save your session data or you will have to do them again in vBenchmark. But you won’t have a reference data set..

vBenchmark is a OVF appliance that can be downloaded from this link: http://labs.vmware.com/flings/vbenchmark

Test Driving

I have downloaded vBenchmark and I will deploy vBenchmark to my test Lab. My lab environment are 3 ESXi 5.5 hosts managed by a vCenter 5.5 server appliance. There are around 4 VM’s operational so not quite the environment, but large enough to get some metrics in.

The deployment is straightforward, deploy as OVF. Add name, location, storage (deploying on VSAN) and add networks. From the OVF template the IP is defined as static, but there is no configuration.

image

vBenchmark actually supports IPv4 DHCP (and you can change the network settings via the appliance admin app at https://<vbenchmark&gt;:5480/), and that’s what I’m using. Finish and power on.

After power on go to a console and set the root password. You will have to do this else you can’t access the web interface.

image

Open up your favorite browser and go to the system. In the startscreen you will have to add a vCenter server to vBenchmark.

image

After that you can select a statistics period and start adding statics.

image

When finished collecting you can select to include your cluster in the reports. And you are redirected to your dashboard.

image

From here on you can create vCenter server groups, share (upload) your metrics to included peer metrics and save your configuration for future references.

-Enjoy vBenchmarking!

VMware IO Analyzer – lab Flings

Flings in VMware labs is a great place for (very) useful tools or applications. This time I want to blog about a fling I often use in a test phase for implementation projects or in health assessments, see what synthetic load an environment can handle and if your vSphere design is up to the right io charactics and capacity.

Important in these kinds of test is your test methodology and plan: Assess, Filter test, plan, collect, analyse and report. With several of these steps IO Analyzer can be the player.

IO analyzer can configure, schedule and run several IOmeter workloads or replay vSCSI traces.

Download at:

http://labs.vmware.com/flings/io-analyzer

Import ovf to your environment. Start with more then one, so you have some dedicated workers thoughout your environment.

After deployment change the second vmdk for the defaulted “small” configuration to approx 4GB plus (and Thick Eager). Why? Because the small amount of disk is used as disk test and fits in most storage cache. We need to get out of that and hit some real scenario’s.

One (or yes two) more things, logon to the consoles of all the appliances. Open a console, choose first option or press enter and login with root and password vmware. *ssst a very secret vmware user*. An other usage of the console is checking or monitoring the IOmeter tests i the console when they are running.

Demo:

Type down one of the ip’s or hostnames of the appliances, and will use that one as the controller.

Open a browser (chrome or firefox) and type http:// and you will reach IO analyzer in your environment.

There we have the following options

image

For this I will use the workload configuration to add two tests to two appliances and check the results. Test scheduler is not used, will run immediately.

In this screen we first add the hosts where our test machines are, use the root password to connect to the hosts. When a connection is established the VM’s on that host are visible in the Add Workload Entry. Here you can find all kinds of IOmeter tests.

image

I have created two workloads, one Exchange 2007 on our first appliance and SQL 64K blocks on the other appliance. The duration is changed from the default 120 seconds to a 5 minute (300 seconds) schedule. This configuration is saved as Demo config.

Click on Run Now to let the test run. After a initialization you can see the progress in the console of one of appliances.

image

After completion of the tests you can view the test results in View Test Results (so it is not just a clever name :)).

Here you can check the two tests and the different VM and host metrics saved from IOmeter and esxtop (if there are any, you will get as a bonus the metrics of other VM’s on the hosts). More detailed information about these metrics, see the following URL: https://communities.vmware.com/docs/DOC-9279. Duncan Epping also has a good article about esxtop metrics and more. Go see his site when waiting for the test to finish: http://www.yellow-bricks.com/esxtop/

Here see the results of our Demo tests (I’m not going over in detail in this post).

image

Enjoy stress testing.