vCenter Server Appliance 6.0 in VMware Workstation

For demo’s, presentations, breaking environments or just killing time I have a portable testlab on my notebook. Yes I know there are also options for permanent labs, hosted labs and Hands-on labs for these same purposes. Great places for sure, but that is not really what I wanted to discuss here.

As I am break.. ehhh rebuilding my lab to vSphere 6.0 I wanted to install VCSA 6.0 in VMware workstation. Nice, import a my vmware downloaded VCSA-versionsomething.ova (after e-mail address number ####### registered over there) and we are done! …….. Well not quite.
First the vCenter download contains the OVA, but it is a little bit hidden. The guided installer will not help you here. You will need to mount or extract the downloaded ISO and look for vmware-vcsa in the vcsa/ folder.

VCSA Location
Copy the vmware-vcsa file to a writable location (when just mounted the ISO) and rename vmware-vcsa to vmware-vcsa.ova. And now we can import the ova to VMware Workstation. When the import finishes, do not start the VM yet. Certain values that are normally inserted via the vSphere Client or ovftool are to be appended to the VMX file of the imported VCSA. Open the vmx in the location where you let Workstation import the VM. Append the following lines:

guestinfo.cis.appliance.net.addr.family = “ipv4”
guestinfo.cis.appliance.net.mode = “static”
guestinfo.cis.appliance.net.addr = “10.0.0.11”
guestinfo.cis.appliance.net.prefix = “8”
guestinfo.cis.appliance.net.gateway = “10.0.0.1”
guestinfo.cis.appliance.net.dns.servers = “10.0.0.1”
guestinfo.cis.vmdir.password = “vmware-notsecurepassexample”
guestinfo.cis.appliance.root.passwd = “vmware-notsecurepassexample”

Note: Change the net and vmdir/appliance.password options to the appropriate values for your environment.

If not appended when you start the VCSA an error: vmdir.password not set aborting installation is shown on the console (next to root password not set) and network connection will be dropped even if you configure these on the VCSA console (via F2).

Save the VMX file.

And now it is time to let it rip. Start up your engines. And be patience until…lo and behold:

VCSA Running in workstation

And to check if the networking is accepting connections from a server in the same network segment open up the VCSA url in Chrome for example. After accepting the self signed certificate unsecure site (run away!) message you will (hopefully) see:

VCSA in Workstation

Next we can logon to the Web client (click and accept the unsecure connection/certificate) and logon via Administrator@VSPHERE.LOCAL and the password provided in the VMX (in the above example vmware-notsecurepassexample). As a bonus you now know where to look when you forget your lab VCSA password ;-).

VCSA 6 Web Client

(And now I notice the vCenter Operations Manager icon in the Web Client Home screen. Why is this not updated like vRealize Orchestrator :-) )

-Enjoy!

 

 

VMware Utility Belt must have tools – RVTools 3.7 released

March 2015 RVTools version 3.7 is released. 

This, in my opinion, is the tool each VMware consultant must have in his VMware utility belt together with the other standard presented tools. At this time RVTool is still free, so budget is no constrain to use this tool. More important it’s lightweight, very simple in usage and shows much wanted information in a ordered overview or allows for exporting the information in Excel format to analyse this offline. 

Before using this tool, it is important to understand the tool is used to make a point in time snapshot of the infrastructure configuration items in place. In short what is configured and what is the current operational state. No more, no less. The information can then be used in for example operational health checks or AS IS starting point in projects (consolidation or refresh projects) in the analysis/inventory phase. See more use cases further below, and I am sure there can be some more examples out there.

No trending or what if’s for example, that is something you will have to do yourself or use other solutions/tools available for the software defined data center. VMware has some other excellent tools for SDDC management and insights in your virtual environment (for example vRealize Operations and Infrastructure Navigator). But that is a complete other story.

What is RVTools?

RVTools is a Windows .NET application which used the VI SDK (which is updated to 5.5 in this release) to display information about your VMware infrastructure.
A inventory connection can be made to vCenter or a single host, to get as is information about hosts, VM’s, VM Tools information, Data stores, Clusters, networking, CPU, health and more. This information is displayed in a tabpage view. Each tab represents a specific type of information, for example hosts or datastores.

RVTools can currently interact with Virtual Center 2.5, ESX Server 3.5, ESX Server 3i, Virtual Center 4.x, ESX(i) Server 4.x, Virtual Center 5.0, Virtual Center Appliance, ESXi Server 5.0, Virtual Center 5.1, ESXi Server 5.1, Virtua lCenter 5.5, ESXi Server 5.5 (no official 6.0 in this version).

RVTools can export the inventory to Excel and CSV for further analysis. The same tab from the GUI will be visible in Excel.

image

image

There is also a command line option to have (for example) a inventory schedule and let the results be send via e-mail to an administrative address.

Use Cases?

– On site Assessment / Analysis; Get a simple and fast overview of a VMware infrastructure. The presented information is easy to browse through, where in the vSphere Web Client you would go clicking through screens. When there is something interesting in the presented data you can go deeper with the standard vSphere and ESXi tools. Perfect for fast analysis and health checks.

– Off site Assessment / Analysis; Get the information and save the Excel or CSV dump to get a fast overview and dump for later analysis. You will have the complete dump (a point in time reference that is) which you can easily browse through when writing up an analysis/health check report.

– Documentation; The dumped information can be used on or offline to write up documentation. Excel tabs are easily copied in to the documentation.

– (Administrator) reporting; Via the command tool get a daily overview of your VMware infrastructure. Compare your status of today with the point in time overview of the day before or last week (depending on your schedule and/or retention). Use this information in the daily tasks of adding/changing documentation, analysis, reporting and such.

Release 3.7 Notes

For version 3.7 the following has been added:

  • VI SDK reference changed from 5.0 to 5.5
  • Extended the timeout value from 10 to 20 minutes for realy big enviroments
  • New field VM Folder on vCPU, vMemory, vDisk, vPartition, vNetwork, vFloppy, vCD, vSnapshot and vTools tabpages
  • On vDisk tabpage new Storage IO Allocation Information
  • On vHost tabpage new fields: service tag (serial #) and OEM specific string
  • On vNic tabpage new field: Name of (distributed) virtual switch
  • On vMultipath tabpage added multipath info for path 5, 6, 7 and 8
  • On vHealth tabpage new health check: Multipath operational state
  • On vHealth tabpage new health check: Virtual machine consolidation needed check
  • On vInfo tabpage new fields: boot options, firmware and Scheduled Hardware Upgrade Info
  • On statusbar last refresh date time stamp
  • On vhealth tabpage: Search datastore errors are now visible as health messages
  • You can now export the csv files separately from the command line interface (just like the xls export)
  • You can now set a auto refresh data interval in the preferences dialog box
  • All datetime columns are now formatted as yyyy/mm/dd hh:mm:ss
  • The export dir / filenames now have a formated datetime stamp yyyy-mm-dd_hh:mm:ss
  • Bug fix: on dvPort tabpage not all networks are displayed
  • Overall improved debug information

Who?

RVTools is written by Rob de Veij aka Robware. You can find Rob on twitter (@rvtools) and via his website http://robware.net.
Big thank to Rob for unleashing yet another version of this great tool!

As the tool is currently free please donate if you find the application useful to help and support Rob in further developing and maintaining RVTools.

vSphere: Working with traffic filtering in the vNetwork Distributed Switch

Introduction

Within a physical and virtual infrastructure there are several options to limit the inbound and outbound traffic from and to a network node, part of the network or entire network (security zone). A limit can be, filtering (allow or dropping certain traffic) or the prioritization of traffic (QoS / DSCP tagging of the data) where a defined type of traffic is limited versus a kind of traffic with a higher prioritization.

Options include filtering with ACL, tagging and handling sort of traffic with QoS / DSCP devices, firewalling (physical or virtual appliances), physical or logical separation or Private VLAN’s (PVLAN for short). Furthermore, an often overlooked component, keep all your layers in view when designing the required security. If required to filter traffic from a specific data source to a specific group of hosts where the requirement is that those VM’s are not allowed to see or influence the other hosts, traffic filters setup on the physical network layer will not always be able to “see” the traffic as for example blade servers in certain blade chassis can access the same trunked switch ports / VLAN, or VM’s with same portgroup / VLAN are able to connect to each other’s network as the traffic is not reaching or redirected to the physical network infrastructure where these filters are in place. That is when not using a local firewall on the OS. You could say this is bad designing, but I have seen these described “flaws” pop up a little too often.

 Options in the VMware virtual infrastructure

You have to option to use third party virtual appliances as firewalls, vCloud suite components or network virtualization via NSX (SDN) for example. Not always implemented due to constraints overheard around, like: overhead of the handled traffic by the virtual firewall (sizing), single point of failure when just using one appliance, added complexity for certain IT Ops where networking and virtualization are strict separated (Bad bad bad) or just no budget/intention to implement a solution that goes further than just the host virtualization the organization is at (as they probably just started). These are just a few, not all are valid in my opinion….

From vSphere 5.5 there is another unused option (mostly unknown); use the traffic filtering and tagging engine in the vNetwork Distributed Switch (vDS or dvSwitch). That is when you have an Enterprise Plus edition, but hey without this a vDS is not available in the first place. Traffic filtering is introduced in version 5.5 and therefore can only be implemented on vSphere 5.5+ members of the 5.5+ version of vDS. This vDS option is the one I want to show you in this blog post.

Traffic filters, or ACL, control which network traffic is allowed to enter or return (ingress and/or egress rules) from a VM, a group of VM’s or network via the port group, or a uplink (vmnic). The filters are configured at the Uplink or port group, and allow for an unlimited number of rules to be set at this level. These handle the traffic from VM to the portgroup and/or the traffic from portgroup to the physical uplink port, and vice versa. The rules are processed in the VMkernel, this is fast processed and there is no external appliance needed. With outgoing traffic rule processing happens before the traffic leaves the vSphere host, which also possibly will save on the ACL on the physical layer and networking traffic when only types of traffic or to a specific destination are allowed.

With the traffic filter we have the option to set rules based allow drop (for ACL) on the following Qualifiers:

vDS - image1

The tag action allows setting the traffic tags. For this example we don’t use the tag action.

System Traffic are the vSphere traffic types you will likely see around, where we can allow a certain type of traffic to a specific network. MAC let’s us filter on layer 2, and specific source and/or destination MAC addresses or VLAN ID’s. IP let’s us filter on Layer 3 for the IP traffic types TCP/UDP/ICMP traffic for IPv4 and IPv6.

The following System traffic type are predefined:

vDS - image2

Make it so, number One

I will demonstrate the filtering option by creating a vDS and adding a ESXi host and VM to this configuration. Just a simple one to get the concept.

My testlab vDS is setup with a VM like this screenshot:

vDS - image3

I got a DSwitch-Testlab vD switch with a dvPortgroup VM-DvS (tsk tsk I made a typo and therefore not consistent with cases, please don’t follow this example ;-)). A VM Windows Server 2012 – SRDS is connected to this portgroup.

 The VM details are as follow:

vDS - image4

The IP address 192.168.243.165 we will be looking at.

A the VM-DvS and going to the manage tab, we can choose Policies. When we push the edit button we can add or change the traffic filtering (just look for the clever name).

vDS - image5vDS - image6

As you see I already have created an IP ICMP rule which action currently says something completely the opposite as the rule name. This is on purpose to show the effect when I change this action. When I ping the VM from a network outside of the ESXi host, I get a nice ICMP response:

vDS - image7

When we change the ICMP rule to drop action, we get the following response:

vDS - image8

 

That’s what we want from the action. Other protocols are still available as there are no other rules yet, I can open an RDP to this Windows Server.

vDS - image9

When wanting to allow certain traffic and others not you will have to create several rules. The applied network traffic rules are in a strict order (which you can order). If a packet already satisfies a rule, the packet might not be passed to the next rule in the policy. This concept does not differ from filtering on most physical network devices. Document and draw out your rules and traffic flows carefully else implementation/troubleshooting will be a pain in the $$.

This concludes my simple demonstration.

 – Enjoy!

Sources: vmware.com

Hey come out and play. Join the vSphere Beta Program

At June the 30th VMware announced the launch of the latest vSphere Beta Program. This program is now open for anyone to register and participate in the beta program. The Beta program used to be for just a select group, but with VSAN Beta VMware started to allow a wider group of participants. In my opinion this is good as the group of software testing participants is larger and the amount of feedback, learned lessons and experience will be greater. The community and it’s testers will make the product and it’s features even better. Hopefully more Beta’s will also be open to a larger group of participants.

A larger group will make it harder to keep information within the group and not publicly shared on for example the big bad Intarweb. As this Beta program seems open to everyone, it is still bound to NDA rules. Details are in the VMware Master Software Beta Test Agreement (MSBTA) and the program rules which you are required to accept before joining. After that share your comments, feedback and information in the private community that is offered with the program.

What can you expect from the vSphere Beta program?

When you register with a my VMware account, the participant can expect to download, install, and test vSphere Beta software for his or hers environment. The vSphere Beta Program has no established end date. VMware strongly encourages participation and feedback in the first 4-6 weeks of the program (starting on June 30 2014). What are you waiting for? 

Some of the reasons to participate in the vSphere Beta Program are:

  • Receive access to the vSphere Beta products
  • Gain knowledge of and visibility into product roadmap before others.
  • Interact with the VMware vSphere Beta team, a chance to interact with engineers and such.
  • Provide direct input on product functionality, configuration, usability, and performance.
  • Provide feedback influencing future products, training, documentation, and services
  • Collaborate with other participants, learn about their use cases, and share advice and learned lessons of your own.

What is expected from the participants?

Provide VMware with valuable insight into how you use vSphere in real-world conditions and with real-world test cases, enabling VMware to better align the products with business needs.

Where?

Sign up and join the vSphere Beta Program today at: https://communities.vmware.com/community/vmtn/vsphere-beta.

– Go ahead, come out and play. Join the vSphere Beta program now!

Source: vmware.com

vSphere Auto deploy TFTP on Citrix PVS

I occasionally do a deployment of Citrix on vSphere. When using Citrix a returning component is Provisioning Services (PVS) for automated deployment of Citrix session servers or VD images. When using a management cluster (to prioritize and separate control layer or infrastructure components) an option is to use auto deploy for the session images hypervisor hosts (depending on the edition, depending on the cluster requirements).
One of the components to use in auto deploy is TFTP for the boot image. This can be leveraged for PVS as well, as PVS also needs a type of boot image to start up the VM’s. With PVS this is also a TFTP service (or boot image, but we leave this out for this posting) that is installed at the PVS server.

Why not use this TFTP service as vSphere Auto Deploy does not include one?

What is Auto Deploy?

vSphere Auto Deploy facilitates a infrastructure for automatic server provisioning and network deployment (streaming) of the ESXi hypervisor image. It uses a central managed image, administrators just need to manage the central image and the host profiles. The deployment can be on local storage, state full on HDD, SD or USB or stateless to the hosts ram. It works in conjunction with:

– vCenter,
– host profiles,
– TFTP server,
– Auto Deploy server and Image Builder,
– a PXE boot infrastructure with a DHCP service.

These services can be installed on the vCenter host or hosted/integrated on specific services. When using the stateless host option be sure to have a high available Auto Deploy infrastructure. For example load balancing the TFTP and auto deploy services.

Auto Deploy and host profiles are available from the Enterprise plus Edition. Okay, not always will a virtual desktop cluster be set up with Enterprise Plus, but we can also do this in the management cluster as long as we prioritize the right machines.

What is PVS?

Provisioning Services infrastructure is also based on software-streaming technology. This technology allows computers to be provisioned and re-provisioned in real-time from a single shared-disk image. In doing so, administrators can completely eliminate the need to manage and patch individual systems. Instead, all image management is done on the master image.

PVS works with:

– PVS vDisk store for the master image,
– PVS Console for setting up farm, device collections and managing updates and assignments,
– PVS Streaming service,
– PVS Citrix Shared components, such as MSSQL data store, License server,
– PVS Network services DHCP, PXE and TFTP.

Boot image

A boot image or boot strap file is a small kernel used to start up the machine, connect to the network and receive it’s image via network. The boot file must be configured so that it contains the information needed to communicate with the streaming services, Auto Deploy or PVS.

Putting it all together

As the list above we have some components that are used on both the infrastructures, DHCP, TFTP and PXE. The DHCP is used to provide a bootp image server and image name to the PXE clients (options 66 to the PXE/TFTP on the PVS server and 67 for the boot image name). The PXE infrastructure is a networking zone where client connect to. These clients are configured to boot from network (change the VM bios boot order for example). This can be a separate network, a logical separated network (vLAN) or just a one in all network (policies are advisable to guarantee some networking bandwidth to different types) depending on the requirements of the organization. We will be using a DHCP scope already set up on Microsoft DHCP and TFTP service on the PVS server (why set up more than one).

We can setup the DHCP options on scopes (if logical separate networks) or on leases (DHCP reservations for the smallest amount of host. If we have less ESXi hosts than streaming VM’s we put a scope option to the common image and use DHCP reservations to the other image). We will use to same TFTP service (66), but depending on the sort of machine (VM or ESXi host) we use a different boot image name.

First we get the Auto deploy TFTP image from the vCenter service and put it in the PVS image locations. We connect to the vCenter services via the Web client and browse to vCenter, vCenter server name, manage and auto deploy. Here we have the option to download the TFTP boot zip.

image

The boot zip is a collection of vSphere boot straps with the correct IP of your auto deploy service (it is created on installation).

image

Next we connect to the PVS server and go to the PVS TFTP image location. This is in C:ProgramDataCitrixProvisioning ServicesTFTPboot.

image

Here we place the Auto Deploy images from the boot zip.

image

Next up we change the DHCP options accordant, I used the DHCP scope options for the PVS image and used a reservation for the ESXi hosts (as these are just three hosts). Citrix is using the ARDBP32.bin image from the 10.0.0.100 lab PVS server.

image

And VMware is setup to use the undionly.kpxe.vmw-hardwired image from the same 10.0.0.100 lab PVS server.

image

Let’s boot the machines up. First the ESXi host. It receives the correct VMware image and starts to boot the base image from auto deploy.

image

Next up start a VM to use the PVS stream image:

imageIt boot’s up to connect to the PVS server. Unfortunately I apparently forgot to add an entry for this device (auto create is off as well), no vDisk in this example. But if we add a vDisk this will load as well.

image

Alternatively we can setup a TFTP service in on a other host if we want to separate this service from the PVS service and do the same for the Citrix boot image. Just follow the same procedure and add the Citrix images as well.

High Available

Standard the TFTP service is not high available, and when using multiple dependent services the need to increase availability is even higher. Set up a High available PVS, auto Deploy service by for example leveraging a Netscaler (or other LB with service check technique) to load balance the TFTP services and streaming services.

VMware Utility Belt must have tools – RVTools 3.6 released

This februari the 22nd RVTools version 3.6 was released. As I use this tool very often, and I notice on some occasions not all consultants are familiar with this tool, I wanted to write a post about RVTools to further spread the word.

This in my opinion is the tool each VMware specialist must have in his VMware utility belt together with the other standard presented tools. At this time RVTool is free and lightweight, very simple in usage, and budget is small (just a donation!) and will not have to be a constrain to use this tool.

What is RVTools?

RVTools is a Windows .NET application which used the VI SDK to display information about your VMware infrastructure.
Connection can be made to vCenter or a single host to get information about hosts, VM’s, VM Tools information, Data stores, Clusters, networking, CPU, health and more. This information is displayed in a tab view. Each tab represents a type of information for when clicking on the tab you will be displayed that specific kind of information from your environment.

RVTools can currently interact with Virtual Center 2.5, ESX Server 3.5, ESX Server 3i, Virtual Center 4.x, ESX(i) Server 4.x, Virtual Center 5.0, Virtual Center Appliance, ESXi Server 5.0, Virtual Center 5.1, ESXi Server 5.1, Virtua lCenter 5.5, ESXi Server 5.5.

RVTools can export the inventory to Excel and CSV for further analysis. The same tab from the GUI will be visible in Excel.

image

image

There is also a command line option to have (for example) a inventory schedule and let the results be send via e-mail to an administrative address.

Use Cases?

On site Assessment / Analysis; Get a simple and fast overview of a VMware infrastructure. The presented information is easy to browse through, where in the vSphere Web Client you would go clicking through screens. When there is something interesting in the presented data you can go deeper with the standard vSphere and ESXi tools. Perfect for fast analysis and health checks.

– Off site Assessment / Analysis; Get the information and save the Excel or CSV dump to get a fast overview and dump for later analysis. You will have the complete dump (a point in time reference that is) which you can easily browse through when writing up an analysis/health check report.

– Documentation; The dumped information can be used on or offline to write up documentation. Excel tabs are easily copied in to the documentation.

– (Administrator) reporting; Via the command tool get a daily overview of your VMware infrastructure. Compare your status of today with the point in time overview of the day before or last week (depending on your schedule and/or retention). Use this information in the daily tasks of adding/changing documentation, analysis, reporting and such.

Release 3.6 Notes

From the version information page, at 3.6 the following has been added:

  • New tab page with cluster information
  • New tab page with multi-path information
  • On vInfo tab page new fields HA Isolation response and HA restart priority
  • On vInfo tab page new fields Cluster affinity rule information
  • On vInfo tab page new fields connection state and suspend time
  • On vInfo tab page new field The vSphere HA protection state for a virtual machine (DAS Protection)
  • On vInfo tab page new field quest state.
  • On vCPU tab page new fields Hot Add and Hot Remove information
  • On vCPU tab page cpu/socket/cores information adapted
  • On vHost tab page new fields VMotion support and storage VMotion support
  • On vMemory tab page new field Hot Add
  • On vNetwork tab page new field VM folder.
  • On vSC_VMK tab page new field MTU
  • RVToolsSendMail: you can now also set the mail subject
  • Fixed a data store bug for ESX version 3.5
  • Fixed a vmFolder bug when started from the command line
  • Improved documentation for the command line options

Pretty fly…

Who?

RVTools is written by Rob de Veij aka Robware. You can find Rob on twitter (@rvtools) and via his website http://robware.net.
Big thank you Rob for letting this excellent tool in the cyberspace!

As the tool is currently free please donate if you find the application useful to help and support Rob in further developing and maintaining RVTools.

vSphere the statistics gathering

For VMware vSphere infrastructures and the how and why my environment is doing that, it is helpful to understand how vSphere and vCenter standardly collect and store statistics, and how these are displayed. The point here is that there is an awful lot of assumption in performance reports or troubleshooting done. When does there assumptions come in to play? When looking at several counters and the way the data is collected (and the what), storage and graphs are made. Especially when selecting intervals how to display (or gather), peaks can become less because they’ve been averaged out over the displayed graph time of the historical (historical vs realtime) or metrics are missing when needed.

How does the statitics gathering work?

Each host stores statistics data for up to a hour via the local performance manager. The performance manager receives realtime instance data from for example the CPU instances. Within the vCenter data collection interval, vCenter performance manager queries each host (that is the hosts that are managed by this vCenter) and vCenter will retrieve a subset of the host statistics data, and stores it in the vCenter database. When, what and how much are configurable in your VMware infrastructure. We have two values for that, interval and levels that can be set on the vCenter.

These collected historical metrics can then be displayed via the vSphere client.

A small model taken from the VMware Documentation Center.

image

Statistics rollups and intervals

As the ESXi host collects the statistics realtime (that is every 20 seconds) these are rolled up to the vCenter database for historical purposes. The vCenter collects data from all of the hosts that the vCenter Server manages. The PerformanceManager defines performance intervals that specify time periods for performance data rollup, a methodology for combining data values. The server stores the rolled up performance counter data in the vCenter database. This is done in four performance intervals that determine how collected instance data is aggregated and stored. The aggregated data is a set of instance data values collected for a performance counter. These intervals can be modified to a limited extent via the collection intervals. These determine the duration for which statistics are aggregated, calculated, rolled up, and archived. Together, the collection interval and collection level determine how much statistical data is gathered and stored in your vCenter Server database.

image

Are those rollups evil? Yes they can be. Peaks can become less because they’ve been averaged out over the graph time. But you must not forget the fact these are historical data, graphs made for a month with 12 data points can show different peak values than week graphs. Know what you are looking for, for which periode and don’t base your conclusion on just one graph.

Statistics Levels

To reduce traffic to the vCenter database vCenter uses a technique to limit which metrics are archived in the database. Certain statistics might be deemed more valuable for your like then others. The statistics levels varies from one to four, with one being the least-detailed statistics level and four being the most detailed.

  • Level one; statistics cluster Services, CPU, Disk, Memory, Network, System, and Virtual Machine Operations counters. Default level.
  • Level two; level one plus all disk, memory and VM operations metrics. Use for long term performance monitoring when device statistics are not required but you want to monitor more than the basic statistics.
  • Level three incorporates level one and two plus per-device statistics, such as CPU usage of a host on a per-CPU basis, or per-virtual machine statistics .Use for short-term performance monitoring after encountering problems or when device statistics are required.
  • Level four. All possible metrics. Use for short-term performance monitoring after encountering problems or when device statistics are required. Only to be used in the shortest amount of time due to the large quantity of data.

The statistics level is used to dictate whether or not a statistic is stored in the vCenter database. If a metric is a level two statistic, but vCenter is configured to level one, this metric is not stored in the database. Not stored means users are not able to query its historical values. Not a problem all the time, but also not good to have if you are looking for just these counters.

These levels sure have their benefits (information at minimal database costs) and drawbacks (possible missing metrics), but the ability to have and change the statistics levels gives something back. We can gather basic information at minimal database cost for the normal running environment with the level one counters. When needed at a troubleshooting scenario we can temporarily increase the statistics level to get more detailed information.

– Happy statistics gathering!

Should I hot add CPU/Mem or go now?

I am doing my VCAP5-DCA prep and using the unofficial official VCAP5 DCA Study Guide to guide me true the subjects (great resource! Check it out at http://www.virtuallanger.com/vcap-dca-5/). And apparently I’m easily distracted, this time with the subject of resource optimization/management and in particular the hot plugging CPU and memory options.

As you probably know there are several device options you can add to a running VM like virtual nics and virtual disks. vSphere 5.5 even has support for hot add/remove of PCIe SSD. These normally work hot out of the box.
For CPU and memory (the virtual ones vCPU and vRAM) this is not the case. They are disabled by default. Why? Well because support of the guest OS is limited, not all hypervisor features are operable (for example FT) and for applications running in the VM support is even more limited. Often you will have to recycle the application service to let it use the added resource or there is possibility of resource degradation/stability. That means downtime and that isn’t what hot add/plug is about. What is the point of using hot add/plug when there is downtime involved? Sure a guest OS restart vs an application service restart takes probably more time on the OS part, but applications work in chains you will know which part (or all) of the layers need attention. This tends to count more for the vCPU hot add, memory hot add/plug is longer around and is incorporated in more guest OS’ses/applications then hot plug vCPU is.

Secondly if you haven’t selected hot add/plug/remove vCPU and vRAM in the creation of your VM (or template), and enabled guest OS support, you will have to power down the VM before being able to change these options. Planning ahead (capacity management) is key here, but that goes for resource management as well. You might as well adjust the resource values you require before powering on the VM.

And what about with elastic resources, is there a hot remove option? For vSphere this is currently a no no, no matter what the guest OS support on this is (on some of the Linux distributions, Windows is also a no no). Memory remove is only done by powering down the VM.

What OS’ses support hot add/plug/remove CPU/Memory?

image

NB. I have left out Windows Server 2003. Yes there is support in there, but this product is bound for EOL. If you are thinking on hot add for this product, rather start thinking about doing a step up in life cycle.

NB. For Linux I just included some distributions. There are of course more supported on VMware.

What applications support hot add/plug/remove CPU/Memory?

This is the harder list to create. For a very few there is public information on support (or just it can be done) of hot add vCPU/vRAM. For applications it is harder to determine whether the resource is added to the OS for usage in the application or for other purposes. Applications must support multi processors to take the full advantage of a virtual socket that is added.

IIS? No. Needs a recycle of service or application pool.

SQL Server? Needs Enterprise edition. After adding vCPUs, execution of the RECONFIGURE statement is needed before it is used.

Exchange? A maybe, again a recycle. There are performance issues rumored when using these features, some have to do with paging depletion afterwards.

Be sure to test test test.

What are you prerequisites to hot add/plug/remove CPU/Memory a running VM?

  • The virtual machine has a guest operating system that supports hot functionality, and must be turned on in the OS.
  • The virtual machine is using hardware version 7 or later. With KB 2050800 there is mentioned an issue with 2012 and windows 8 on hardware version 9 (5.1).
  • VMware Tools is installed.
  • Hot add must be enabled per VM, at the creation time or by powering off.
  • The hardware must be able to support hot plug as well, else there must be resources available to evict the VM’s to an other host while doing maintenance on the physical host.
  • Does not need FT.
  • Does not need vNUMA.
  • Use vSphere Essentials, Essentials plus, Standard, Enterprise or Enterprise Plus. The are limitations per edition you will have to take in account, and versions (for example 5.0 standard edition will not let you use hot add)
  • Check application licensing, how will this effect a per core or per processor application license.

Edit settings of VM to change hot plug/add.

image

image

Conclusion

Do we need to enable hot plug by default on my VM’s? Depending on the usage, only set it on for VM’s that have a tendency to run out of resources quickly, are very in demand and cannot tolerate a little down time in re-configuring. For a small part of the private desktops of the VD environment this can also be a good option to configure before hand, this is normally is a small group of your users for example developers with some private type desktops. For the standard bunch of VM’s/VD’s leave it off, like the products default. You can normally plan ahead the resources for these kind of VM’s. vCenter Operations is great for resource capacity planning. So plan ahead and enable default on all your VM’s? No not yet, there is to little application support out there specially on the vCPU’s. Planning ahead means knowing some more variables and they are not around yet (on the application level that is). Planning ahead is having your resource configured and monitored correctly, when there is need to change, change the resource settings.

The limited OS support and very limited application support is the why to the non adoption of this hot plug features that have already been around a little while. Will we expect it to grow with all the cloud movements and software defined data centers? This depends, when there is a need it will be incorporated in more applications. This list is currently small.

– Back to studying.

vSphere Performance monitoring tools standard available

I am currently working on a project where we are optimizing the virtual infrastructure which consist of vSphere and XenServer hypervisors. In the project we want to measure and confirm some of the performance related counters. We got several standard tools at the infrastructure components to see what the environment is capable of and check if there are some bottlenecks regarding IO flow and processing. 

With any of the analyzing it is important to plan (or know) what to measure on what layer so this is repeatable when wanting to check what certain changes can do to your environment. This check can also be done from some of the tools available, such as earlier written in the blog post about VMware View planner (to be checked at this url https://pascalswereld.nl/post/66369941380/vmware-view-planner) or is a repeat of your plan (which then can be automated/orchestrated). Your measuring tools need to have similar counters/metric throughout the chain, or at least show what your putting/requesting from a start and at the end (but if there is a offset you got little grey spots in the chain).
A correct working time service (NTP) is next to correct working of for example clustering and logging, also necessary for monitoring. To get to right values at the right intervals. Slightly off will in some cases give you negative or values that are off at some components.

Some basics about measuring

You will have to know what the measuring metrics are at a point. Some are integers, some are floating, some are averages over periods or amounts used, some need a algorithm to calculate to human or a similar metric (Kb at one level and bytes on the other, some of them are not that easy). A value that is high in first view but consists of several components and is an average of a certain period, could be normal when devided by the amounts of worlds.

Next up know or decide on your period and data collection intervals. If you are measuring every second you probably get a lot of information and are a busy man (or woman) trying to analyze trough all the data. Measuring in December gives a less representative workload then measuring in a company’s peak February period (and for Santa it is the other way around ;-)). And measure the complete proces cycle, try to get a 4 weeks/month period to get the month opening and closing processes in there (well depending on the workload of course).

Most important is that you know what your workloads are, what the needs for IO is and what your facilitating networking and storage components are capable off. If you don’t know what your VD image is build of for a certain group of users and what is required for these, how will you know if a VD from this groups requesting 45 IOPS is good or bad. At the other hand if you put all your management, infrastructure and VD’s on the same storage how are you going separate the cumulative counters from the specific workload.

Hey you said something about vSphere in the title, let’s see what is standard available for the vSphere level.

VM monitoring. In guest Windows Perfmon counters or Linux guest statistics. The last is highly depending on what you put in your distribution, but think of top, htop, atop, vmstat, mpstat et al.
Windows Perfmon counters are supplemented with some VM insights with VMware tools. There are a lot of counters available, so know what you want to measure. And use the data collection sets to group them and have them for reference/repeatable sets (scheduling of the data collection). 

– Host level; esxtop or vscsistats. Esxtop is great tool for performance analysis of all types. Duncan Epping has an excellent post about esxtop metrics and usage, you can find it here http://www.yellow-bricks.com/esxtop// Esxtop can be used in interactive or batch mode. With the batch mode you can load you output file in Windows Perf mon or in esxplot (http://labs.vmware.com/flings/esxplot). Use VisualESXtop (http://labs.vmware.com/flings/visualesxtop) for enhancements to the esxtop commandline and a nice GUI. On the VMA you can use resxtop to remotely get the esxtop stats. vscsistats is used when wanting to get scsi collections or get storage information that esxtop is not capable of showing. And ofcourse PowerCLI can be an enormous help here.

vCenter level; Statistics collection which depends on your statistics level. Graphs can be shown on several components in the vSphere Web Client, can be read via the vSphere API or again use PowerCLI to extract the wanted counters. To get an overview of metrics at the levels please read this document http://pubs.vmware.com/vsphere-55/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-55-monitoring-performance-guide.pdf or check documentation center for your version.

– vCenter™ Operations Management Suite (vCOPS). Well standard, you still have to option to not include operations in your environment. But your missing out on some of the automated (interactive/proactive) performance monitoring, reporting and insight in your environment options. Root cause analysis is part of the suite, and not down to your own understanding and analytic skills. If you are working on the previous levels your life could have been simpler with vCOPS suite.

Next up

These standard tools need to be supplemented with specific application, networking (hops and other passed components) and storage (what are the storage processors up to is there latency build up in the device it self) counters.

– Happy measuring!

Sites for you bookmark list – VMware WalkThroughs

VMware has a site on the world wide web called vmwarewalkthroughs.com. This site contains product walk throughs. These walk throughs provide a step by step overview of VMware architecture components, and a how to configure these. For the viewer this is not only following the slides, but some interactive clicking of menu or object items. Several subjects are up there like VSAN, NSX and version 5.5 features.

image

The subjects are walked through in an overview knowledge layer of the components and features. For a in depth technical deep dive VMware offers a different site Hands-on-Labs. Hans on Labs can be found at http://labs.hol.vmware.com/.

image

A list of VMware products and their Walkthrough URL’s

Unfortunately the root of the walk through site does not include a subject list. You will have to know the URL’s for your subjects. Some are easy to guess (like VSAN at http://vmwarewalkthroughs.com/VSAN/), others are a bit harder to find (they are included on the VSAN site but not on NSX).

image

The list of subjects:

– Virtual SAN: http://vmwarewalkthroughs.com/VSAN/– NSX: http://vmwarewalkthroughs.com/NSX/
– vSphere Data Protection: http://vmwarewalkthroughs.com/vCloudSuite5-5/– vSphere App HA: http://vmwarewalkthroughs.com/vCloudSuite5-5/
– vCloud Directorhttp://vmwarewalkthroughs.com/vCloudSuite5-5/
– vSphere Replication: http://vmwarewalkthroughs.com/vCloudSuite5-5/
– vSphere Flash Read Cache: http://vmwarewalkthroughs.com/vCloudSuite5-5/