Note to myself and my (finished) VCAP5-DCD exam experience

One of my objectives for this year is to pass the VCAP5-DCD exam and get certified in data center design. Seeing that the year is almost finished I have to put some actions to my words. I already did that by the way, I followed the VMware vSphere Design Workshop earlier this year (so preparations started way earlier than this blog post). Due to some events (of family art that I will not go into in blog post) that happened in the beginning of the year, I had to reschedule and my planning just went over the moon by ending up in October.
But… I have scheduled my VCAP-DCD5 exam at VMworld Barcelona.

I want to write down my experience of pursuing this certification and have a place to store my learning tool links for further reference in the last stint before the exam. So here goes….

What is VCAP5-DCD?

I probably have some readers outside of the VMware world so I first try to explain VCAP5-DCD. This data center virtualization certification track is composed of some acronyms; the first acronym part VCAP which is VMware Certified Advanced Professional, an advanced track of the VMware certification tracks. The second part DCD stands for Data Center Design. DCD is designed for IT architects who design (that a lot of designing ;-) ) and integrate VMware solutions in multi-site, large enterprise, virtualized environments.The third part is the 5, which is the version the certification is for. In this case vSphere 5.x.

The tracks are composed of the following levels:

– VCA. Associate. The first fundamentals.
– VCP. Professional. The IT Professional working with VMware environments.
– VCAP. The advanced professional. Advanced professional working, implementing and designing VMware environments.
– VCDX. The expert. Experts on designing the VMware environments.

Like the data center track VMware also offers Cloud and End User Computing certification tracks.

What is needed?

To achieve the VCAP5-DCD status you will have to be VCP5-DCV and you will have to pass the DCD exam. Pretty straightforward. There is no course prerequisite, but there are some recommended courses to follow like the design workshop I wrote about. The VMware vSphere: Design Workshop [V5.x] is a classroom or online three day course. An other highly recommended course is the free online self paced course Business Continuity and Disaster Recovery Design [v5.X].
Last but not least you will have to have autorisation from VMware to start and schedule your examen, you can do that at

Tools.. what can we use or find?

– The VCAP5-DCD Exam blueprint. The blueprint is intended to provide information about the VCAP5-DCD objectives covered in the exam, related resources, and recommended courses. This is your starting point. Download at
– VMware documentation included in the blueprint. Consist of whitepapers, customer references and such. Jason at created a zip package of all the documents. You can find and download them at the blogpost of Jason
– Take the interactive exam simulation. This gives you insight in the simulations used at the exams (not the subject). Unfortunately this is rather small, but I have not yet find any others around. Access it at
– The VMware vSphere Design 2nd Edition book (also available for Kindle). Get it at:
– VCAP5-DCD Official Cert Guide: VMware Certified Advanced Professional 5  Data Center Design (VMware Press Certification). Kindle edition on Amazon. Great with scenario’s and questions to test your state of knowledge for the subjects. Also do visit writers (Paul McSharry) website
– Free instructional vSphere video’s. Look them up at Be sure to stick to the objectives and not watch all the video’s.
– There are several locations to get the vBrownbags covering the VCAP5-DCD subjects. Go to and register or search for the VCAP-DCD objectives (as they are not yet categorized). The sessions are also available via iTunes.
– Hands on experience. The baggage of knowledge you hold from real world experience. Else get your experience from Hands on Labs. VMware’s HOL is a great online place, go to You can also build your own labs (resources!) in VMware Workstation for example.
– VMware vSphere Plan and Design Service Delivery Kit. If your a solution partner you can grab this kit of partner central.


You can find a lot of VCAP5-DCD experience posts out there, just use your google-fu. Most of those post have one critical component in there (just next to preparation) and that one is time management. The exam consists of 100 questions and 225 minutes. But that not all multiple choices, there are several scenario’s in there. At the beginning of the exam you will be shown how many design tool scenario’s there are, and those together are time bound around one hour. So there is not lot of time to wander and take your time on obscure questions.

For now this is my list. I will try to update when I have some news. When you have some input or advice please drop them in the comments (or tweet them at me).

Update VMworld Exam Experience

So during VMworld I tried the exam. Next to it’s part of my goals, an important reason to do it on the VMworld site is the 75% off the price. But it is hard to do an exam in the middle of the VMworld hurricane, I missed passing with a couple of points. Main reason running out of time and missing some 10 questions and one scenario.
On Wednesday morning I had my chance. I had a bad night of sleep (something with my diner or nerves got my stomach doing somersaults), but I managed to reach the site (and some coffee) on time.

After signing in I entered the exam room. The exam starts with an assessment where you are in the VMware world of designing. I think they use the answers to present you with a specific set of questions, or an order of them. After this the exam starts. At the beginning you are told how much of the 100 questions are scenario’s. In my case there were six in total. After this the show starts for real.

The design question where a real difficulty. I had practiced with the interactive exam simulation (see above for linkie), but in the real world it was hard (maybe fatigue was a problem there to). I lost time just getting re-introduced to the system and lost a couple of drawings when trying to go back in the drawing to much. Next time use the scissors for disconnecting the connections and check if connection are connected properly else moving objects around really screws up your drawing (I knew this upfront, but somehow I didn’t do this in the real exam. Tried the undo to much). I kept count of the number of scenario’s on my note-board. Most of them were in the second part of the exam (well I mean after a started introduce a faster pace as already 2,5 hours were over and I wasn’t even past the half of the questions).

Most of the multiple questions have multiple answers where two or more answers are common. Several have resource subjects where the calculator comes in handy. The drag and drops are straightforward, but you will have to check if you dropped them in the right column. Sometimes the upper part of the column makes your answer go to the column above (same for the below). Stability is ok, I had to move computers when one of my scenario’s returned an error (what was probably due to my excessive use of shuffling around objects or hitting the undo button). But the time missed from the exam by moving is added when starting at the return point of the exam. Your open scenario (the one you are working on) is lost. All previous answers are saved. Unfortunately you miss some concentration.

Four hours weren’t enough for me. I tried to randomly tick some boxes on some remaining question when the countdown started. But I still had some ten questions open. After that the exam forcefully terminated. At somewhat past two in the afternoon it unfortunately did not show the congratulations, I missed about 20 points for that.

I was devastated the first half hour and tried some RR and walk it off around the venue. Exhausted the rest of the day. In the rest of the afternoon I followed some sessions and went to the VMworld party in the evening. There I did some drinks, food and some games. But I left early to catch some sleep.

Okay lesson learned: Don’t do this at a venue like VMworld. Pace should be up. Design tool simulation, repeat close to the exam. Check score report for subjects that need attention, and learn.

Update November 2013 – Passed

Today 15th of November I did my retake of the VCAP5-DCD exam. And this time I managed to finish all the questions and scenario’s in time. Furthermore this time the grade result was a Pass. I introduced a faster pace this time. Again I had six scenario’s, but this time the where almost all in the last part of the exam. But luckily time management was this time on my side.

I did a last repeat of VCAP5-DCD Official Cert Guide and VMware vSphere Design 2nd Edition book in the last week before the exam (just to get the 5.5 out of my system). I also checked the exam simulation and looked if I was confident with the exam blueprint. I was feeling better prepared and settled nicely in the exam. Question where faster answered then the first time, I probably got used to the question style and the exam drag and drops/scenario’s from the first time (however this is still partly a blur in my memory). The only hesitation was at the ending of the exam, shall I push the end exam button? Fortunately for me it showed congratulations!

– So with this I finish my DCD experience. What is going to be next?

Evaluations – Veeam ONE v7

A few blog post back I did a evaluation of Veeam Backup and Replication v7 (read it at A logical step from a backup and replication solution is a management solution to manage your back-up and replication and monitor your virtual infrastructure from one solution. Veeam has the Veeam ONE product for that.
Veeam actually has an other solution for management and monitoring of your back-up solution, which is targeted at enterprise customers; Veeam Backup Management Suite. You will miss the monitoring of the virtual infrastructure with the latter (and probably gain some, but that is a vs evaluation and outside this scope). For now I will concentrate on Veeam ONE.

What is Veeam ONE?

Veeam ONE is a single solution for managing virtual infrastructures and Veeam Backup and Replication. This solution enables real-time monitoring, capacity planning, documentation, mapping and reporting for virtual infrastructures based on VMware vSphere or Microsoft Hyper-V, and Veeam Backup and Replication.

Veeam ONE comes in a licensed full (per CPU socket of a managed/monitored host) and a free edition. The free edition includes all the core functionality of the full, but is restricted in some of the features (either a lower threshold or not available). These restricted or not available features limit the scale of the monitored infrastructure, amount of historical data and reporting. Thus limiting your capabilities to thoroughly analyse, trend and forecast your environment. But for small deployments this is less of a problem (as there are other means) then bigger environment (those means are not automated or from a single management solution).


So we now know the why, now we need the what some Veeam ONE Architecture. The Veeam ONE architecture is composed of the Veaam ONE components and the components of the monitored infrastructures. As stated above, Veeam ONE can monitor virtual infrastructures from VMware vSphere and Hyper-V. Veeam ONE is deployed as either a virtual in these environments (probably in a back-end or infrastructure cluster) or as a physical server outside these infrastructures. The virtual infrastructure nodes can be monitored as hosts or via management such as vCenter or SCVMM.
Secondly Veeam ONE monitors Veeam Backup and Replication so it needs to be able to access the Veaam Backup Server.


But Veeam ONE of course has it’s one architecture as well. Veeam ONE is a client server architecture and incorporates the following structural components:
– Veeam ONE Server – a virtual or physical server responsible for collecting data from virtual infrastructure components (hosts, vCenter or vCloud Director), Veeam backup and replication and storing this to a SQL Database. Veeam One Server actually is about two parts Veeam ONE Monitoring Server and Veeam ONE Reporting Server.
– Veeam ONE Web UI – the client part that communicates with the SQL database to access data for viewing reports and customizing infrastructure views. The client is composed from the Reporting Client and the Business View Client.
– Veeam ONE Monitor Client – the client part that is used to connect to the Veeam ONE Monitoring server. This is the primary tool for monitoring your environment.
– Veeam ONE Database. Internally MSSQL Express 2008R2 or MSSQL server (2005 on to 2012) outside the environment. For reporting MSSQL 2008 Reporting service could be included.

Deployment of Veeam ONE can be done in a typical setup with all components on one server, or an advanced setup with components separated on several servers.


For the evaluation I’m doing a simple typical deployment with a Server 2012 host as Veeam backup server and repository, Veeam ONE server, and a VMware ESXi 5.5 host managed by a vCenter Server 5.5 Appliance (which are not yet supported but will find out if it works. Do not use this in a production environment).


The wizard starts when you push the appropriate installer option, in my case the Veeam ONE server. You can input your license file for the full edition or use the free edition when you have got none. I’m using a NFR license for lab/demo purposes.


Next up I’m selecting the typical setup option. After this prerequisties are checked and your are shown the results. If there are failed in there (my setup wasn’t prepared) you have to option to let the installer install (push that button) for you.


Wait for those to be installed and a re-check is performed. Continue when status is passed.

I’m using the default for the locations, be sure to change them to your needs. Add a service account. Preferably from the domain. My lab consist of a single Windows 2012 server which I haven’t added the AD DS role, so I’m going for local.
For the database use a existing SQL instance or let the installer add a MS SQL Express 2008R2 one for you. I’m going for an existing database instance, the express I installed with the local Veeam Backup and Replication installation.


When using an existing one, be sure your service account is granted access and permissions.

Ports can remain the default ones.
You can now connect to your virtual  infrastructure from the installer. Same goes for you Veeam backup and replication. I’m doing it from the installer, connecting to VCSA (as accessing vCenter has not changed from 5.1 to 5.5 Veeam connects to vCSA 5.5) and local Veeam Backup and replication.

You can add or change these connections at a later time, in that case just select the skip options.

And hit the install button to start going for distance….


To finish the installation you will have to log off.

This gives us the three application icons.

Opening business View icon will open a Internet Explorer (or other preferred browser) with the application URL. For IE you probably have to trust the applications or they will standard be blocked. Veeam ONE monitor gives you insight in your infrastructure, business (after defining), data protection (backup and replication like shown below) and alerting.

Next up are the defining of notifications, rules, categories (for example SLA), groups, etc. to have your environment be monitored with organizational needs. But for now we will stop here.

This concludes the introduction of Veeam ONE and the typical (and basic) installation of Veeam ONE.

– Enjoy Veeam ONE with your environment!

VMware vCloud Director – the what and evalution

Last week I was asked about vCloud Director. I haven’t worked with vCloud Director the last couple of projects (single users of their private infrastructure). I thought I can use this question to buff up my vCloud Director skills, update my lab to 5.5 and do a blog post in one.

vCloud director, what’s that?

vCloud director is a solution from VMware to enable organisations to build multi-tenant private clouds. This is done by creating virtual data centers on infrastructure resources. Users (or tenants) of those virtual data centers can use vCloud Director to consume their resources in a service that is offered through a Web portal. The web portal enables self service to cloud provider customers (Infrastructure as a Service, IaaS) or internal tenants for multi-tenant organisations.

And what do you need? – A litte vCloud Director architecture first.

vCloud director consist of the following components:

– A vCloud director “cell”. A single instance of vCloud Director is known as a “cell.” This can be vCloud Director installed on a VM with supported OS or this can be a vCloud Directory appliance. Important Note here: the vCloud Director appliance is not for production (5.5 still not supported). It is designed for Proof of concepts or demo environment (lower scale single cell). Appliance are designed for removing complexity. So yes.. I’m using the appliance in this evaluation.
– vCloud Director database. Information about objects, users and other vCloud related data is stored in a MS SQL or Oracle database. With more then one cell in your environment, all cells communicate with one vCloud Director database. This component is critical and should be high available.
– vCenter Server. The vCenter instance supplies a connection for vCloud Director with the needed resources, such as CPU and memory. vCloud can be connected to one or more vCenter instances. For my demo I’m using a vCenter Server Appliance 5.5.
– ESXi hosts. They provide the computing power. They are grouped in clusters or resource pools via vCenter. With these hosts also comes attached storage resources, which in their case can be clustered also. I have 5.5 in my lab.
– vCloud Networking and Security Manager (again 5.5). To enable the support of automated management of vCloud networking and security (integrating vShield), an instance of vCloud Networking and Security Manager is required for each vCenter Server attached to vCloud Director.


The above model shows all the mentioned components brought together.

Lab time – evaluation with initial installation of components

I have one vCSA and one ESXi host added in my notebook lab. I’m downloading the vCloud director appliance and vCloud Networking and security vShield manager appliance.
These can be easily deployed to your inventory as an OVF template. These deployments are really straightforward.
One note for the networks of the vCloud director: a vCloud Director appliance requires two network adapters and IP addresses. The one is used for HTTP traffic and to connect to the vCloud Director user interface. The other is for the console proxy connection that is used for all VMware Remote Console (VMRC) connections and traffic. When the networks are on the same IP subnet, the lowest number (eg. 132 when the other ends with 133) is assigned to the web service (https://<lowest_Ip>/cloud).

In a production environment these should connect to two different networks. For the lab I’m connecting both network adapters to the same network (and thus ignoring the warning shown).

I’m using the internal database option, so my demo lab does not need a database server.

My Lab now looks like this:


So with VCSA 5.5 deployed and initially setup (accepting EULA and accepting defaults). I’ll add a ESXi host. Nothing fancy. Minimum 4GB memory. Well it actually can go down, but you first have to install ESXi with 4GB and then downsize when finished. And vCD needs some memory for Oracle XE. So better stick to 4GB minimum.

I will go a head and initially configure vShield Manager…. ehhrm I mean vCloud Networking and Security Manager.
I add the VM to the same network, minimize memory for lab purpose and power on. When booted the manager login is shown. I logon with admin and password default. Go to the privileged mode (by using enable, at that time my Cisco days are passing in my memory) and enter set-up. Here I configure the IP address in the same range as my other hosts (that are DHCP clients). Open up a web browser to the configured IP address and the interface is shown. We can the same default user as the one used at the manager login.

The minimal item we have to setup is a connection to vCenter. Click the edit button next to it and add the vCenter server to the VSM.


After this we can close VSM and check that vCloud networking and security manager is successfully manageable from vCenter.

Next up deploying vCloud Director appliance. We will need to do this from the vCSA and deply as ovf.

So deploying to correct datacenter, storage and getting the warning for the same networks, setting root and guest password and adding IP’s or using DHCP (the latter for now)



And deploy when complete (and yes I switched from web client to vSphere client for the screen shots).

Start your engines. When the VM is booted, you will have to complete the initial installation by open https://<your VCD ip>/cloud. I have not changed the default user and password so root and vmware it is.

Accept the license agreement, add a valid (trial) license, and a administrator account. Add a unique system name and ID. The system name will be used to add a folder to your infrastructure. The ID should be unique to prevent conflicts. Starting with 1.

And finish. You are now returned to the vCD login screen. The last items for this eval:
Login and choose attach New vCenter from the Quick Start you are shown. Insert the name, ip, port and users for your vCenter. And next add to VSM. Ready to complete.

After this you will have to configure a provider data center and define resources (storage, networking) that will be consumed by organisations. You will have to create these organizations as well.

But for now this will be enough.

– Enjoy!

Exchange DAG Rebuilding steps and a little DAG architecture

At a customers site I was called in to do a Exchange health check and some troubleshooting. As I have not previously added Exchange content to this blog, I thought on doing a note experience and new blog post in once.

The environment is a two site data center where site A is active/primary and site B is passive/secondary. Therefor a two node Exchange is deployed on Hyper-V. The node in site A is CAS/HT/MBX en the node in site B is CAS/HT/MBX. The mailbox role is DAG’ged, where active is site A and database copies are on site B. There is no CAS array (Microsoft best practice is to set it even if you have just one, but this wasn’t the case here). This is not ideal as a fail-over in CAS doesn’t allow clients to auto connect to another CAS, Exchange uses the CAS Array (with load balancer) for this. The CAS fail-over is manual (as is the HT). But when documented well and small amount of downtime is acceptable for the organisation, this is no big issue.
Site A has a Hyper-V cluster where the exchange node A is a guest hosted on this cluster. Site B has a unclustered Hyper-V host where Exchange node B is a guest. Exchange node A is marked high available. This again is not ideal, yes maybe for the CAS/HT role it can be used (should then be separated from the mailbox role), but for the mailbox role this is application layer clustered already (the DAG) so preferably off. Anyhow these are some of the pointers I could discuss with the organisation. But there is a problem at hand that needs to be solved.

The Issue at hand (and some Exchange architecture)
The issue is that the secondary node is in a failed state and currently not seeding it’s database copies. Furthermore the host is complaining about the witness share. You can check the DAG health with PowerShell Get-MailboxDatabaseCopyStatus and Test-ReplicationHealth.

You can check the DAG settings and members with Get-DatabaseAvailabilityGroup -Identity DAGNAME -Status | fl. Here you can see the setup file witness server.

RunspaceId : 0ffe8535-f78a-4cc1-85fd-ae27934a98e0
Servers : {Node A, Node B}
WitnessServer : Servername
WitnessDirectory : Directory name on WitnessServer
AlternateWitnessServer : Second Servername
AlternateWitnessDirectory : Directory name on Second Witness Server

(AlternateWitness server is only used with Datacenter Activition (DAC) Mode DAGOnly, here it is off and therefore not used and not needed)

Okay witness share, some Exchange DAG architecture first. Exchange DAG is a Exchange database mirror service build on Fail over cluster service (Microsoft calls it hidden cluster service). You can mirror the databases in a active/passive solution (one node is active to other is only hosting replica’s), or in an active/active solution (both nodes have active and passive databases). In both solutions that is high availability and room for maintenance (in theory that is). The mirror service is done by replicating the databases as database copies between members of the dag. The DAG uses Fail over clustering services where the DAG members participate as cluster nodes. A cluster uses a quorum to tell the cluster which server(s) should be active at any given time (a majority of votes). In case of a failure in heartbeating networking there is a possibility of split brain, that both nodes are active and try to bring up the cluster resources as they are designed to do. Both nodes can serve active databases with the possibility of data mismatch, corruption or other failures. In this case a quorum is used to find out which node has more votes to be active. A shared disk is often used for the cluster quorum. An other option is to use a file share on a server outside the cluster, the so called file witness quorum or file witness share in Exchange.


The above model shows the CAS and DAG HA components. With Exchange architecture best practice the File Witness share is to be placed on the HT role, but in the case of mixed roles you should select a server outside the DAG and in this case outside the Exchange organisation. Any file server can be used, preferably a server in the same datacenter as the primary site serving users (important).

So back to the issue. File witness share (FWS) access. I checked if I could see the file share (\servernameDAGFQDN) from the server and checked permissions (Exchange Trusted Subsystem and the DAG$ computer object should full control). The Exchange trusted subsystem must be a Adminstrators local group member. The FWS is placed on a domain controller in this organization. Not ideal again (Exchange server now need domain level administrators group membership as domain controllers don’t have local groups), but working.

I checked the failover service and there the node is in a down state, including it’s networks. But in the Windows guest networks are up and traffic is flowing from and to the both nodes and the FWS. No firewall on or between the nodes, no natting. Okay……Some other items (well a lot) where checked as the where several actions done in the environment. Also checked Hyper-V settings and networking. Nothing blocking found (again some pointers for future actions).

Well, try to remove and add the failed state node to the DAG. This should have no impact on the organization and the state is already failed.

Removing a node from the DAG.

Steps to follow:
1. Depending on the state, suspend database seeding. When failed, suspend via Suspend-MailboxDatabaseCopy -Identity <mailbox database><nodename>. When status is failed and suspended this is not needed.
2. Remove Database copies of mailbox databases on the failed node. Use  Remove-MailboxDatabaseCopy -Identity <mailbox database><nodename>. Repeat when needed for the other copies.
3. Remove Server from DAG.  Remove-DatabaseAvailabilityGroupServer -Identity <DAGName> -MailboxServer <ServerName> -ConfigurationOnly
4. Evict from cluster.
As the cluster is now only one node, the quorum is moved to node majority automatically. The FWS object is removed from the config.

Rebuilding the DAG by adding the removed node back

Steps to follow:

1. Add server to DAG. This will add the node back to the cluster.  Add-DatabaseAvailabilityGroupServer -Identity <DAGName> -MailboxServer <ServerName>. Succes the node is healthy.
2. Add the database copies as preference 2 (the other node is still active). Add-MailboxDatabaseCopy –Identity <Mailbox Database> -MailboxServer <ServerName> -ActivationPreference 2.
3. In my case to time between fail state and returning to the DAG was a bit long. The database came up, but returned to failed state. We have to suspend and manually seed. Suspend-MailboxDatabaseCopy -Identity <mailbox database><nodename>.
4. Update-MailboxDatabaseCopy -Identity “<Mailbox Database><Mailbox Server>” -DeleteExistingFiles. Wait for the bytes are transferred across the line. When finished the suspended state is automaticaly lifted.
Repeat for the other databases.
5. You will now see a good state of the DAG and databases in Exchange Management console. Not yet. The file witness share is not yet back.
6. Add the Witness share from Exchange powershell. Set-DatabaseAvailabilityGroup -WitnessDirectory “<Server Directory” -WitnessServer “<Servername>” -id “<DAG Name>”. When the DAG members are minimal two the FWS is recreated. This is also visible in Failover Cluster.

Root Cause Analyse

Okay this didn’t went so smooth as described above. When trying to add the cluster node back to the cluster this fails with the FWS error again. In cluster node command output it is noticed that on Node A Node B is down. And on Node B Node A is down and Node B is Joining. Hey wait there is a split and the Joining indicates that Node B is trying to bring up it’s own cluster. Good that it is failing. When removing the node from the DAG Kaspersky Virus protection is loosing connection as this is configured to the DAG databases. At the same time Node A has the same errors and something new, RPC Server errors with Kaspersky. Ahhhh Node A networking services not correctly working is the culprit here. The organisations admins could not tell if networking updates and changes had a maintenance restart/reboot. So there probably something is still in memory. So inform the users, check the backup and reboot the active node. The node came up good and low and behold node B could be added to the fail over cluster. At this time I could rebuild the DAG. Health checks are okay, and documented.

– Hope this helps when you have similar actions to perform.


VMware vSphere Auto Deploy and the GUI fling

As in my earlier IO Analyzer blog post, flings in VMware labs are an excellent place for very useful tools and extra’s for your VMware environment. You can find the VMware flings page at

One of the flings I want to blog about is the Auto Deploy GUI. This fling is a front end graphical user interface to the Auto deploy server. The standard auto deploy proces is heavily VMware PowerCLI based. This can be a problem at some organisation where IT personnel is not yet PowerCLI/Powershell familiar. No excuse, but it can be helpful to lower prerequisite knowledge and add a GUI to the process. This way it might be easier for those organizations to accept VMware auto deploy.

Let me be clear…… this is no excuse to not learn PowerCLI. So please up your PowerCLI/Powershell skills as you will use that at a lot of places (also outside of the VMware infrastructure) and makes you life a lot easier (well, that is… after you learn it).

What is Auto Deploy?

Before you add something to a infrastructure, the Auto Deploy components should be known. vSphere Auto Deploy facilitates a infrastructure for automatic server provisioning and network deployment of the ESXi hypervisor. The deployment can be on local storage, statefull on HDD, SD or USB or stateless to the hosts ram. It works in conjunction with:

– vCenter,
– host profiles,
– TFT server,
– Auto Deploy server and Image Builder,
– a PXE boot infrastructure with a DHCP service.

These service can be installed on the vCenter host or hosted/integrated on specific services.  When using the stateless host option be sure to have a high available Auto Deploy infrastructure.

Auto Deploy and host profiles are available from the Enterprise plus Edition.


Auto Deploy server can be installed on a Windows based server, or can be used on or with the vCenter Server Appliance (vCSA).

Why a VCSA, when the 5.1 version with embedded database is for small deployments (maximum of 5 hosts) I hear you ask? Automation and centralized management! …And the fact the vCSA 5.5 will support a lot more hosts…..

As we need a Windows based service for the GUI and normally would need a Windows server for Update Manager, we can combine those on Windows based server.

For this blog post I’m using a 5.1 vCSA installation and Windows based server operating on Windows 2008R2 running Update Manager and Auto Depoy services (including DHCP for PXE boot).

Setting up Auto Deploy services

Here we had a choice (why always these choices…..) to use the vCSA vCenter server components together with Auto Deploy service or you a standalone Windows server for auto deploy services working together with the vCSA for vCenter services. The last makes a little more sense as the Auto Deploy GUI also needs Windows components, and so will Update Manager. If you happen to want to use auto deploy on the vCSA the service needs to be started. Like stated above I’m doing a Windows based installation in conjunction with the VCSA for vCenter services.

How the lab is build:

– VCSA5.1 downloaded and setup.
– Also downloaded the vCenter VIM installer to install Auto Deploy on the Windows host (You will also need this installer if any other vCenter service need to be on a Windows system. For example Update Manager).
– Windows 2008R2 Set up. All defaults.
– DHCP Role added to Windows 2008R2. You can setup your IPv4 scope here, but I will set it up when I’m ready for the TFTP to service. (And don’t forget other devices that offer DHCP such as your internet router, separate the traffic. I have added a LAN segment and let the W2K8R2 DHCP only serve this network. ESXi VM should be connected to this same network).
– Downloaded TFTP Server from Solarwinds (Free version at
– The TFTP server needs .Net Framework 3.5, and so does Auto Deploy Gui so install it to the server (Add Feature).
– Installed TFTP server on Windows 2008R2 and setup starts it. The folder c:TFTP-Root is used default.
– Install PowerCLI.
– Install Auto Deploy service. The default installer will register Auto Deploy with the VCSA.
– Setup a DHCP scope with option 66 = Boot Server Host Name to the IP of the Windows server. And add option 67 = Nothing Yet. We are gonna add the BootFile name later.
– Let’s check if the DHCP scope can serve a ESXi host. Create a VM to the DHCP served LAN segment. And start it up. DHCP is received, TFTP is looked up. And fails because it can’t find Nothing Yet. But we know DHCP is ok.


We can setup the boot image by opening the vSphere client and connecting to the vcenter. Click Auto Deploy (Administration). This will open the following screen:


Now click the download TFTP Boot Zip link and save to (and extract) to the TFTP-Root directory. You will probably need to change the file download to enabled to the IE security settings for Internet zone.

Change the DHCP option 67 to “undionly.kpxe.vmw-hardwired”. (still the PXE boot will fail because no ESXi image is yet prepared)

Setting up Auto Deploy with GUI

First we install the GUI this is really straightforward. This adds an plugin under Solutions And Application to your environment.


(This actually also has the link to download TFTP Boot zip).

Next up configuring your environment.

1. First up VMware depot, Right click and check or add VMware depot url to­‐depot-­‐index.xml. (default it is in)
2. Next HA depot. Right click HA depot url. This should read http://<vchostname>/vSphere-­‐HA-­‐depot/index.xml. Else add it.
3. If you need a specific custom component (for example a Nexus VEM) you can add a zip depot.
4. This will fill up the Images in the Image Builder screen. Here you can build up your organization specific images (with specific software packages). For now I leave the defaults and move on.
5. Now we create the first deploy rule. Click the add rule and fill in the name,
6. I set it to the ESXi5.1 standard image, select where it must land (I select a host folder I created), select an host profile we skip (not yet created, if you have select your appropriate profile), in the rule set you can setup up a specific pattern (for example asset tag or vendor) very useful but not for this demo. I select apply all.

This will start up tasks to inject VIB’s to the cache. After this the rule is created.


7. Activate the newly created rule by right clicking and selecting active.

Start up the test VM created earlier (or reboot when it’s still is failing) and see if the host is added to the Auto Deploy Host folder.


Yup this time it found an image. Loading and you will notice a cache loading of ESXi next.


Received a DHCP address for the booted host. Let see if vCenter shows the host added to the correct host folder.


Success, base image and add to a vCenter managed infrastructure is done. Warning is about the unconfigured state and to non-persistant storage for the scratch partition.

What’s next?

To use your ESXi hosts in auto deployment scenario you will have to set up host profiles and add this to a deployment rule (or more if you have several environments).
Configure a host to be setup according to your environment (DNS, NTP, networking, name it….). Create a host profile from this host and fill up an answer file. Check compliancy to be sure this one’s correct.

Add this profile to the deployment Rule and voila you ESXi is deployed and setup in a profile.

– Enjoy your Auto deploy infrastructure with the Auto Deploy GUI! Be sure to learn PowerCLI another time!

Evaluations – Veeam Backup and Replication version 7- What and Installation.

And now for something completely different… Well different, still has to do with a virtual infrastructure. Evaluating the version 7 of Veeam Backup and Replication.

What is Veeam Backup and replication?

Veeam Backup and replication is a data protection and disaster recovery solution for virtual infrastructures. It supports virtual infrastructures from VMware or Hyper-V.
It brings features such as instant file-level recovery and VM recovery, scalability, backup & replication, built-in de-duplication and bringing centralized back-up and replication management to your infrastructure.

To produce a backup, Veeam Backup leverages VMware snapshot capabilities. When you need to perform backup. The VMware snapshot technology lets you back up VMs without suspending them; also known as online hot backup.



The picture above (picture credits to the Veeam Evaluation guide. Get this guide at shows the components that make up the Veeam Backup and replication infrastructure:

  • Veeam Backup server—a physical or virtual machine. The Veeam Backup server is the core component: responsible for configuration and management.
  • Backup proxy—a “data mover” component used to process VM data and transfers to the datastore targets.
  • Backup repository—a storage location for storing backup files, VM copies and replicas.
  • Virtual infrastructure servers—ESXi or Hyper-V hosts which are sources and targets for backup and replication operations.


For the evaluation I’m doing a simple deployment with a Server 2012 host as backup server and repository, and a VMware ESXi host managed by a vCenter Server Appliance. I’m not using multi core/processors so you will get a warning about data processing times.


The wizard starts when you push the appropriate installer. You can input your license file or use the free edition when you have got one. I’m using a NFR license for demo purposes.


I’m doing the complete setup. Not changing the default install, I currently just have one disk connected. Prerequisite software checks are done next. If you are not compliant push the install button to get the required software.

Connect with a local admin (from domain or not) and use a existing SQL instance or let the installer add a MS SQL Express 2008R2 one for you (I’m currently going for the latter). Ports can remain the default ones. Same goes for the locations, be sure to change them to your needs. And hit the install button to start the engines….


And have a little patience for the installer to finish. And lift off..


Now let’s add the virtual servers. Go to Backup Infrastructure – Managed Servers and right click to select add server. You can select vSphere, vCloud, Hyper-V and Windows hosts. Add the VCSA via the vSphere option.

Add the VCSA credentials to the Wizard (in my case the standard root vmware combo). It takes a while as my lab has not enough resources…
The Wizard will create a new VMware object in the backup inventory.


Next up, the backup proxy. As described earlier, this is the data mover and needs access to the source and destination datastore. This is a Windows server with either a physical connection (physical server with LUN’s attached) or a VM. Add it as a managed server (add a windows server at managed servers) and assign the backup proxy role (add at backup Proxies.). I am using the same server for all roles, so it is already added to the server list and to the VMware proxy by the Veeam Wizard.


Next up: a backup repository. This can be a:
– Windows Server with storage attached.
– Linux server with local or NFS mounted storage.
– a CIFS share.

I have added a vmdk to my server, and am using this as the backup repository. So I add a repository to a Microsoft Windows server, to this server and use populate to find the appropriate disk. For additional features I’m also adding this as a vPower NFS server. image

And boom, your Veeam infrastructure is up and running in minutes. Just know the architecture components and prepare in advance. Surely this test lab is not sufficient for production as I haven’t taken retention, archiving, access and RTO/RPO in mind.

Next up is creating some jobs and fill up the repository. Go to backup & Replication pane, and add a backup job.

The add backup job is straightforward  Select the source machine and th what. Select the destination and which proxy to use.


One of the important screens is the Advanced Settings.


here the mode can be selected, storage and methods (use vSphere CBT).

And viola start you engines, a test job can be run.


This concludes the Veeam Backup and Replication introduction and basic installation.

– Enjoy Veeaming across your virtual infrastructure.

Dissecting vSphere – Data protection

An important part of a business continuity and disaster recovery plans are the ways to protect your organisation data. A way to do this is to have a back-up and recovery solution in place. This solution should be able to get your organization back in to production with the set RPO/RTO’s. The solution needs to be able to test your back-ups, preferable in a sandboxed testing environment. I have seen situations at organisations where backup software was reporting green lights on the backup operation, but when a crisis came up they couldn’t get the data out and thus failing recovery. Panicking people all over the place….

Back-up and recovery solution can be (a mix of) commercial products to protect the virtual environment like Veeam or from within guest with agents like Veritas or DPM or from features of the OS (return to previous version with snapshots). Other ways included solutions on the storage infrastructuur. But what if your budget constrained….

Well VMware has the vSphere Data Protection that is included from the Essentials Enterpise Plus kit. This is the standard edition. The vSphere Data Protection Advanced edition is available from the enterprise license.
So there are two flavours, what is standard giving and lacking from advanced?
First the what; like previous stated VDP is the backup and recovery solution from VMware. It is a appliance that is fully integrated with vCenter. It’s easy to be deployed. It performs full virtual machine and File-LevelRestore (FLR) without installing an agent in every virtual machine.It uses data deduplication for all backup jobs, reducing disk space consumption.


VDP standard is capped with a 2TB backup data store, where VDP advanced allows dynamic capacity growth. This allows a growth of capacity to 4TB, 6TB or 8TB backup stores. VDP advance also provides agents for specific applications. Agents for SQL Server and Exchange agents can be installed in the VM guest os. These agents provides selecting individual databases or stores for backup or restore actions, application quiescing and advanced options like truncating transaction logs.


At VMworld 2013 further capabilities of VDP 5.5 are introduced:

– Replication of backup data to EMC.
– Direct-to-Host Emergency Restore. (without the need for vCenter, so perfect for backing up your vCenter)
– Backup and restore of individual VDMK files.
– Specific schedules for multiple jobs.
– VDP storage management improvements. Selecting separate backup data stores.

Sizing and configuration

The appliance is configured with 4vCPU’s and 4GB RAM. For the available backup stores storage capacity 500GB, 1TB or 2TB they will consume respectivily 850GB, 1,3 TB and 3,1TB of actual storage. There is a 100 VM limit, so after that you would need another VDP appliance (maximum of 10 VDP appliances per vCenter).

After the appliance deployment the appliance need to be configured at the VDP web service. The first time it is in installation mode. Items such as IP, hostname, DNS (if you haven’t added these with the OVF deployment), time and vCenter need to be configured. After completion (and sucessful testing) the appliance needs to be rebooted. A heads up, the initial configuration reboot can take up to 30 minutes to complete so have your coffee machine nearby.

After this you can use the webclient connected to your VDP connected vCenter to create jobs. Let the created jobs run controlled for the first time; the first backup of a virtual machine takes time as all of the data for that virtual machine is being backed up. Subsequent backups of the same virtual machine take less time, here changed block tracking (CBT) and dedup is preformed.


Well this depends on the kind of storage you are going to use as the backup data store. If you going for low cost storage (let say most of the SMB would want that), your paying in performance (or lacking it most of the time).

Storage Offsite

Most organizations want their backup data stored offsite in some way. vDP does not offer replication (or with VDP5.5 to only EMC), so you want to have some offsite replication or synchronization in place (and a how are you able to restore from this data if your VDP is lost also). vSphere Replication only protects VM’s and not your backup data store. Most SMB’s don’t have a lot of storage able replication devices in place, and when they do, there using it for production and not use that as a backup datastore. Keep this in mind when researching this product for your environment.

– Enjoy data protecting!

Dissecting vSphere 5.5 Enhancements – HA improvement and App HA

With the introduction of vSphere 5.5 there are two mayor HA improvements announced:

– vSphere App HA, on the Intarweb also known as App aware HA; High Availability at the application layer.
– vSphere HA detecting VM antiaffinity rules.

I’ll start with the latter.

HA detecting VM Anti-affinty rules

With vSphere DRS… Hey wait isn’t the subject supposed to be HA… Well yes, but the anti- or affinity rules are DRS rules. So a bit of DRS rule explanation;..these rules helps maintain the order of placements of VM’s on hosts throughout the cluster.  Affinity rules are rules that places VM’s together on certain hosts. Anti-affinity rules are rules that places VM’s separate from those VM’s in the rule. Think of VM’s that are already in a software availability service, such as the nodes of a cluster. You don’t typically want the nodes on one physical host.
With vSphere 5.1 and earlier vSphere HA did not detect a violation of these rules (these rules are unknown to vSphere HA). After a HA failover the VM’s could be place on the same host, after vSphere DRS would kick in and vMotion the VM’s so the anti-affinity rules are satisfied (DRS needs to be in full automated to enable the auto vMotion). Applications with high sensitivity to latency would not like this vMotion and there is a (very slight) moment that HA application clustering service are at higher risk as both VM’s are on the same physical host. A failure of the physical host before the vMotion is completed, would impact a downed service.
In a application cluster service you could also choose to use VM Overrides to disable HA restart for the VM cluster nodes as the application service handles the application HA actions. After a failure you would have to manually get the failed node online (or add a new one) in the application service. But that looses automation…

With vSphere 5.5 HA has been enhanced to conform with the anti-affinity rules. In a case of a host failure the VM’s are brought up accordant to the anti-affinity rules without the need of a vMotion action.This enhancement is configured as an advanced option.

vSphere App HA aka App aware HA

We already have host and VM monitoring, with vSphere 5.5 lifts this to application monitoring. vSphere App HA can be configured to restart an application service when an issue is detected with this service. It is possible to monitor applications as IIS, MSSQL, Apache Tomcat and vCenter. When the application service restart fails App HA can also reset the virtual machine. Service actions can be configured with the use of App HA policies. VM monitoring must be enabled to use application monitoring.

App HA Policies are definitions of the number of times vSphere App HA will attempt to restart a service, the number of minutes it will wait for the service to start, and the options to reset the virtual machine if the service fails to start and to reset the virtual machine when the service is unresponsive. They can also be configured to use other triggers, such as e-mail notifications or vCenter alerts.

When a configured App HA policy is assigned to a specific application service, vSphere App HA is enabled for that service.

Pretty nice.

But what’s needed:

For App HA to work two appliances are needed in the environment (per vCenter), vSphere App HA and vFabric Hyperic. The latter is used by the App HA architecture to monitor applications and is a vFabric Hyperic Server that communicates with vFabric Hyperic agents.
The roles of the both appliances are as follow: the vSphere App HA virtual appliance stores and manages vSphere App HA policies. The vFabric Hyperic appliance monitors applications and enforces the assigned vSphere App HA policies. For monitoring the applications of a VM, vFabric Hyperic agents must be installed inside the VM’s of these applications. These agents are communication brokers for the applications of the VM’s and the vFabric Hyperic appliance.

The vFabric Hyperic agents are supported to be deployed at Linux and Windows os’ses for 32-bit or 64-bit applications. How and what is supported for vSphere 5.5 HA is not yet completely clear (service support for IIS6/7/8, MSSQL 2005/2008/2012, Apache Tomcat, Apache HTTP and vCenter). Following the current vFabric Suite supported OS’ses these include Windows 2003, Windows 2008R2, Red Hat Enterprise Server and Suse Enterprise Linux.



Well. Good Question. App HA is part of the vSphere Enterprise plus edition only. Costs of vSphere 5.5 is expected to be around the current vSphere 5.1 costs. But with what options, constrains and limits…..unknown. The General Availability of vSphere 5.5 is yet unknown.

Separately VMware vFabric Suite is currently available as a one-time perpetual license under which support and subscription (SnS) contracts can be renewed annually – See more at:

How the both are combined at what options/editions/prices keep a look out for further vSphere 5.5 product announcements.

– Exiting. I have the HA BCO5047 – vSphere High Availability – What’s New and Best Practices in my Barcelona schedule to get some more insight at VMworld EU 2013.