Hyper-V Live Migration: A Step-by-Step Guide

Live migration is probably the most important technology that Microsoft has added to Hyper-V in Windows Server 2008 R2. It enables virtual machines (VMs) to be moved between Hyper-V hosts with no downtime. Using live migration, you can migrate all VMs off the Hyper-V host that needs maintenance, then migrate them back when the maintenance is done. In addition, live migration enables you to respond to high resource utilization periods by moving VMs to hosts with greater capacities, thereby enabling the VM to provide end users with high levels of performance even during busy periods.
      Live migrations can be manually initiated, or if you have System Center Virtual Machine Manager 2008 R2 and System Center Operations Manager 2007, you can run automated live migrations in response to workload. You need to complete quite a few steps to set up two systems for live migration, and I’ll guide you through the process. First, I’ll explain how live migration works. Then I’ll cover some of the hardware and software prerequisites that must be in place. Finally, I’ll walk you through the important points of the Hyper-V and Failover Clustering configuration that must be performed to enable live migration.

How Live Migration Works

Live migration takes place between two Hyper-V hosts. Essentially, the VM memory is copied between Hyper-V hosts. After the memory is copied, the VM on the new host can access its virtual hard disk (VHD) files and continue to run. Both hosts access shared storage where the VM’s VHD files are stored. When you initiate a live migration, which Figure 1 shows, the following steps occur:

      1. A new VM configuration file is created on the target server.
      2. The source VM’s initial memory state is copied to the target.
      3. Changed memory pages on the source VM are tagged and copied to the target.
      4. This process continues until the number of changed pages is small.
      5. The VM is paused on the source node.
      6. The final memory state is copied from the source VM to the target.
      7. The VM is resumed on the target.
      8. An Address Resolution Protocol (ARP) is issued to update the network routing tables.

Requirements for Live Migration

On the hardware side, you need two x64 systems with compatible processors. It’s best if the host processors are identical, though it’s not required. However, they do need to be from the same processor manufacturer and family—you can’t perform a live migration when one host has an AMD processor and the other host has an Intel processor. Learn more about Hyper-V processor compatibility in the Microsoft white paper “Virtual Machine Processor Compatibility Mode.”
      In addition, each of the servers should be equipped with at least three NIC cards, running at 1GHz: one for external network connections, one for iSCSI storage connectivity, and one for node management. Ideally, you’d have another NIC dedicated to the live migration, but the live migration can also occur over the external network connection—it will just be a little slower. It’s important to note that if you’re implementing a server consolidation environment, you will want additional NICs for the network traffic of the VMs.
      On the software side, all the nodes that take part in live migration must have Server 2008 R2 x64 installed. This can be the Standard, Enterprise, or Datacenter editions. Live migration is also supported by the Hyper-V Server 2008 R2 product. In addition, the Hyper-V role and the Failover Cluster feature must be installed on all servers participating in live migration.
      You also need shared storage, which can be either an iSCSI SAN or a Fibre Channel SAN. In this example, I used an iSCSI SAN. Be aware that the iSCSI SAN must support the iSCSI-3 specifications, which includes the ability to create persistent reservations, something that live migration requires. Some open-source iSCSI targets such as OpenFiler don’t have that support at this time. If you’re looking to try this for a local test and don’t want to buy an expensive SAN, you might want to check out the free StarWind Server product at http://www.starwind.com/.

Failover Cluster Networking Configuration

Failover clustering is a requirement for live migration. You can live-migrate VMs only between the nodes in the failover cluster. The first step in creating a failover cluster is to configure the networking and storage. You can see an overview of the network configuration used to connect the Windows servers in the cluster to the external network and to the shared storage in Figure 2.

      In Figure 2 the servers are using the network with the subnet of 192.168.100.xxx for client connections. The iSCSI SAN is running on an entirely separate physical network which was configured using the 192.168.0.xxx IP addresses. You can use different values for either of these IP address ranges. I selected these values to more clearly differentiate between the two networks. Ideally, you would also have additional NICs for management and an optional live migration connection, but these aren’t strictly required. Live migration can work with a minimum of two NICs in each server.

Deploying Network Load Balancing (NLB) and Virtual Machines on Windows Server 2008 R2

In Hyper-V, the VM host prevents dynamic MAC address updates as an extra layer of security in the datacenter.  This is because the VM may have full administrator rights, yet it may be untrusted in the datacenter, for example when the VM hosting is provided by an independent hosting company.  In this scenario, we need to make sure that one VM cannot cause a DOS or information disclosure attack against another VM.  If a VM is able to spoof its MAC address, then it can spoof the MAC addresses of other VMs and impact other VMs on that host.  The physical switches have similar protections and it is up to the admin to enable that protection or not.

If you do not enable spoofing of MAC address prior to configuring NLB on the VM you could potentially have problems with the NLB cluster. 

When configuring NLB in unicast mode on Hyper-V with enable spoofing of MAC Address disabled you may see some of the following symptoms:

·         When initially configuring NLB you will lose network connectivity on the network adaptor NLB was configured on.

·         There will be an NLB error event in the Windows Event Log stating that the network adaptor does not support dynamic MAC address updates.

·         After rebooting the server, NLB will appear to be bound to the network adapter, but the cluster VIP will not have been added to the network adaptor.

·         The cluster MAC address will still be the original MAC address associated with the network adaptor prior to configuring NLB.   Use CMD> ipconfig /all to view the MAC address.  It should start with "02-BF-***"

·         If you ignore all previous symptoms and manually add the VIP you could get an IP conflict if there are other nodes in the cluster that have the same VIP. 

With that said, to allow VM guests to run NLB you need to set the VM property for "Enable spoofing of MAC Address". 

To enable spoofing of MAC Addresses open the Hyper-V management console.  Make sure the VM is stopped open the properties of the VM.  Select the Network Adaptor for the NLB VM and check the "Enable spoofing of MAC Address" and click OK.  Then start the VM. 

Virtual Machine (VM) Failover Policies

the behavior of the failure policies for a highly available Hyper-V virtual machine running on a Windows Server 2008 R2 Failover Cluster.For the most part, cluster failover policies are generic and apply to all workloads on a cluster, whether it is SQL, a File Server, etc.There is a wealth of documentation out there, but I thought I would take a focused view at understanding what influences where VMs go when a node

Default Failover Policies:

When there is a failure of a node, VMs are spread across the remaining cluster nodes and distributed to the nodes hosting the fewest number of VMs.For example, let’s say that NodeA crashes and it was hosting 10 VMs.The Cluster Service will take one of the VMs, it will then look across the surviving nodes and find the node currently hosting the fewest number of VMs (technically it looks at cluster Group ownership and selects the node with the lowest count).The VM is then started on that node.Then the next VM is selected, then again looks across the cluster again for the node now currently hosting the fewest VMs is selected and the VM is started there.This happens for all VMs until they are all placed.VMs will be distributed across the cluster to different nodes based on who is currently hosting the fewest VMs.

Advanced Failover Policies:

In general the default settings are the right choice for most people, and I recommend you stick with those.  However, Clustering does offer a wealth of granular options so you can fine tune the default behavior.

Other Influencers of Failover:

There are a few other settings worth discussing which can influence VM placement and behavior on failover.  Let’s discuss some of those:

·         Pause Node – At the server level, you can Pause a node.  When a node is paused, it means that no VMs (technically no cluster Group) can failover to that node.  If a node is paused, it will be removed from the possibility to be a failover target.  Pausing a node is handy when doing maintenance tasks like applying a patch, this prevents VMs from failing over to the node when you are doing something to it.

·         Disable Failover – For a given VM (technically any cluster Group) you can configure the “Auto Start” setting.  If auto start is disabled for a VM, it means that a VM will not be started on failover.  This could be useful if you have a low priority VM that you don’t necessarily want to failover, but you still want it to be clustered so that you can, for example, perform live migrations.

Startup Placement Policies

·         Persistent Mode – When a cluster as a whole is shut down and then restarted clustering will attempt to start VMs back on the last node they were hosted on.  This is controlled by the “Persistent Mode” setting, and is enabled by default.  The default amount of time the cluster service will wait for the original node to rejoin the cluster is 30 seconds; this is configurable via the cluster common property ClusterGroupWaitDelay.  You may choose to disable Persistent Mode for high priority VMs, where you do not want to wait for the original node to come back… just start the VM as soon as possible.

·         Possible Owners – For a given VM (technically any cluster Resource) you can configure the nodes which the VM has the possibility of failing over to.  By default it’s all nodes, but if you have a specific node you never want this VM to failover to you can remove it from being a possible owner and prevent it.

·         Preferred Owners – For a given VM (technically any cluster Group) you can configure the preference for node order on failover.  So let’s say that this VM normally runs on NodeA and you always want it next to go to NodeC if it is available, then preferred owners is a way for you to define a preference of first go to this node, then next go to this other node, then go to this next node.  It’s a priority list, and clustering will walk that list in where to place the VM.  This will override the default behavior of selecting the node currently hosting the least VMs I described above, and gives you explicit control of where VMs go.  More information about Preferred and Possible Owners: http://support.microsoft.com/kb/299631.

·         Anti-Affinity – For a given VM (technically any cluster Group) there is a cluster group property called AntiAffinityClassNames that allows you to configure the preference to attempt to keep that VM off the same node as other similar VMs.  Let’s say for example you have two domain controllers running in VMs.  It would probably be best to keep those running on different nodes if possible.  When determining failover, the cluster service will deprioritize any node which is hosting a similar VM.  If there is no other option (in the goal of making VMs available) it will place them on the same host.  More information: http://msdn.microsoft.com/en-us/library/aa369651(VS.85).aspx.

ESX vs. ESXi

Capability	VMware ESX	VMware ESXi
Service Console	Service Console is a standard Linux environment through which a user has privileged access to the VMware ESX kernel. This Linux-based privileged access allows you to manage your environment by installing agents and drivers and executing scripts and other Linux-environment code.	VMware ESXi is designed to make the server a computing appliance. Accordingly, VMware ESXi behaves more like firmware than traditional software. To provide hardware-like security and reliability, VMware ESXi does not support a privileged access environment like the Service Console for management of VMware ESXi. To enable interaction with agents, VMware has provisioned CIM Providers through which monitoring and management tasks – traditionally done through Service Console agents – can be performed. VMware has provided remote scripting environments such as vCLI and PowerCLI to allow the remote execution of scripts.
CLI-Based Configuration	VMware ESX Service Console has a host CLI through which VMware ESX can be configured. VMware ESX can also be configured using vSphere CLI (vCLI).	The vSphere CLI (vCLI) is a remote scripting environment that interacts with VMware ESXi hosts to enable host configuration through scripts or specific commands. It replicates nearly all the equivalent COS commands for configuring ESX. Notes:   vCLI is limited to read-only access for the free version of VMware ESXi. To enable full functionality of vCLI on a VMware ESXi host, the host must be licensed with vSphere Essentials, vSphere Essential Plus, vSphere Standard, vSphere Advanced, vSphere Enterprise, or vSphere Enterprise Plus.   VMware vSphere PowerCLI (for Windows) and vSphere SDK for Perl access ESXi through the same API as vCLI. Similarly, these toolkits are limited to read-only access for the free version of VMware ESXi. When the host is upgraded to vSphere Essentials, vSphere Essential Plus, vSphere Standard, vSphere Advanced, vSphere Enterprise, or vSphere Enterprise Plus these toolkits have write-access and provide a scriptable method for managing ESXi hosts. Certain COS commands have not been implemented in the vCLI because they pertain to the management of the COS itself and not ESXi. For details,
Scriptable Installation	VMware ESX supports scriptable installations through utilities like KickStart.	VMware ESXi Installable does not support scriptable installations in the manner ESX does, at this time. VMware ESXi does provide support for post installation configuration script using vCLI-based configuration scripts.
Boot from SAN	VMware ESX supports boot from SAN. Booting from SAN requires one dedicated LUN per server.	VMware ESXi may be deployed as an embedded hypervisor or installed on a hard disk.   In most enterprise settings, VMware ESXi is deployed as an embedded hypervisor directly on the server. This operational model does not require any local storage and no SAN booting is required because the hypervisor image is directly on the server. The installable version of VMware ESXi does not support booting from SAN.
Serial Cable Connectivity	VMware ESX supports interaction through direct-attached serial cable to the VMware ESX host.	VMware ESXi does not support interaction through direct-attached serial cable to the VMware ESXi host at this time.
SNMP	VMware ESX supports SNMP.	VMware ESXi supports SNMP when licensed with vSphere Essentials, vSphere Essential Plus, vSphere Standard, vSphere Advanced, vSphere Enterprise, or vSphere Enterprise Plus. The free version of VMware ESXi does not support SNMP.
Active Directory Integration	VMware ESX supports Active Directory integration through third-party agents installed on the Service Console.	VMware ESXi does not support Active Directory authentication of local users at this time.
HW Instrumentation	Service Console agents provide a range of HW instrumentation on VMware ESX.	VMware ESXi provides HW instrumentation through CIM Providers. Standards-based CIM Providers are distributed with all versions of VMware ESXi. VMware partners include their own proprietary CIM Providers in customized versions of VMware ESXi. These customized versions are available either from VMware’s web site or the partner’s web site, depending on the partner.    Remote console applications like Dell DRAC, HP iLO, IBM RSA, and FSC iRMC S2 are supported with ESXi.
Software Patches and Updates	VMware ESX software patches and upgrades behave like traditional Linux based patches and upgrades. The installation of a software patch or upgrade may require multiple system boots as the patch or upgrade may have dependencies on previous patches or upgrades.	VMware ESXi patches and updates behave like firmware patches and updates. Any given patch or update is all-inclusive of previous patches and updates. That is, installing patch version “n” includes all updates included in patch versions n-1, n-2, and so forth.  Furthermore, third party components such as OEM CIM providers can be updated independently of the base ESXi component, and vice versa.
VI Web Access	VMware ESX supports managing your virtual machines through VI Web Access. You can use the VI Web Access to connect directly to the ESX host or to the VMware Infrastructure Client.	VMware ESXi does not support web access at this time.
Licensing	For licensing information, see the VMware Sphere Editions Comparison.	For licensing information, see the VMware Sphere Editions Comparison.
Diagnostics and Troubleshooting	VMware ESX Service Console can be used to issue commands that can help diagnose and repair support issues with the server.	VMware ESXi has several ways to enable support of the product: Remote command sets such as the vCLI include diagnostic commands such as vmkfstools, resxtop, and vmware-cmd. The console interface of VMware ESXi (known as the DCUI or Direct Console User Interface) has functionality to help repair the system, including restarting of all management agents. Tech Support Mode, which allows low-level access to the system so that advanced diagnostic commands can be issues. For more information,
Jumbo Frames	VMware ESX 4.0 fully supports Jumbo Frames.	VMware ESXi 4.0 fully supports Jumbo Frames.

Active Directory Federation Services (ADFS)

Federation Services were originally introduced in Windows Server 2003 R2. F provides an identity access solution, and AD Federation Services provides authenticated access to users inside (and outside) an organization to publicly (via the Internet) accessible applications. Federation Services provides an identity management solution that interoperates with WS-* Web Services Architecture–enabled security products. WS-Federation Passive Requestor Profile (WS-F PRP) also makes it possible for federation to work with solutions that do not use the Microsoft standard of identity management. The WS-Federation specification defines an integrated model for federating identity, authentication, and authorization across different trust realms and protocols. This specification defines how the WS-Federation model is applied to passive requestors such as Web browsers that support the HTTP protocol. WS-Federation Passive Requestor Profile was created in conjunction with some pretty large companies, including IBM, BEA Systems, Microsoft, VeriSign, and RSA Security.

What Is Federation?

As we described earlier in this chapter, federation is a technology solution that makes it possible for two entities to collaborate in a variety of ways. When servers are deployed in multiple organizations for federation, it is possible for corporations to share resources and account management in a trusted manner. Earlier in this chapter, we were discussing Active Directory Rights Management Server. This is just one way companies can take advantage of FS. With ADFS, partners can include external third parties, other departments, or subsidiaries in the same organization.

Why and When to Use Federation

Federation can be used in multiple ways. One product that has been using federation for quite some time is Microsoft Communication Server (previously, Live Communication Server 2005, now rebranded as Office Communication Server 2007). Federation is slightly different in this model, where two companies can federate their environments for the purposes of sharing presence information. This makes it possible for two companies to securely communicate via IM, Live Meeting, Voice, and Video. It also makes it possible to add “presence awareness” to many applications, including the Office suite, as well as Office SharePoint Server. If you want to know more about OCS and how federation works for presence, we recommend How to Cheat at Administering Office Communication Server 2007, also by Elsevier.

A little closer to home, Federation Services can also be used in a variety of ways. Let’s take an extranet solution where a company in the financial service business shares information with its partners. The company hosts a Windows SharePoint Services (WSS) site in their DMZ for the purposes of sharing revenue information with investment companies that sell their products. Prior to Active Directory Federation Services, these partners would be required to use a customer ID and password in order to access this data. For years, technology companies have been touting the ability to provide and use single sign-on (SSO) solutions. These worked great inside an organization, where you may have several different systems (Active Directory, IBM Tivoli, and Solaris), but tend to fail once you get outside the enterprise walls.

With AD FS, this company can federate their DMZ domain (or, their internal AD) with their partner Active Directory infrastructures. Now, rather than creating a username and password for employees at these partners, they can simply add the users (or groups) to the appropriate security groups in their own Active Directory (see Figure 1). It is also important to note that AD FS requires either Windows Server 2008 Enterprise edition or Datacenter edition.

Figure 1. The Active Directory Federation Services Structure

Configuring ADFS

In this exercise, we are going to create the account side of the ADFS structure. The resource is the other half of the ADFS configuration, which is the provider of the service that will be provided to an account domain. To put it in real-world terms, the resource would provide the extranet application to the partner company (the account domain).

Configuring Federation Services

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

1.	Click Start | Administrative Tools | Server Manager.
2.	Scroll down to Role Summary, and then click Add Roles.
3.	When the Before You Begin page opens, click Next.
4.	On the Select Server Roles page, select Active Directory Federation Services (see Figure 2) from the list and click Next. Figure 2. Selecting the Role
5.	Click Next on the Active Directory Federation Services page.
6.	In the Select Role Services window, select Federation Service, and then click Next. If prompted, add the additional prerequisite applications.
7.	Click Create A Self-Signed Certificate For SSL Encryption (Figure 3), and then click Next. Figure 3. Creating a Self-Signed Token-Signing Certificate
8.	Click Create A Self-Signed Token-Signing Certificate, and then click Next.
9.	Click Next on the Select Trust Policy page.
10.	If prompted, click Next on the Web Server (IIS) page.
11.	If prompted, click Next on the Select Role Services page.
12.	On the Confirm Installation Selections page, click Install.
13.	When the installation is complete, click Close.

The next step in configuring AD FS is to configure IIS to require SSL certificates on the Federation server:

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

1.	Choose Start | Administrative Tools | Internet Information Services (IIS) Manager.
2.	Double-click the server name.
3.	Drill down the left pane to the Default Web Site and double-click it.
4.	Double-click SSL Settings and select Require SSL.
5.	Go to Client Certificates and click Accept. Then, click Apply (Figure 4). Figure 4. Requiring Client Certificates
6.	Click Application Pools.
7.	Right-click AD FS AppPool, and click Set Application Pool Defaults.
8.	In the Identity pane (Figure 5), click LocalSystem, and then click OK. Figure 5. Setting Application Pool Defaults
9.	Click OK again.
10.	Before we close IIS, we need to create a self-signed certificate. Double-click the server name again.
11.	Double-click Server Certificates.
12.	Click Create Self-Signed Certificate.
13.	In the Specify Friendly Name field, enter the NetBIOS name of the server and click OK.

Next, we need to configure a resource for use with AD FS. In this case, we are going to use the same domain controller to double as a Web server. What we will be doing is installing the AD FS Web Agent, essentially adding an additional role to the server, as part of the AD FS architecture. This will allow us to use our federated services within a Web application.

<><><>

1.	Choose Start | Administrative Tools | Server Manager. Scroll down to Role Summary, and then click Add Roles.
2.	When the Before You Begin page opens, click Active Directory Federation Services.
3.	Scroll down to Role Services and click Add Role Services.
4.	In the Select Role Services window, select Claims-aware Agent (Figure 6), and then click Next. Figure 6. Setting Services
5.	Confirm the installation selections (Figure 7), and then click Install. Figure 7. Confirming the Installation
6.	When installation is complete, click Close.

Now we need to configure the trust policy which would be responsible for federation with the resource domain.

1.	Choose Start | Administrative Tools | Active Directory Federation Services.
2.	Expand Federation Service by clicking the + symbol (see Figure 8). Figure 8. AD FS MMC
3.	Right-click Trust Policy, and then choose Properties.
4.	Verify the information in Figure 9 matches your configuration (with the exception of the FQDN server name), and then click OK. Figure 9. Trust Policies
5.	When you return to the AD FS MMC, expand Trust Policy and open My Organization.
6.	Right-click Organization Claims, and then click New | Organization Claim.
7.	This is where you enter the information about the resource domain. A claim is a statement made by both partners and is used for authentication within applications. We will be using a Group Claim, which indicates membership in a group or role. Groups would generally follow business groups, such as accounting and IT.
8.	Enter a claim name (we will use PrepGuide Claim). Verify that Group Claim is checked as well before clicking OK.
9.	Create a new account store. Account stores are used by AD FS to log on users and extract claims for those users. AD FS supports two types of account stores: Active Directory Domain Services (AD DS) and Active Directory Lightweight Directory Services (AD LDS). This makes it possible to provide AD FS for full Active Directory Domains and AD LDS domains.
10.	Right-click Account Store and choose New | Account Store.
11.	When the Welcome window opens, click Next.
12.	Since we have a full AD DS in place, select Active Directory Domain Services (AD DS) from the Account Store Type window (Figure 10), and then click Next. Figure 10. The Account Store Type Window
13.	Click Next on the Enable This Account Store window.
14.	Click Finish on the completion page.

Now, we need to add Active Directory groups into the Account Store.

1.	Expand Account Stores.
2.	Right-click Active Directory, and then click New | Group Claim Extraction.
3.	In the Create A New Group Claim Extraction window (Figure 11), click Add and click Advanced. Figure 11. The Create A New Group Claim Extraction Window
4.	Click Object Types, remove the checkmarks from everything except Groups, and then click OK.
5.	Click Find Now.
6.	Select Domain Admins from the list of groups by double-clicking.
7.	Click OK.
8.	The Map To This Organization Claim field should show the claim we created earlier. Click OK to close the window.

Finally, we will work to create the partner information of our resource partner, which is prepguides.ads.

1.	Expand Partner Organizations.
2.	Right-click Resource Partners, and then select New | Resource Partner.
3.	Click Next on the Welcome window.
4.	We will not be importing a policy file, so click Next.
5.	In the Resource Partner Details window (Figure 12), enter a friendly name for the partner, and the URI and URL information of the partner. Note it is identical to what we entered earlier in Figure 9. When the information is complete, click Next. Figure 12. Resource Partner Details
6.	Click Next on the Federation Scenario page. This is the default selection, which is used for two partners from different organizations when there’s no forest trust.
7.	On the Resource Partner Identity Claims page, check UPN Claim and click Next. A UPN Claim is based on the domain name of your Active Directory structure. In our case, the UPN is uccentral.ads.
8.	Set the UPN suffix. Verify that Replace All UPN Suffixes With The Following: is selected and then enter your server’s domain name. This is how all suffixes will be sent to the resource partner. Click Next.
9.	Click Next to enable the partner.
10.	Click Finish to close the wizard.

We’re almost at the end of our account partner configuration. The last thing we need to do is create an outgoing claim mapping. This is part of a claim set. On the resource side, we would create an identical incoming claim mapping.

1.	Expand Resource Partners.
2.	Right-click your resource partner, and then choose New | Outgoing Group Claim Mapping.
3.	Select the claim we created earlier, enter PrepGuide Mapping, and then click OK.