Let me tell you first some background info: we set up a new application a few months ago with a somewhat complex architecture deployed on a two host VMware cluster (several Linux web application servers published through a Linux load balancer). Since a few weeks ago the application owners were complaining that copying files through SSH/SCP was extremely slow through the WAN from one server.

At first, I thought the cause was some MTU issue due to restricted firewall configurations or some problem with the VM or custom gateway because between the servers in the same VLAN everything was ok. Then the users found out that other servers were having the same problem and all were on the same host. So we started to do some packet captures to see what was going on (click on the pictures to see their full width):

So it seemed that the gateway was receiving packets larger than 1500 with DF bit set and the app server ignored the ICMP messages and kept on sending non fragmentable large messages! At first I didn’t understand what could be going on. We tried uninstalling VMware Tools and it magically started working… Installed them back, stopped working. During the Linux boot process I noticed something about activating TCP Segmentation offload. Disabled this using ethtool -K eth0 tso off, and BINGO! ssh copying started to work!

So it seems that VMware enhanced driver optimizes intrahost communication by sending the whole tcp segment to the next virtual machine but ignores the possibility of it being a router and the packet marked as non-fragmentable! We are taking this to VMware support, I don’t think it should be happening like this.

Update: VMware support responded with this: http://kb.vmware.com/kb/1010939
So it’s a known issue .

So I configured Mem.AllocGuestLargePage to 0 on some hosts, rebooted and voila, we have Transparent Page Sharing again (Memory Granted-Memory Consumed=20-25GB, cool ).

After this I tried to measure the performance loss using 7-Zip, a simple benchmark that stresses memory and CPU but I could not find any measurable difference between hosts using this setting and hosts using default settings (I did find some minor differences but it was actually faster when Mem.AllocGuestLargePage was disabled, probably because of differences in number of DIMM slots occupied – one host had 12x4GB DIMMS, another one 6x8GB DIMMS or Power Management settings or other activities on the hosts). I tried both 32 bit and 64 bit operating systems, even Windows 2008 but nothing emerged. The performance gain because of lower memory usage and no more ballooning and swaping on the other hand is definitely measurable .

So go ahead and test and activate this setting, it will save you tons of RAM!

For more information about this setting, check out http://www.vmware.com/files/pdf/large_pg_performance.pdf

I decided to use VMWare Update Manager to help upgrade the virtual machines like this:

1. In the Update Manager VM View I created a baseline group that contained both VMWare tools upgrade and the VM Hardware upgrade baselines
2. In the VMs and Templates I selected the Datacenter and in the Update Manager tab I Scanned for VM Hardware and Tools upgrades to check that nothing is needed before we move the VMs
3. I shut down the VMs en masse (not too many at a time so that we don’t overwork the host and storage)
4. When they are down I migrate them to the new cluster (I chose this approache so that I don’t enable Enhanced VMotion Compatibility as all new servers will run with Nehalem CPUs) and power them on
5. After I waited for them to up properly I scanned again for VM updates in the datacenter tab (this was needed because the VMs are scattered around in a quite elaborate folder structure)
6. I then clicked Remediate and checked that only the right VMs are in the list to be upgraded and finished the wizard without creating a snapshot (no need to snapshot VM Tools upgrade)
7. After the task finished I let everything settle down a few minutes and then:
   1. Connected on all Windows 2008 machines to make online the offline disks (see http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013109 ). I did this manuall because we had only a few Win 2008 Ent machines
   2. Rebooted all the Virtual Machines once more (after the VM Hardware upgrade Windows redetects devices and asks for a reboot; dynamic DNS records are also not registered properly)

So now I have a new beautiful working virtual infrastructure, yuppiee!

The performance benefit of this upgrade is beyond expectations, CPU Ready% got to almost 0 and CPU Usage went to almost half on the most used machines – see the following graphs for 4 various Virtual Machines running SQL, an Oracle Monitoring app or System Center Configuration Manager:

There was a downside to all this though : due to EPT features in Nehalem I get much less Transparent Page Sharing and thus much more memory usage – see http://www.vmware.com/files/pdf/large_pg_performance.pdf . I probably will disable the allocation of large pages to guests on some hosts that run less important VMs.

In conclusion I like vSphere a lot, the upgrade to 4.0 was MUCH easier than the upgrade to 3.0, thank you VMWare for this fine product! Storage VMotion rules, it’s so easy now to move VMs around datastores!

I decided that the hosts will have fresh installs and form a new cluster since we will replace all our existing hosts with  new hardware (I see no reason to do upgrade to the hosts, it’s cleaner and safer to just reinstall, maybe only if you have hundreds of hosts – but we only had 4).

1. I made sure the hosts were running the latest firmware by checking the HP site
2. I prepared an installation script customized for each server with the IP address (it’s a pain to change it after the server is installed). I received inspiration from here.
3. I copied the content of the ESX CD to a subfolder on the WebServer (such as esx4.0)
4. I entered BIOS setup and made sure all virtualization options were active and changed in Power Management Options HP Power Regulator to OS Control Mode so that ESX can control processor frequency and manage power savings
5. Then I booted the ESX CD and chose ESX Scripted Install using USB ks.cfg, pressed F2 and changed ks=USB to ks=http://__WEBSERVER__/ks__ServerName__.cfg . Here is the script template I used. I configured the installation to download the files from the WebServer because I used iLo to boot the ESX CD which is very sloooowww. By using the webserver the installation went much faster. If you already configured VLANs trunking on the network ports you need to specify the VLAN that will be used to access the webserver by passing to the kernel at boot also vlanid=xxx.

1. If you put the ks.cfg and ESX CD on an IIS server you need to register the cfg (text/plain) and pkl (application/octet-stream) to MIME Types under HTTP Headers int the site properties otherwise it won’t be visible. Also you need to give NTFS read rights to the IIS anonymous user

1. After reboot I logged in at the console and changed the root password with the passwd command
2. I then logged in with vSphere client and checked the installation
3. Checked the time on the host and made sure NTP service was started (Configuration – Time Configuration) – I couldn’t get the install script to properly configure NTP during the installation.
4. Enabled dynamic PowerCPUPolicy to activate power savings (Configuration – Advanced Settings – Power – Power.CPUPolicy)
5. Time to join vCenter!
6. Networking configuration:
7. 1. We are using blades so we are limited to two network cards per blade (it would cost too much to buy two additional switches and NICs for each blade and our VMs don’t do that much traffic to justify the investment – I see no more than 100Mbps of traffic). In order to separate VM traffic from Kernel and Console traffic but still have redundancy built-in I made the following setup:
   • I created a single Virtual Switch with both network cards connected to all VLANs
   • I configured vmnic1 as active and vmnic0 as standby adapter
   • Then, for VMKernel and Service Console portgroups, I configured to override default NIC Teaming and configured the other way around (vmnic0 active and vmnic1 standby)
   • Using this setup VMotion traffic goes through vmnic0 and VM traffic through vmnic1 if both nics are ok and when one nic fails I have no interruption

   

8. 
   1. Created the VMKernel portgroup and enabled VMotion
   2. Used a nice PowerCLI script to create the other portgroups from CSV – here it is
   3. Storage configuration
      1. I zoned the new hosts In the FC Switches (I use single initiator policy so for each HBA port I created a new zone with all the storage controllers)
      2. Added the hosts in EVA CommandView
      3. In order to give access to all the LUNs easily I used the EVA Storage Scripting utility (sssu):
         1. i.      I connected to the CommandView server
         2. ii.      I selected the EVA system (select sys EVA_Name)
         3. iii.       I exported the current configuration (capture conf file_name)
         4. iv.      Edited the file to contain only the ADD LUN commands for one host and replaced the host name with the new name
         5. v.      Copied the commands from the file and pasted them in the SSSU
9. Rescanned for new storage (Configuration – Storage Adapters – Rescan)
10. Installing the latest updates
    1. In Hosts view I selected the update manager tab
    2. I attached all the updates baselines, scanned the host and clicked Remediate
    3. After reboot I found out that the build number did not change so I started investigating. It turned out to be normal:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1012514

Output of rpm -qa | grep -i vmware:

Output of esxupdate query

Cool, everything got installed properly, host is ready for VMs!

This is a series of posts where I will tell how I upgraded my environment to ESX 4.0.

Happy reading!

Goals

1. Migrate our current environment consisting of four overworked ESX 3.5 U4 hosts running 111 active VMs to ten vSphere 4.0 U1 shiny new G6 Nehalem wonderful goodies
2. Modify as little as possible from defaults – „Everything should be made as simple as possible, but not simpler.”
3. Don’t screw up too many things during the process

Plan from 10.000ft

1. Upgrade Virtual Center
2. Install vSphere on the hosts and connect them to vCenter
3. Migrate Virtual Machines

Sounds easy, doesn’t it?

Background

The current environment consists of 4 ESX 3.5 U4 hosts running on HP Bl460 G5. VirtualCenter runs as a Win 2003 virtual machine and connects to a separate physical SQL 2005 SP3 server. Storage is hosted on two HP EVA systems connected through 4Gb Fibre Channel.

In order to simplify the SAN I chose to configure the blade chassis switches in Access Gateway mode (in this mode they use NPIV to present all the blades on the master switch on a single port and the blade switch is transparent). Using this feature I have only one real Fibre Channel switch where I setup zoning and the others are just “port extenders”. For zoning I chose the “single initiator” strategy where I create a single zone for each server with all the storage ports included on each redundant switch (I tried at first to create a zone for each storage port and each server but the configuration that would result would be extremely difficult to manage and as far as I found on the internet there is no downside to my strategy).

Upgrading Virtual Center

1. I backed-up the VirtualCenter database like  good boy
2. I installed the new version of the vSphere client on my machine
3. In Virtual Center client I renamed the old virtual machine so that I could deploy a shiny new Windows 2008 64bit virtual machine with the old name
4. I gave the new VM 3GB of RAM, 2vCPUs and two 20GB disks (minimum supported requirements as we were already oversubscribed with resources)
5. I made a note of the host where the new and old Virtual Center VMs were running
6. I forgot to copy the SSL files from the old VM C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\SSL to a safe location before going to the next step so don’t you do like me ! Also do copy the sysprep files.
7. Stopped the old VM and recreated its account in Active Directory
8. Connected directly to the host of the new VM with the vSphere client
9. Configured the operating system of the new VM and joined the domain
10. I then installed the 64bit SQL native client for both SQL 2008 and SQL 2005 as we will migrate the SQL to 2008 in the future and configured the ODBC datasource  by running c:\Windows\SysWOW64\odbcad32.exe and selecting the SQL Native Driver (choose version 10 if you have SQL2008) and SQL authentication with the old username (you do remember it, don’t you?). You can give any name to the datasource.
11. Now I copied the SSL files in C:\Users\All Users\VMware\VMware VirtualCenter\SSL . I couldn’t browse the folder using Explorer so I started PowerShell as Administrator and issued a mkdir for the folder and then copied the files using the copy command.
12. To make things upgrade faster I configured Bulk Logged recovery model in the database options in SQL Management Console

1. Started the vCenter setup and sailed through the installation choosing to run the vCenter as SYSTEM, otherwise I left the defaults. The installation took about 16 minutes. When it finished it upgraded the vpx clients on all the hosts without problems (this happened very quickly, less than a minute).
2. Started the vCenter client and saw no issues – yuppiee!
3. Actually there were some . It lost the 3.5 Licensing server. I put it back but I still had no VMotion. After googling a little I disabled and enabled back VMotion on the VMKernel network port group – this solved it. Also the permissions we assigned on one folder of virtual machines were gone – you have been warned .

1. Copied back the sysprep files from where I saved them
2. Upgraded update manager and Converter and installed the plugins on my Admin PC
3. Activated back Full Recovery Logging on the database

I started this blog to document my implementation of VMWare so that others will not make the same mistakes I did .

I hope you will find my experience useful, I wish you happy reading!