I inherited a bunch of virtualized Citrix Presentation 4 servers virtualized on XenServer 4.1, and had to start the new farm. Since the XenServer 4.1 had memory leaks, and the hosts had to be rebooted periodically, sometimes in the middle of the day, we wanted to build the new XenApp farm on the latest, most stable brand of Xen available from Citrix. Xen Server 5 had been around for a while, and was up to “Update 3” on Citrix’s website, but there was also a “XenServer 5.5” release, and there was a new format for the virtual machines. In the test lab, I downloaded the latest, but ran right into problems: my network connectivity was almost un-useable. A quick look at the forums that first weekend, and apparently the XenTools NIC driver was the problem – we had to dump the tools and load a NIC driver to get the server built. I made a quick call to Citrix the next Monday morning, and they told me that if you want something stable, stick with XenServer 5. So I downloaded XenServer 5 Update 3 ISO and set out to build the latest standard.
I had an extra host, so I didn’t have to do an in-place upgrade. The plan was to build a new XenServer 5 host, and migrate the existing vm’s over, then eventually build some new vm’s and start a new Citrix XenApp farm.
The first catch in taking a server with XenServer 4.1 and going to XenServer 5, is that the NICs are “re-ordered”; I had been careful to document the previous configuration: there were onboard and add-in NIC cards. The Service console network was connected to the first on-board NIC, which had been labeled nic4, after 0-3 were taken up by the add-in card, so I had carefully picked “eth4” during my Xen5 install as where to configure the service console. After reboot I had no connectivity. Turns out the Xen5 install orders the NICs beginning with the on-board NIC’s, and so the same NIC I needed configured to the Service Console network was now labeled “eth0”
The way we figured that out was using the command line utilities built into XenServer, specifically the “xe pif-list” & “xe pif-param-list uuid=<pif uuid>”, to determine which pif corresponds to nic to be used as mgmt console, then “xe pif-reconfigure-ip” to change ip settings, and xe host-management-reconfigure pif-uuid=<pif uuid>”, or you can just use the text console that comes with XenServer5 and up:
The second obstacle I ran into was with the storage. We had a brand-new chunk of SAN to allocate, and at first I had been given the whole “chunk” as one big LUN, the whole 1777GB. I was getting “SR- Backend Failure”. This is another task that can be done in the sonsole, but if you are getting errors you might want to see more detail.
At the command shell prompt:
xe sr-probe device-config:SCSIid
got the SCSI id of the lun: 36008 05f300 1a905 0a7dc 818da b8b0075
got the uuid of the host: c62c95c7-5f41-4240-a6e7-c2e8baa82766
xe sr-create host-uuid=<host uuid> content-type=LVM name-label=LVMHBA shared=true device-config:SCSIid=<scsi id>
but got error, “SR_BACKEND_FAILURE_55”
I tried “fdisk /dev/sda” and changed type. There was no “partprobe” command available (as there is in RedHat), so I rebooted to update the kernel. On reboot, many errors at first: “Buffer I/O error on device”. I still got error 55, but now I had a clue, and I wasn’t surprised: the LUN was too big. In VMware the Best Practice is 500GB LUNs; after I asked the SAN administrator for a 500GB LUN, everything worked, right through the console.
Once we had our Xen5 pool configured, the migration task was pretty straight forward. Within the pool we can move servers from one host to another, by changing the “home server” and rebooting the vm. But between pools, between versions of Xen, there is no such utility. Instead, we just used the “export as backup” feature, exporting each vm as a “vxa” file, then going to the new Xen5 farm and importing the xva file. The result is a new copy of the old vm. We rename our old vm’s with the appendage “ – do –not – start”, then we start the new copy on the Xen5 host. The only modification needed is the XenTools from the Xen41 host need to be removed manually, in Add/remove programs, then the new XenTools on Xen5 need to be installed, and after getting a new NIC driver, then static IP settings had to be re-added.
After we’d migrated a few of these, I was shutting down a vm in off-hours to be the next one to go over to Xen5, and when it was shut down, it disappeared off the Xen Console; with no object that is in “down” state to right-click, I couldn’t migrate.
The “xe” command at the console – “xe vm-start name-label=cai-ctx08” – came in handy, to restart the vm, so that I could change the “home server” in the properties to one of the servers that was still up in the farm, then migrate it out after powering it down again.