XenServer 5.6 FP1 VDI Issue

Recently I ran into situation where one of my production VM's  went unresponsive,  the VM console showed various VDI related (xvda) I/O errors and the machine was halted.

It's worth mentioning that my XenServers (5.6 FP1) nodes operate in pool mode, and the problematic VDI resided on iSCSI LUN which seemed to be OK.

There weren't much other options rather than shutting down the VM with:

#xe vm-shutdown vm=vm0001 force=true

However, when I've tried to power on the VM back I got this nasty error:
18-Mar-12 9:42:16 AM Error: Starting VM 'vm0001' - Internal error: Failure("The VDI e17e2406-dbe9-40f6-98c3-af470e8aa91b is already attached in RW mode; it can't be attached in RO mode!")

Here is the workaround which did the job for me:

1. Find the UUID of the Storage Repository and the VM problematic VDI.

#xe sr-list |grep -i -C2 'your LUN name'
#xe vdi-list |grep -i -C2 'vdi name'

2. Next, we need to to remove VDI from the listing:

 #xe vdi-forget uuid=

Do not worry about the contents of the VDI they are fine :)

Verify the VDI is indeed gone:

#xe vdi-list |grep -i 'vdi name'

3. It's time to re-scan the storage repository that hosts the VDI via:
# xe sr-scan sr-uuid=

4. Verify the VDI is back in the listing:
#xe vdi-list sr-uuid=

Please note that the "name" and "description" fields are now empty.

5. Use XenCenter to reattach the VDI to your VM , and start it on different XenServer host inside your pool (right click on the VM, select "storage"->"attach"->).


This should do the magic.


Cheers!

Howto resize XenServer LUN's Online

In the following procedure I will show how to extend iSCSI attached LUN's on XenServer (v 5.6 SP2 in my case) on the fly ,so no service restart or downtime are needed and the VDI's that reside on the resized LUN are not affected, read on!

First of all on the storage side (NetApp filer in my case)list the available LUN's, the LUN I want to resize is  called "my_lun30", with LUN ID of 22, which is currently allocated with 90GB.

For the test I will extend it by another 10GB, making it a 100GB LUN.

filer1> lun show
        /vol/vol1/my_lun10         300.0g (322163441664)  (r/w, online, mapped)
        /vol/vol2/my_lun20    200g (214748364800)  (r/w, online, mapped)
        /vol/vol2/my_lun30       90.0g (107388862464)  (r/w, online, mapped)


filer1> lun resize /vol/vol2/my_lun30 +10g
lun resize: resized to:  100.0g (107388862464)


As you can see, my_lun30 is now 100GB:

filer1> lun show
        /vol/vol1/my_lun10         300.0g (322163441664)  (r/w, online, mapped)
        /vol/vol2/my_lun20    200g (214748364800)  (r/w, online, mapped)
        /vol/vol2/my_lun30        100.0g (107388862464)  (r/w, online, mapped)

We are done with the storage side, let's head to the XenServer side.
 In case you work in pool mode, login to the pool master as root:

I suggest installing "lsscsi" which provides a nice way of viewing SCSI attached disks/LUNs:

[root@xen3]#yum install lsscsi -y

[root@xen3]# lsscsi
[0:0:0:0]    cd/dvd  Optiarc  DVD RW AD-7561S  AH52  /dev/scd0
[2:0:0:0]    disk    NETAPP   LUN              7340  /dev/sda
[2:0:0:22]   disk    NETAPP   LUN              7340  /dev/sdb
[2:0:0:33]   disk    NETAPP   LUN              7340  /dev/sdc
[3:0:0:1]    disk    NETAPP   LUN              7340  /dev/sdd
[3:0:0:3]    disk    NETAPP   LUN              7340  /dev/sde

If YUM traffic to a repository is blocked - It's also possible to see the SCSI id's under /proc via:

# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: Optiarc  Model: DVD RW AD-7561S  Rev: AH52
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 22
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 33
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 01
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 03
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04

From the output we can see the SCSI ID of the LUN + it's corresponding device on the system, and check the device physical size, as you can see it is not updated yet:

[root@xen3 backup_scripts]# fdisk -l /dev/sdb

Disk /dev/sdb: 96.6 GB, 96647249920 bytes
255 heads, 63 sectors/track, 11750 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Since XenServer uses LVM, we can see the physical volume info of the disk, as you can see the size is still 90GB.
  
[root@xen3]# pvdisplay /dev/sdb
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               VG_XenStorage-e71389d1-4dc3-2518-aa30-9f5f0c70ba12
  PV Size               90.01 GB / not usable 6.12 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              23039
  Free PE               16624
  Allocated PE          6415
  PV UUID               oSZlnA-VxlA-3qbp-07nI-Ql0b-3cG2-9wo2lC


Now, we will tell XenServer to rescan the SCSI bus, we will provide the SCSI id which we previously got from the "lsscsi" command (2:0:0:22):

[root@xen3]# echo 1 > /sys/class/scsi_disk/2:0:0:22/device/rescan

You can notice the immediate change of /dev/sdb:

[root@xen3]# fdisk -l /dev/sdb

Disk /dev/sdb: 107.3 GB, 107388862464 bytes
255 heads, 63 sectors/track, 13055 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Now, resize the physical volume with:

[root@xen3]# pvresize /dev/sdb

  Physical volume "/dev/sdb" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Check again the physical volume size, notice the change:

[root@xen3]#  pvdisplay /dev/sdb
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               VG_XenStorage-e71389d1-4dc3-2518-aa30-9f5f0c70ba12
  PV Size               100.01 GB / not usable 6.12 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              25600
  Free PE               19185
  Allocated PE          6415
  PV UUID               oSZlnA-VxlA-3qbp-07nI-Ql0b-3cG2-9wo2lC

Let's get the SR uuid, I know the LUN ID is 22, so:

[root@xen3]# xe sr-list|grep lun22 -B1
uuid ( RO)                : e71389d1-4dc3-2518-aa30-9f5f0c70ba12
          name-label ( RW): iSCSI_filer1_lun22

Notice that the SR size has yet to be updated:

[root@xen3]#  xe sr-param-list uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12|grep physical-size
           physical-size ( RO): 96632569856

Now, finally update the relevant SR:

[root@xen3]#  xe sr-update uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12

And at last, the SR is updated with the correct new LUN size :

[root@xen3]#  xe sr-param-list uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12|grep physical-size
           physical-size ( RO): 107374182400


You are done!

XenServer - Cannot Shut VM

Have you ever experienced  a situation where you could not shut one of your running XenServer running VM's?   In this brief article I'll present a workaround for this common problem...  From my personal experiences this problem is often caused when the domain of the specific VM is still active (this often happens as a result of inproper shutdown and/or storage cutoffs) - in such case the domain needs to be destroyed manually.  OK, so here's the problem - from the XenServer (v 5.6 SP1) dom0 CLI I'm trying to shut a VM (using the force option): #xe vm-reset-powerstate vm=xxxxxxx force=true
 However the output I get is:  The operation could not be performed because a domain still exists for the specified VM.
vm: f087dxxxxx (xxxxxx)
domid: 5  From the output we will need  two parameters :the domain id (5 in our case) and the UUID of the problematic VM , so write them down somewhere. Now we need to destroy the currently active domain, XenServer includes couple of good tools under /opt/xensource, the command that is relevant is:   #/opt/xensource/debug/destroy_domain -domid 5  Now, finally, we will be able shut the VM with:  #xe vm-reboot uuid=XXXX --force  And we're done.  Hope that helped some one out there. Cheers!

XenServer Backup Script

This handy bash script I wrote, enables Xen Server Administrators to create On-Line backups (Importing the VM into.xva file) while the backed up virtual machine is up and running.

I found this extremely useful for big production environments - where lots of VM's require an appropriate & efficient back-up solution and downtime is not an option.

The script is very simple and user friendly, it's executed as root from the dom0 machine and used as follows:

In order to list all the available VM's on the system:

#./xen_onlinebackup.sh -l

In order to back up a VM:

#./xen_onlinebackup.sh -p /path/to/backupdir -s vm-name

The script supports both interactive and unattended mode (-u flag), the later designed to be built into scripts as an automation solution for large production environments.

For any help/usage, just run:

#./xen_onlinebackup.sh -h

Tested succesefully on XenServer 5.6 FP1 & XenServer 5.6 SP2.

Any comments, feed-backs or suggestions for improvements are very welcomed.

 #!/bin/bash

#

#This script used to create XenServer VM's online backup #Importing the VM into .xva file #Written by Paul Podolny - July 2011

#Variables

COUNT=0

AUTO=0 #Used for unattended mode

VMLIST=`xe vm-list is-control-domain=false |grep -i name-label|awk -F": " '{print $2}'`

#Functions

list() {

echo "====================================="

echo "Listing available VM's on $HOSTNAME..."

echo "====================================="

for i in $VMLIST;do

      echo $i;let COUNT+=1

done

echo "================================="

echo "Total $COUNT Virtual Machines"

echo "================================="

exit 0

}

usage() {

cat << EOF

This script will backup your Xen VMs on-line:

------

Usage:

------

-h Help/Usage.

-l List VM's.

-u Run in unattended mode (No questions asked).

-s Server Name (Backed Up VM).

-p Path to backup.

--------

Example:

--------

$0 -s server01 -p /nfs/backupdir/ -u

EOF

exit 0

}

check_dir() {

echo "=============================================================="

echo "XenServer Online Backup"

echo "=============================================================="

if  [ -e ${BACKUPPATH}/${VMNAME} ];then

        echo "Directory ${VMNAME} on path ${BACKUPPATH} seem to exist, will use it."

else

      echo "Creating dir. ${VMNAME} on backup path: ${BACKUPPATH}"

      mkdir ${BACKUPPATH}/${VMNAME}

fi

echo "`date +%H:%M` Backing Up ${VMNAME}, please hold on...Do NOT interrupt with CTRL+C !!!"

echo "================================================="

}

################################

#MAIN Prog

################################

while getopts "s:p:lhu" flag ; do

        case $flag in

                l ) list;;

                s ) VMNAME=$OPTARG;;

                p)  BACKUPPATH=$OPTARG;;

                h ) usage;;

                u)  AUTO=1;;

                *)  usage;;

        esac

done

#Sanity check  - Make sure backup path + VM name specificed if [ -z ${VMNAME} ] || [ -z ${BACKUPPATH} ];then echo  "Error:Both backup path AND VM name must be specified, exiting..."

echo  "Use $0 -h  for help"

exit 1

fi

#If running in interactive mode

if [ ${AUTO} != "1" ];then

      echo "Going to back-up ${VMNAME} to ${BACKUPPATH} ,is this OK? [y/n]"

      read ANSWER

      if [ ${ANSWER} != "y" ] && [ ${ANSWER} != "Y" ];then

            echo "Exiting by users request..."

            exit 1
      fi
fi

      #else proceed with main program...

      else

            check_dir

            TEMPUUID=$(xe vm-snapshot vm=$VMNAME new-name-label=snap_tmp_`date +%F`)

            xe template-param-set is-a-template=false ha-always-run=false uuid=$TEMPUUID

            xe vm-export vm=${TEMPUUID} filename=${BACKUPPATH}/${VMNAME}/${VMNAME}_backup_`date +%F`.xva

            echo "Removing Temporary Snapshot....(DO NOT PANIC!)"

            xe vm-uninstall uuid=${TEMPUUID} force=true

            echo "==============================================================="

            echo "### Back Up of ${VMNAME} Completed!###"

            echo  "`date +%H:%M`"

#END OF SCRIPT

Howto Connect NetApp iSCSI to XenServer 5.6 FP1

Since iSCSI is a great choice as a SR for your XenServer, in terms of performance and scalability, I've decided to created a straight forward and easy to understand guide of how to connect NetApp iSCSI based storage to XenServer, read on...

On the NetApp side:

1) Enable iSCSI service on the filer:

2) Create the desired LUN:

·        

3) Dedicate a separate NIC for iSCSI traffic, be sure the filer iSCSI dedicated NIC is assigned with a proper IP address:

·        

4)From the XenServer copy it's iscsi FQN:

5)Add the new initiator:

6)Map the new LUN to the new initiator:

On the XenServer side:

1)      Be sure you have a NIC assigned to iSCSI traffic.

2)      Right click on the XenServer -> New storage-> Software iSCSI

Click on Discover IQNs, it should find the NetApp IQN.

Next, click on Discover LUN's, it should find the available LUN's.

Click "Finish" to attach the SR.

Your newly created LUN should be successfully attached to XenServer at this point.

Howto rescan NICs on XenServer

1. First list the physical NIC's:
#xe pif-list

2. After making a note of the information, run the following command for each one of the new NIC's that appear as disconnected:
#xe pif-forget uuid=UUID


3. Next, have the system scan for the correct uuids, you will need the uuid of your server (find it with "xe host-list" ):
#xe pif-scan host-uuid=UUID

4. List the NICs again to see what we have found:
#xe pif-list

5. You should see the correct UUID's of your NICs. 
Then, run the following command for each one of the new NIC's to add them back:
#xe pif-plug UUID

At this point your new NICs should appear as connected.

Howto Backup VM's in XenServer

I found this technique to be the easiest and most efficient, importing the whole virtual machine to *xva file. Though it's not the quickest method it worked flawlessly for me.

1. At XenServer console as root, mount some SR (CIFS or NFS) to a folder:

#mount -t cifs //myfiler01/ntfs_share /backup/

2. Check the mount:

#mount |grep backup

3. To find the VM you want to export Run:

#xe vm-list

4. Run the export VM command:

#xe vm-export vm=centos5 filename=/backup/centos5_backup.xva

That's it, the VM exported as *.xva file which is easy to import in case of failure. 

In case you want to import the VM, run:

#xe vm-import vm=centos5 filename=/backup/centos5_backup.xva