Thursday, December 15, 2011

XenServer - Cannot Shut VM

Have you ever experienced  a situation where you could not shut one of your running XenServer running VM's?   In this brief article I'll present a workaround for this common problem...  From my personal experiences this problem is often caused when the domain of the specific VM is still active (this often happens as a result of inproper shutdown and/or storage cutoffs) - in such case the domain needs to be destroyed manually.  OK, so here's the problem - from the XenServer (v 5.6 SP1) dom0 CLI I'm trying to shut a VM (using the force option): #xe vm-reset-powerstate vm=xxxxxxx force=true
However the output I get is:  The operation could not be performed because a domain still exists for the specified VM.
vm: f087dxxxxx (xxxxxx)
domid: 5
  From the output we will need  two parameters :the domain id (5 in our case) and the UUID of the problematic VM , so write them down somewhere. Now we need to destroy the currently active domain, XenServer includes couple of good tools under /opt/xensource, the command that is relevant is:   #/opt/xensource/debug/destroy_domain -domid 5  Now, finally, we will be able shut the VM with:  #xe vm-reboot uuid=XXXX --force  And we're done.  Hope that helped some one out there. Cheers!

Wednesday, December 7, 2011

Perl SSH Log-in

Hi Folks, 

In the following post I will demonstrate a password-less login into Cisco appliance via the SSH protocol using a Perl script.

When combined with cron, it can be a great solution for saving your appliance configuration (show run) or checking status without the need to log-in into the appliance each time.

Pre requirements:
  • Perl
  • SSH client (duh!)
  • Net-SSH-Perl module

On Red-Hat (and friends) the module can be obtained via:
#yum install perl-Net-SSH-Perl -y 

Be sure to have a proper repository installed such as rpmforge 
Our simple script connects to the Cisco appliance via SSH and runs "show run" command (this account has enabled privilege).
The script itself should look like this:

#!/usr/bin/perl -w
use strict;
use warnings;
use Net::SSH::Perl;
my $ssh = Net::SSH::Perl->new('hostname');
$ssh->login('username', 'password');
my($out) = $ssh->cmd("show run");
print $out;

Since the script contains your appliance username & password don't forget to remove permissions for others:
#chmod o-rwx

Now, let's run the script:

Building configuration...

Current configuration : 8011 bytes
...more output ommited...

Works like charm!
Now, the only thing is left is to synchronize it with cron :)


Sunday, November 27, 2011

Howto configure LDAP with TLS

Configuring LDAP over TLS is a crucial step in order to achieve a secure directory based authentication.
In the following tutorial I'll demonstrate how to configure OpenLDAP with TLS.

For the demonstration both server & client I've used are CentOS 5.5 x86_64
My LDAP server is OpenLDAP v2.3.43

Let's get started.

Step1 - Generating and Signing Certificates:

First of all we need to generate and sign the OpenLDAP server certificates.
In this scenario we will both generate and sign our certificate (self signed), this solution is good for internal and usually small environments.
If you ever plan to use this in a big enterprise production environment you will need a proper CA, like Verisign or Terrena to sign the certificates.

Before continuing please make sure the open SSL package is installed.

Generate your private key and a certificate request form:

#cd /etc/openldap/cacerts
#openssl req -nodes -newkey rsa:2048 -keyout server.key -out server.csr  Sign the certificate with:
#openssl x509 -in server.csr -out server.crt -req -signkey server.key -days 3650
Now you got both private and public keys.

After the certificates have been generated, assign the correct permissions and ownerships:

#chown ldap server.*
#chmod 0400 server.key
#chmod 0644 server.crt

Step2 - Server Side:

The following two lines need to be added in your server configuration file

TLSCertificateFile      /etc/openldap/cacerts/server.crt
TLSCertificateKeyFile   /etc/openldap/cacerts/server.key
Restart the LDAP server:
#/etc/init.d/ldap restart 

Step3 - Client Side:

Make sure the ssl parameter in your LDAP client configuration file (/etc/ldap.conf) looks like this:

ssl start_tls

Also, make sure /etc/openldap/ldap.conf includes the following parameters:

TLS_CACERTDIR /etc/openldap/cacerts

That's it, from this point your OpenLDAP is ready to be used over TLS.
Stay secure ;)

Monday, September 26, 2011

XenServer Backup Script

This handy bash script I wrote, enables Xen Server Administrators to create On-Line backups (Importing the VM into.xva file) while the backed up virtual machine is up and running.

I found this extremely useful for big production environments - where lots of VM's require an appropriate & efficient back-up solution and downtime is not an option.

The script is very simple and user friendly, it's executed as root from the dom0 machine and used as follows:
In order to list all the available VM's on the system:
#./ -l

In order to back up a VM:
#./ -p /path/to/backupdir -s vm-name

The script supports both interactive and unattended mode (-u flag), the later designed to be built into scripts as an automation solution for large production environments.

For any help/usage, just run:
#./ -h

Tested succesefully on XenServer 5.6 FP1 & XenServer 5.6 SP2.
Any comments, feed-backs or suggestions for improvements are very welcomed.

#This script used to create XenServer VM's online backup #Importing the VM into .xva file #Written by Paul Podolny - July 2011

AUTO=0 #Used for unattended mode
VMLIST=`xe vm-list is-control-domain=false |grep -i name-label|awk -F": " '{print $2}'`

list() {
echo "====================================="
echo "Listing available VM's on $HOSTNAME..."
echo "====================================="
for i in $VMLIST;do
      echo $i;let COUNT+=1
echo "================================="
echo "Total $COUNT Virtual Machines"
echo "================================="
exit 0

usage() {
cat << EOF
This script will backup your Xen VMs on-line:
-h Help/Usage.
-l List VM's.
-u Run in unattended mode (No questions asked).
-s Server Name (Backed Up VM).
-p Path to backup.

$0 -s server01 -p /nfs/backupdir/ -u
exit 0
check_dir() {
echo "=============================================================="
echo "XenServer Online Backup"
echo "=============================================================="
if  [ -e ${BACKUPPATH}/${VMNAME} ];then
        echo "Directory ${VMNAME} on path ${BACKUPPATH} seem to exist, will use it."

      echo "Creating dir. ${VMNAME} on backup path: ${BACKUPPATH}"
      mkdir ${BACKUPPATH}/${VMNAME}

echo "`date +%H:%M` Backing Up ${VMNAME}, please hold on...Do NOT interrupt with CTRL+C !!!"
echo "================================================="

#MAIN Prog
while getopts "s:p:lhu" flag ; do
        case $flag in
                l ) list;;
                s ) VMNAME=$OPTARG;;
                p)  BACKUPPATH=$OPTARG;;
                h ) usage;;
                u)  AUTO=1;;
                *)  usage;;

#Sanity check  - Make sure backup path + VM name specificed if [ -z ${VMNAME} ] || [ -z ${BACKUPPATH} ];then echo  "Error:Both backup path AND VM name must be specified, exiting..."
echo  "Use $0 -h  for help"
exit 1

#If running in interactive mode
if [ ${AUTO} != "1" ];then
      echo "Going to back-up ${VMNAME} to ${BACKUPPATH} ,is this OK? [y/n]"
      read ANSWER
      if [ ${ANSWER} != "y" ] && [ ${ANSWER} != "Y" ];then
            echo "Exiting by users request..."
            exit 1
      #else proceed with main program...
            TEMPUUID=$(xe vm-snapshot vm=$VMNAME new-name-label=snap_tmp_`date +%F`)
            xe template-param-set is-a-template=false ha-always-run=false uuid=$TEMPUUID
            xe vm-export vm=${TEMPUUID} filename=${BACKUPPATH}/${VMNAME}/${VMNAME}_backup_`date +%F`.xva
            echo "Removing Temporary Snapshot....(DO NOT PANIC!)"
            xe vm-uninstall uuid=${TEMPUUID} force=true
            echo "==============================================================="
            echo "### Back Up of ${VMNAME} Completed!###"
            echo  "`date +%H:%M`"


Wednesday, July 20, 2011

Howto:Reduce LVM Root Partition

In this short tutorial I'll demonstrate how to reduce the size of LVM (/dev/VolGroup00/LogVol00) mounted on "/" (aka root partition).
Since the procedure cannot be done on the run (you will need to unmount the root partition...) , you will have to boot your machine either with some sort of Linux LiveCD or either with the original CentOS DVD/ISO which offers special rescue mode.

I didn't have a LiveCD so I used the original CentOS DVD for this task.

Boot the machine with a CentOS DVD/ISO (v 5.5 in my case) and enter the rescue mode by typing:
#linux rescue

CentOS will boot in rescue mode when a menu pops up and asks you if you want to mount your old partitions, chose "SKIP" (otherwise your original disk partitions will be mounted under /mnt/sysimage - in that case umount it before proceeding).

After you made sure your original disk partitions are not mounted you will have to activate the LVM in Rescue Mode:
# lvm vgchange -a y

Check the LVM for any filesystem errors, this is crucial for next step:
# e2fsck -f /dev/VolGroup00/LogVol00

It's time to resize the filesystem to a new size ,note that you should leave enough space for current data on the root partition or you will suffer a data loss.
In our case we are reducing the partition to a new size of 20GB.
Note that the procedure may take some time (depending on your root partition size, so do not panic!):
# resize2fs -f /dev/VolGroup00/LogVol00 20G

Finally, resize the LVM to the new size (20 GB):
#lvm lvreduce -L20G /dev/VolGroup00/LogVol00

You should boot your OS into regular mode and cross your fingers ...

Disclaimer: Do at your own risk, I'm not responsible for any data loss which may be caused to your system.

Tuesday, July 12, 2011

Enable SNMP on Cisco Devices (part 1)

In this short tutorial I'll demonstrate how to quickly configure SNMP on your Cisco appliance and test it's working.

There will be another tutorial dealing with how to configure the SNMP manager side.

1) Let's start by logging in to your Cisco appliance (In my case I used 851 series Cisco router with IOS version 12.3).

2) Enter configuration mode:
cisco851#configure terminal

3) Set your community string and the mode (read only, read write):
cisco851(config)#snmp-server community myComunity1 RW

4) Point the appliance to the SNMP manager (in our case , with the same community string you've set before: 

 cisco851(config)#snmp-server host version 2c myComunity1

5) Enable the trap types you wish to monitor, for example:

cisco851(config)#snmp-server enable traps snmp linkdown linkup coldstart warmstart

6) Save the configuration:
cisco851(config)#do wr

At this point the basic configuration on the Cisco appliance is done, let's see if we able to querry the appliance from our server.

I have used Ubuntu 11.04 x64 box with "snmp" package installed.

Check it's indeed installed with:
root@ubuntu:~#dpkg --list |grep snmp
If not get it with:
root@ubuntu:~#apt-get install snmp

After snmp package has been succefully installed it's time test the configuration.

Issue the following command and see if you're able to retrieve the SNMPv1 agent  or our Cisco appliance ( MIB tree list :

root@ubuntu:~#snmpwalk -v 1 -c myComunity1

At this point you should be able to list the entire tree (output was ommited), means Cisco side is configured successfully.

You can now get the desired info for monitoring purposes, like the system uptime:

root@ubuntu:~#snmpget -v 1 -c myComunity1

DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (490514418) 56 days, 18:32:24.18
Or our appiance hostname:

root@ubuntu:~#snmpget -v 1 -c myComunity1

SNMPv2-MIB::sysName.0 = STRING:
Mission accomplished.
In the next SNMP article I will touch the server side configuration.

Monday, May 2, 2011

Howto remove old NAT configuration from Cisco router

Recently I came upon a situation where my ISP has assigned me a static IP - instead of L2TP dialer (Virtual PPP Interface in Cisco) I was using before ...
Anyway, I started to remove old and irrelevant NAT settings that were associated with my old dialer ,but when I've tried to configure the new NAT settings I got an error:
%Dynamic mapping in use, cannot change

After scratching my head for awhile I found the solution.

To remove old NAT settings on Cisco router you need to 

1)Clear all old NAT translations
router#clear ip nat translatiom *

2)Disable old NAT pool settings
router(config)#no ip nat pool public_access netmask

3)And finally, disable the translation:
router(config)#no ip nat inside source list 1 pool public_access overload

From this point you can safely configure the new NAT settings.

Thursday, April 14, 2011

Howto Connect NetApp iSCSI to XenServer 5.6 FP1

Since iSCSI is a great choice as a SR for your XenServer, in terms of performance and scalability, I've decided to created a straight forward and easy to understand guide of how to connect NetApp iSCSI based storage to XenServer, read on...
On the NetApp side:
1) Enable iSCSI service on the filer:

2) Create the desired LUN:

3) Dedicate a separate NIC for iSCSI traffic, be sure the filer iSCSI dedicated NIC is assigned with a proper IP address:


4)From the XenServer copy it's iscsi FQN:

5)Add the new initiator:

6)Map the new LUN to the new initiator:

On the XenServer side:
1)      Be sure you have a NIC assigned to iSCSI traffic.

2)      Right click on the XenServer -> New storage-> Software iSCSI

Click on Discover IQNs, it should find the NetApp IQN.
Next, click on Discover LUN's, it should find the available LUN's.

Click "Finish" to attach the SR.

Your newly created LUN should be successfully attached to XenServer at this point.

Wednesday, April 6, 2011

Howto rescan NICs on XenServer

1. First list the physical NIC's:
#xe pif-list

2. After making a note of the information, run the following command for each one of the new NIC's that appear as disconnected:
#xe pif-forget uuid=UUID

3. Next, have the system scan for the correct uuids, you will need the uuid of your server (find it with "xe host-list" ):
#xe pif-scan host-uuid=UUID

4. List the NICs again to see what we have found:
#xe pif-list

5. You should see the correct UUID's of your NICs. 
Then, run the following command for each one of the new NIC's to add them back:
#xe pif-plug UUID

At this point your new NICs should appear as connected.

Sunday, April 3, 2011

Howto Backup VM's in XenServer

I found this technique to be the easiest and most efficient, importing the whole virtual machine to *xva file. Though it's not the quickest method it worked flawlessly for me.

1. At XenServer console as root, mount some SR (CIFS or NFS) to a folder:
#mount -t cifs //myfiler01/ntfs_share /backup/

2. Check the mount:
#mount |grep backup

3. To find the VM you want to export Run:
#xe vm-list

4. Run the export VM command:
#xe vm-export vm=centos5 filename=/backup/centos5_backup.xva

That's it, the VM exported as *.xva file which is easy to import in case of failure. 

In case you want to import the VM, run:

#xe vm-import vm=centos5 filename=/backup/centos5_backup.xva

Saturday, February 19, 2011

Howto Configure Heartbeat Cluster

Server clustering gains more and more popularity these days.
One of the leading open-source products for HA is called "Heartbeat" (

Heartbeat offers great (and free!) way to achieve HA on a Linux machine.
While configuring Heartbeat may not be the easiest thing , it's surely worth the effort, especially when we talking about some fancy production machine which requires HA.

In this article I will demonstrate how to configure Heartbeat and achieve full HA, read on.

For my test I have used 2 virtual stations (both installed CentOS v5.5 x64) - node1 & node2.

Pay attention that every step needs to be done on both hosts.
Also pay attention that your firewall is configured properly (port 694 udp needs to be allowed).

Step 1:
The first thing you want to do is to edit your /etc/hosts, in my case it looks like this: node1 node1-e1 node2 node2-e1

When node1-e1 & node2-e2 are virtual interfaces on both of the machines (more explanation ahead).

Step 2:
Configure virtual interfaces on both of the nodes:

root@node1# ifconfig eth0:1 netmask
root@node2# ifconfig eth0:1 netmask

Step 3: 
Install heartbeat on both of the nodes:
# yum install heartbeat

Step 4:
Edit "authkeys" file (/etc/ha.d/authkeys) to include:

auth 1
1 crc

Step 5:
Edit "haresources" file (/etc/ha.d/haresources) to include:

node1 IPaddr2::
node2 IPaddr2::

Step 6:
Edit "" file (/etc/ha.d/ include:

logfile /var/log/ha-log
debugfile /var/log/ha-debug
logfacility local0
udpport 694
ucast eth0
ucast eth0
ucast eth0
ucast eth0
ping node1-e1 node2-e1
node node1 node2
respawn hacluster /usr/lib64/heartbeat/ipfail
apiauth ipfail gid=haclient uid=hacluster
auto_failback on
keepalive 2
deadtime 10
warntime 3
deadping 10
realtime on
initdead 50
debug 1

Step 7:
Restart Heartbeat:
# /etc/init.d/heartbeat restart

That's should be it, now let's test the HA.
I'm pinging node1 from 3rd (Windows) machine, then rebooting it, pay attention that after short time node2 takes over and I'm getting response:


Thursday, February 3, 2011

Howto enable passwordless SSH on NetApp filer

When working in an environment with lots of filer servers, you might consider enable password less SSH on the filer for easier administration (from some administration host).
The concept stays the same, public keys exchange, but as you know the NetApp OnTapp OS uses slightly different syntax, so I decided to write this small guide that will help you out:

OK, let's get busy , I'll assume that the filer has already has networking configured correctly.

1) The filer does not have SSH enabled by default, so login via telnet:
admin_host> telnet filer01

2) Set up root password:
filer01> passwd

3) Next, you need to enable SSH (preferably version 2 - as it's more secure):
filer01> secureadmin enable ssh2
filer01> secureadmin setup ssh

4) Make sure you exports file is edited correctly and vol0 is exported to admin_host
admin_host> showmount -e filer01

You can edit exports file from the filer with "wrfile" command, if you have modified the file remmember to re-export the new exports with:

filer01 >exportfs -av

5) Next, mount  vol0 from the NetApp filer on the amdministration host:

admin_host> mkdir -p /nfs/filer01/vol0
admin_host> mount -t nfs filer01:/vol/vol0 /nfs/filer01/vol0

Check that you see the mounted volume:

admin_host> ls /nfs/filer01/vol0

If not you're probably having some issue with your firewall, or exports on the filer side.

6) This is the most critical part, here you will create the ssh directory and append your root public key to authorized_keys of the filer:

admin_host> mkdir -p /nfs/filer01/vol0/etc/sshd/root/.ssh/

admin_host> cat /root/.ssh/ >> /nfs/filer01/vol0/etc/sshd/root/.ssh/authorized_keys

7) Last, you may want to turn down rsh and telnet services (for obvious security reasons):

filer01> options rsh.enable off
filer01> options telnet.enable off

Your are done.

Saturday, January 15, 2011

Bash getops - Quick & Easy

Ever wanted to create this super script with tonns of flags and parameters?
In this short tutorial I will show you how to do it in bash shell.

Let's say we have a script
That has 3 flags (-v for verbouse, -h for help, -i for ip address as input).
The two first parameters do not really get input, but more "tell the script how to behave", while the third parameter actually gets a value from user (ip address).

The "getopts" function is a very handful function when you’re working with script where parameters and flags have to be set.

Our script will be executed as (for example):
#./ -v -i

This is the content of our short script:
usage() {
cat << EOF
-h help
-v verbouse mode
-i new ip address
while getopts "i:hv" flag ; do
        case $flag in
                i ) IPADDR=$OPTARG;;
                h ) usage;;
                v ) VERB=1;exit 0;;
Pay attention to the options with a colon (“:”) after them need to have an argument set.
Yet if there is no column, then no argument is needed (-h, -v).

We combine while loop with "getopts" function, that will use case to determine which flags were used and what values has been assigned.