Wednesday, April 18, 2012

Checkpoint FW :Failed to load Policy on Module

While not being exactly a security expert once in a while I have to deal with some security appliances especially in test environments (where FW rules need to be adjusted quite frequently).

Most of the time I was extremely pleased with Checkpoint products ,at least for me their products were rock solid - until one day I wasn't able to install new policies on my Checkpoint FW.  The symptom was quite awkward - after saving the policy and verifying it successfully, during the installation process I always got an error saying "Installation failed. Failed to load Policy on Module" no matter what I tried, no additional info was specified which complicated things a bit.

Here is my workaround for the problem:

After you've logged in into the appliance as admin user (either via console or ssh) ,type:
# Expert

In order to get into privileged (Expert) mode (which basically allows you to work as "root" user on the appliance , as it was a regular Linux box).

After you got into expert mode the prompt will change to:
 [Expert@firewall]#

Now, you need to locate the "fwm" process (which is the FW management), kill it and then restart it.

Please note that if your SmartDashboard (or any other Checkpoint applications) are connected to the FW ,it will terminate them, yet the FW traffic (including any established VPN connections) will not be affected, so proceed without worries:

[Expert@firewall]#ps -ef|grep fwm
[Expert@firewall]#kill fwm-pid
[Expert@firewall]#fwm &

After fwm was started successfully on your FW box, try installing the policy again - usually this should do the trick.

If restarting fwm did not help, as a last resort  only, you will need to restart the CP services.
This will of course disconnect any sessions and every established  VPN connections, so think twice before executing it:

[Expert@firewall]#cpstop && cpstart.

The CP restart process takes around ~1 minute during which the FW may seem unresponsive.


This did the trick for me and I hope it helped some one out there too.
If you have more elegant solution for this issue, please let me know.
 
Cheers. 

Sunday, March 18, 2012

XenServer 5.6 FP1 VDI Issue

Recently I ran into situation where one of my production VM's  went unresponsive,  the VM console showed various VDI related (xvda) I/O errors and the machine was halted.
It's worth mentioning that my XenServers (5.6 FP1) nodes operate in pool mode, and the problematic VDI resided on iSCSI LUN which seemed to be OK.

There weren't much other options rather than shutting down the VM with:
#xe vm-shutdown vm=vm0001 force=true

However, when I've tried to power on the VM back I got this nasty error:
18-Mar-12 9:42:16 AM Error: Starting VM 'vm0001' - Internal error: Failure("The VDI e17e2406-dbe9-40f6-98c3-af470e8aa91b is already attached in RW mode; it can't be attached in RO mode!")

Here is the workaround which did the job for me:

1. Find the UUID of the Storage Repository and the VM problematic VDI.
#xe sr-list |grep -i -C2 'your LUN name'
#xe vdi-list |grep -i -C2 'vdi name'

2. Next, we need to to remove VDI from the listing:
 #xe vdi-forget uuid=

Do not worry about the contents of the VDI they are fine :)
Verify the VDI is indeed gone:
#xe vdi-list |grep -i 'vdi name'

3. It's time to re-scan the storage repository that hosts the VDI via:
# xe sr-scan sr-uuid=

4. Verify the VDI is back in the listing:
#xe vdi-list sr-uuid=

Please note that the "name" and "description" fields are now empty.

5. Use XenCenter to reattach the VDI to your VM , and start it on different XenServer host inside your pool (right click on the VM, select "storage"->"attach"->).


This should do the magic.


Cheers!

Thursday, February 16, 2012

Howto resize XenServer LUN's Online

In the following procedure I will show how to extend iSCSI attached LUN's on XenServer (v 5.6 SP2 in my case) on the fly ,so no service restart or downtime are needed and the VDI's that reside on the resized LUN are not affected, read on!
First of all on the storage side (NetApp filer in my case)list the available LUN's, the LUN I want to resize is  called "my_lun30", with LUN ID of 22, which is currently allocated with 90GB.

For the test I will extend it by another 10GB, making it a 100GB LUN.

filer1> lun show
        /vol/vol1/my_lun10         300.0g (322163441664)  (r/w, online, mapped)
        /vol/vol2/my_lun20    200g (214748364800)  (r/w, online, mapped)
        /vol/vol2/my_lun30       90.0g (107388862464)  (r/w, online, mapped)


filer1> lun resize /vol/vol2/my_lun30 +10g
lun resize: resized to:  100.0g (107388862464)


As you can see, my_lun30 is now 100GB:

filer1> lun show
        /vol/vol1/my_lun10         300.0g (322163441664)  (r/w, online, mapped)
        /vol/vol2/my_lun20    200g (214748364800)  (r/w, online, mapped)
        /vol/vol2/my_lun30        100.0g (107388862464)  (r/w, online, mapped)

We are done with the storage side, let's head to the XenServer side.
 In case you work in pool mode, login to the pool master as root:

I suggest installing "lsscsi" which provides a nice way of viewing SCSI attached disks/LUNs:
[root@xen3]#yum install lsscsi -y

[root@xen3]# lsscsi
[0:0:0:0]    cd/dvd  Optiarc  DVD RW AD-7561S  AH52  /dev/scd0
[2:0:0:0]    disk    NETAPP   LUN              7340  /dev/sda
[2:0:0:22]   disk    NETAPP   LUN              7340  /dev/sdb
[2:0:0:33]   disk    NETAPP   LUN              7340  /dev/sdc
[3:0:0:1]    disk    NETAPP   LUN              7340  /dev/sdd
[3:0:0:3]    disk    NETAPP   LUN              7340  /dev/sde

If YUM traffic to a repository is blocked - It's also possible to see the SCSI id's under /proc via:

# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: Optiarc  Model: DVD RW AD-7561S  Rev: AH52
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 22
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 33
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 01
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 03
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04

From the output we can see the SCSI ID of the LUN + it's corresponding device on the system, and check the device physical size, as you can see it is not updated yet:

[root@xen3 backup_scripts]# fdisk -l /dev/sdb

Disk /dev/sdb: 96.6 GB, 96647249920 bytes
255 heads, 63 sectors/track, 11750 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Since XenServer uses LVM, we can see the physical volume info of the disk, as you can see the size is still 90GB.
  
[root@xen3]# pvdisplay /dev/sdb
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               VG_XenStorage-e71389d1-4dc3-2518-aa30-9f5f0c70ba12
  PV Size               90.01 GB / not usable 6.12 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              23039
  Free PE               16624
  Allocated PE          6415
  PV UUID               oSZlnA-VxlA-3qbp-07nI-Ql0b-3cG2-9wo2lC


Now, we will tell XenServer to rescan the SCSI bus, we will provide the SCSI id which we previously got from the "lsscsi" command (2:0:0:22):

[root@xen3]# echo 1 > /sys/class/scsi_disk/2:0:0:22/device/rescan

You can notice the immediate change of /dev/sdb:

[root@xen3]# fdisk -l /dev/sdb

Disk /dev/sdb: 107.3 GB, 107388862464 bytes
255 heads, 63 sectors/track, 13055 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Now, resize the physical volume with:

[root@xen3]# pvresize /dev/sdb

  Physical volume "/dev/sdb" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Check again the physical volume size, notice the change:

[root@xen3]#  pvdisplay /dev/sdb
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               VG_XenStorage-e71389d1-4dc3-2518-aa30-9f5f0c70ba12
  PV Size               100.01 GB / not usable 6.12 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              25600
  Free PE               19185
  Allocated PE          6415
  PV UUID               oSZlnA-VxlA-3qbp-07nI-Ql0b-3cG2-9wo2lC

Let's get the SR uuid, I know the LUN ID is 22, so:

[root@xen3]# xe sr-list|grep lun22 -B1
uuid ( RO)                : e71389d1-4dc3-2518-aa30-9f5f0c70ba12
          name-label ( RW): iSCSI_filer1_lun22

Notice that the SR size has yet to be updated:

[root@xen3]#  xe sr-param-list uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12|grep physical-size
           physical-size ( RO): 96632569856

Now, finally update the relevant SR:

[root@xen3]#  xe sr-update uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12

And at last, the SR is updated with the correct new LUN size :
[root@xen3]#  xe sr-param-list uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12|grep physical-size
           physical-size ( RO): 107374182400


You are done!

Monday, January 30, 2012

Install & Configure OpenVZ (CentOS 6.2)

OpenVZ is a great tool which offers a great virtualization solution with near zero overhead, thus offering great performance.
In this short tutorial I will show how to install it on CentOS 6.2 machine, read on:


1) Get the OpenVZ repository and update "yum":
#wget -P /etc/yum.repos.d http://download.openvz.org/openvz.repo
#rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ
#yum update


2) Install relevant packages
#yum install openvz-kernel-rhel6 vzctl vzquota bridge-utils -y

3) Modify relevant kernel (networking) settings to allow proper communication with the VPS'es:
#vi /etc/sysctl.conf

#add these lines for sysctl openvz configuration
net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.icmp_echo_ignore_broadcasts=1 

net.ipv4.conf.default.forwarding=1
net.ipv4.conf.default.proxy_arp = 0
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.eth0.proxy_arp=1

Update the new kernel settings:
#sysctl -p

4) Reboot the machine:
#shutdown -r now

A new Kernel should appear (2.6.32-042stab044.17 in my case) in the Grub menu.
Boot into the new Kernel.

5) Check that a new interface (venet0) exists:

# ifconfig venet0
venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet6 addr: fe80::1/128 Scope:Link
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:161 errors:0 dropped:0 overruns:0 frame:0
          TX packets:182 errors:0 dropped:12 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:19255 (18.8 KiB)  TX bytes:15822 (15.4 KiB)


Also, check the the "vz" service is running:
#/etc/init.d/vz status
OpenVZ is running...

6) So far, so good - It's time to get some OS template. Let's get an Ubuntu 11.10 64bit template:

#wget http://download.openvz.org/template/precreated/ubuntu-11.04-x86_64.tar.gz -P /vz/template/cache

All templates come as archives and reside inside /vz/template/cache directory.

As a best practice it's a good idea to keep /vz on a separate partition (or a LUN), the partition needs to be big enough to sustain all the VPS'es that are about to be created, so do the calculation according to your needs.

7) Basic installation is done. You should be able to use the vz* commands and administer your VM's via the CLI.

For example to create a new VM out of the downloaded template use:
#vzctl create 1 --ostemplate ubuntu-11.04-x86_64 --ipadd 10.0.0.12 --hostname vz03

When 1 is the uid of the VPS.

After the creation, initialize the created VPS via:
#vzctl start 1


You can now enter into the VPS by simply SSH'ing into it or via the following command:
#vzctl enter 1


A very cool (and free) web management which I highly recommend is called OpenVZ Web Panel, can be easily installed via this command:

#wget -O – http://ovz-web-panel.googlecode.com/svn/installer/ai.sh | sh

After the installation, check that the OpenVZ web panel is listening on port 3000:
#lsof -i :3000

An initialization script is provided as part of the installation and is located under: /etc/init.d/owp

Once installed the web panel can be accessed from your browser via
http://your-ip:3000

The interface is minimalistic but very convenient and user friendly:











Note: Be sure to modify firewall settings on the hosting machine accordingly to allow access to port 3000.

Happy VZ'ing!

Saturday, January 14, 2012

How to Install & Configure Cfengine (v3) on CentOS 6

In this short tutorial I will demonstrate how to quickly install and configure Cfengine (version 3.x) on CentOS 6 x64 box and also show a basic example of this great automation tool, let's get started:

First, install the needed pre-requirements (EPEL/RPMforge provide these):

#yum install openssl openssl-devel db4 db4-devel flex pcre pcre-devel openldap gcc -y

Next, download the latest version source (3.2.3 at the moment of writing) and install it:

#wget http://cfengine.com/source-code/download?file=cfengine-3.2.3.tar.gz -O /root/cfengine-3.2.3.tar.gz
#tar xvzf cfengine-3.2.3.tar.gz; cd cfengine-3.2.3/
#./configure
#make
#make install

Now we will create the needed folders for CF under /var to work as should ,we will  also copy all relevant binaries and configuration files:

#mkdir -p  /var/cfengine/{masterfiles,bin,inputs}
#cp -rp /usr/local/share/cfengine/masterfiles/*.cf /var/cfengine/masterfiles/
#cp -rp /usr/local/share/cfengine/masterfiles/*.cf /var/cfengine/inputs/
#cp -rp /usr/local/sbin/cf-* /var/cfengine/bin/

Test the installation with:
#cf-promises -v

On the last line output you should get:
cf3>  -> Inputs are valid

Now, in order to demonstrate Cfengine abilities we will create a new file under /tmp called "cftest" with premissions of 744 and owner root.

In order to do that we will create a special configuration input file under:
/var/cfengine/inputs, called cftest.cf

So, let's see what we got inside of the configuration file:



#cat /var/cfengine/inputs/cftest.cf

body common control
{
# Define a bundle sequence
bundlesequence => { "checkperms" };
# Include cfengine_stdlib.cf
inputs => { "cfengine_stdlib.cf" };
version => "1.0.0";
 }
bundle agent checkperms        
{
files:                   
"/tmp/cftest"
create => "true",                             
perms => m("744");
}

Let's verify it's syntax with cf-promise:
#cf-promise -f /var/cfengine/inputs/cftest.cf

We will see that there is no file called cftest under /tmp ,prior running cf-agent.
# ls -l /tmp/cftest
ls: cannot access /tmp/cftest: No such file or directory

Now for the run:
#cf-agent -f /var/cfengine/inputs/cftest.cf

Lets check what's under /tmp:
# ls -l /tmp/cftest
-rwxr--r-- 1 root root 0 Jan 14 01:14 /tmp/cftest


Great, the file was created with the right permissions, just as we wanted. 
This is just the most basic example, via Cfengine you can do much much more, more practical examples to come so stay tuned!