Set Up ElasticSearch cluster on CoreOS (pt1)

For those who are not familiar with CoreOS,  it's an extremely thin version of Gentoo, designed to run and orchestrate Docker containers at scale. In the following tutorial I'll show how to deploy a test ES cluster on top of CoreOS.

1) On your admin node (your laptop?) generate etcd unique discovery string:

$ curl -L https://discovery.etcd.io/new

Copy/paste the output as this will be used by our cluster nodes for discovery.

2) Next, from the AWS console/AWS CLI API launch 3 instances with the following user data:

#cloud-config

coreos:
  etcd:
    discovery: https://discovery.etcd.io/78c03094374cc2140d261d116c6d31f3
    addr: $public_ipv4:4001
    peer-addr: $public_ipv4:7001
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start

I have used the following AMI ID: ami-0e300d13 (which is CoreOS stable 607).


3) On the admin node add your AWS private key to the ssh-agent:

 $ eval `ssh-agent -s`
 $ ssh-add ~/.ssh/test-private-key.pem

4) Get Go Lang + Install fleetctl, we will use it orchestrate our CoreOS cluster:

$ apt-get update; apt-get install golang -y
$ git clone https://github.com/coreos/fleet.git /opt/fleet
$ cd /opt/fleet ; ./build
$ echo "export PATH=${PATH}:/opt/fleet/bin" >> ~/.bashrc 
$ source ~/.bashrc

5) Check fleetctl functionality:

$ fleetctl --tunnel coreos1 list-machines
MACHINE         IP              METADATA
06673ee6...     172.31.13.15    -
3c5c65e8...     172.31.13.16    -
fd02bd21...     172.31.13.17    -

Where 'coreos1' is one of our cluster nodes external IP.

Voila! our CoreOS 3 node cluster is up and running, brilliant ;)

6) Let's create a sample unit file (similar to systemd) which will pull Docker ES container and bind it's port to 9200 of the host:

$ cat << EOF > ES.service
[Unit]
Description=ElasticSearch
After=docker.service
Requires=docker.service

[Service]
ExecStartPre=/usr/bin/docker pull elasticsearch:latest
ExecStart=/usr/bin/docker run --name elasticsearch -p 9200:9200 elasticsearch
ExecStopPre=/usr/bin/docker kill elasticsearch
ExecStop=/usr/bin/docker rm elasticsearch
TimeoutStartSec=0
Restart=always
RestartSec=10s

[X-Fleet]
X-Conflicts=ES@*.service

EOF

7) Launch our newly created unit file:

$ fleetctl --tunnel coreos1 start ES.service
Unit ES.service launched on 06673ee6.../172.31.13.15

$ fleetctl --tunnel coreos1 list-units
UNIT            MACHINE                         ACTIVE  SUB
ES.service      06673ee6.../172.31.13.15        active  running

Boom! Our ES node is up and running. We can verify it's functionality by executing a simple HTTP GET such as:

$ curl -q -L http://coreos1/9200/_status/

We are still missing some important parts such as persistent data for ES Docker containers (to survive reboots), nodes discovery, monitoring and much more so stay tuned for the next part.

Installing CoreOS on BareMetal Server

Installing CoreOS is fairly a simple task.

On the host from which you will administer the CoreOS nodes from (aka the "admin machine"), make sure to copy (or generate new) SSH public key which will be used for authentication to the CoreOS machine(s), so in case there is no public key exist:

ssh-keygen -t rsa -N '' -f ~/.ssh/id_rsa

cat ~/.ssh/id_rsa.pub

Boot your machine with any Linux LiveCD (with internet connection ;) ).
Edit a cloud init file - which is basically a YAML that describes how the CoreOS machine is going to be installed & configured, at the very minimum it should contain the public key we have previously generated/copied:

vim cloud-init.yaml

#cloud-config

ssh_authorized_keys:
  - ssh-rsa AAAblablabla

Save and fetch CoreOS install script:

wget -O core-os-install.sh https://raw.githubusercontent.com/coreos/init/master/bin/coreos-install

Run the install script (this will wipe out /dev/sda of course):

bash ./core-os-install.sh -d /dev/sda -C stable -c cloud-init.yaml

When the installation is done, boot to the new CoreOS kernel and try to login as user 'core' with the public key provided in the YAML above.

MySQL on Docker

Numerous articles were written on how Docker is going to change the IT as we know it. Removing the need for full/para virtualization and configuration management frameworks.

While these statements are a bit exaggerated in my opinion, it seems that the technology is here to stay and is being rapidly adopted especially by the SaaS/web companies for obvious reasons such as portability and lower footprint than traditional Hypervisors.

In this short post I'd like to present a small "Howto" on running a MySQL (now MariaDB) DB on a Docker container, and present you some potential pitfalls ,so let's get started...

We will create a Docker file, which is our bootstrapping manifest for our DB image:



OK, so what we go here? We are pulling an Ubuntu image out of the Docker repository, installing the server and making sure the it is not bound to 'localhost', with some 'sed' magic, all in all pretty standard.

If more modifications were required for my.conf (and in real life scenario this would probably be mandatory), obviously 'sed' will be an ugly way to modify it so we could create a local copy of my.conf, make all the modification , add it to our Docker file and run the build process:

At this point we will be able to connect both from host and from other containers through a TCP socket:

But what about data persistence?  Remember that all the local data that is currently running in the container is ephemeral... While we could do something as:
This would delete our system data (tables with metadata), so what's the solution?

We need to add a wrapper script that will re-initialize the db in case there is no metadata available. The script can be added to the Docker file via 'ADD' statement:



That's better, now our DB runs on a persistent storage (/data/mysql - on the host machine which can be external SAN or NAS storage).

Enable/Disable HAproxy Backend Servers via Python

Sometimes we may want to automatically enable/disable machines behind our HAproxy automatically (during a deploy or maintenance), this is how it's done via Python code. The idea is to use HAproxy Unix Socket based API.

Optimizing Hadoop - Part1 (Hardware, Linux Tunings)

In these series of posts I'll share some of my experience with configuring Hadoop clusters for optimized performance and provide you with general guidance for efficiently optimizing your existing Hadoop cluster. 
I will start from the low level configurations/optimizations & tunings then we will cover OS level tunings, possible JVM tunings and finally the Hadoop platform level tunings.

Hardware Level configurations, tunings and checks:

Before we begin, it is extremely important to make sure our cluster nodes are aligned with their HW specs. Do all DataNode/TaskTracker nodes have the same amount of memory? Do all the DIMM's operate on same speeds? What about number of disks and their speed? What about NIC's speed? Are there any dropped packets? It is important that the actual number of installed DIMM's correspond to the number of channels per CPU, otherwise performance will be sub-optimal.

A good idea is to run some custom scripts combing commands such as 'dmidecode' , 'lspci', 'ifconfig', 'ethtool', 'netstat -s', 'fdisk -l', 'cat /proc/cpuinfo' with tool such as clustershell and make sure our nodes are indeed aligned and healthy. Mitigating low level (HW) issues is mandatory before we begin benchmarking our HW. 

Couple of things I would suggest checking -

• RAID StripeSize - Hadoop benefits most by running in JBOD mode, however certain controllers out there require each disk to be configured as separate RAID0 array , in that case you should tune the stripe size from 64K to 256K, this may have significant impact on the disk IO (I have observed ~25% performance boost while going up from 64K to 256K). Another thing is to enable write back mode if your controller has battery.
• Memory - disable power saving mode to increase memory frequency (usually from 1333 to 1600 Mhz) and throughput.
•  Limiting NIC interrupt rate significantly reduce context switching during the shuffle/sorting phase (where network load is highest) - a good idea will be to consult your vendor how to achieve this.

After we are sure our cluster nodes are aligned for our planning and do not suffer from any HW issue/anomaly, we can continue and conduct the appropriate HW performance tests:

Memory tests -   Stream is a great tool that will help you measure memory bandwith per node, nowdays  Xeon CPU's with 8 Channels per CPU and 1600MHz DIMMs can deliver 70-80GB/sec "Triad" results.

Network tests - Should be conducted from each node, to each node sequentially and as well as concurrently with tool such as 'iperf', you should expected about 90% of NIC BW, meaning 115MBps for 1 GBit or 1150MBps for 10Gbit network.

Disk tests - Tools such as IOzone will help to benchmarks our disks. Current 10K RPM SAS disks achieve optimally about ~170 MB/sec and 7.2K RPM SATA can reach ~140MB/sec for sequential reads/writes, random reads/writes will be roughly as half.

Have you found any sub-optimal performance on any of components above? Perhaps there is still a HW level issue that needs to be solved before diving into higher hierarchy optimizations.


Linux Tunings

Kernel parameters -

At minimum, we do not want our cluster to ever swap, we also want to decrease number of TCP re-transmit retries (we do not want to keep re-transmitting to faulty nodes) ,this setting is not recommended for multi-tenant (cloud environments) with higher latency + higher possible error rate.
It's also a good idea to enable memory over-committing ,since Hadoop processes tend to reserve more memory than they actually use, another important tuning is increasing somaxcon, which is a socket backlog - to be able to deal with connections bursts.

echo 'vm.swappiness  =  0'  >>  /etc/sysctl.conf
echo 'net.ipv4.tcp_retries2 = 2' >> /etc/sysctl.conf
echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf
echo 'net.core.somaxconn = 4096' >> /etc/sysctl.conf
sysctl -p


OS limits -

Linux defaults limits are too tight for Hadoop, make sure to tune limits for user running Hadoop services:

hadoop - memlock unlimited
hadoop - core unlimited
hadoop - nofile 65536
hadoop - nproc unlimited
hadoop - nice -10
hadoop - renice -10

File-system tunings -

Make sure your /etc/fstab mount options for Hadoop disks are with 'noatime' parameter, the gain is that no metadata has to be updated per filesystem reads/writes improving IO performance.

/dev/sdc  /data01  ext4  defaults,noatime  0 0
/dev/sdd  /data02  ext4  defaults,noatime  0 0
/dev/sde  /data03  ext4  defaults,noatime  0 0
/dev/sdf  /data04  ext4  defaults,noatime  0 0

Also, make sure to reclaim filesystem blocks that are by default set to be reserved to be used by privileged processes. By default 5% of total filesystem capacity is reserved. This is important especially on big disks (+2TB), since a lot of storage space can be reclaimed -

tune2fs -m 0 /dev/sdc

Disable Transparent Huge Pages (RHEL6+) -

RHEL 6.x includes a feature called "transparent hugepage compaction" which interacts poorly with Hadoop workloads. This can cause a serious performance regression compared to other operating system versions on the same hardware, the symptom is very high kernel space (sys) CPU usage.

echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo  never  >  /sys/kernel/mm/redhat_transparent_hugepage/enabled

echo 'echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled' >> /etc/rc.local
echo 'echo  never  >  /sys/kernel/mm/redhat_transparent_hugepage/enabled' >> /etc/rc.local

Enable NSCD -

In environments synced to NIS/LDAP for central authentication, it's possible to enable NSCD daemon so user/group information will be retrieved from local cache and not from server.

Python - read configuration file

Consider the following configuration file which consists of section name and key values:

$ cat /opt/myapp/myapp.ini
[master]
host=host01
port=2181

The following code (myapp.py) will parse the configuration file extracting values by section + corresponding key:

#!/usr/bin/python
import ConfigParser
configParser = ConfigParser.RawConfigParser()
configFilePath = r'/opt/myapp/myapp.ini'
configParser.read(configFilePath)
myhost = configParser.get('master', 'host')
myport = configParser.get('master', 'port')
print "Checking host:"+myhost+",port:"+myport


Run:

$ ./myapp.py
checking host:host01,port:2181

Unattended backups for Cisco appliances using scp

It's a good practice to keep your Cisco running configuration backed up to a remote backup repository on a regular basis, most convenient way I have found is using 'archive' function in the IOS and transferring the configuration over 'scp':
 
router01#conf t
router01(config)#archive
router01(config-archive)#path scp://bkpadmin:passw0rd@10.10.1.10//backup/bkp-$h-$trouter01(config-archive)#time-period 720
router01(config-archive)#do wr

Where:

• 10.10.1.10 - is my backup server
• bkpadmin/passw0rd - my remote user credentials.
  
• $h - is the hostname of the appliance
• $t - is the backup time stamp
• Backup time interval is specified in minutes so in my case the backup occurs twice a day (1440 minutes=24h).

Your running-config will be saved in file such as:

#ls /backup
bkp-router01-Jun-30-11-27-15.585-0

AWS VPC port forwarding techniques

Port forwarding using 'iptables' is extremely useful for ad-hoc interactions with your instances located on the private subnet on the VPC in situations when you do not wish to re-design your network architecture. 
As you must already know the instances on private subnet are not able to interact with the external world unless configured to use a NAT instance (located on the public subnet) as their GW.

So, for the example, let's say I want to forward any requests coming from the outside world to port 8080 via my NAT instance Elastic IP (which is an external, routable IP address) to an instance located on my private subnet - Puppet Master server, so:

• My NAT instance external IP address (Elastic IP) is:123.123.123.123
• My NAT instance internal IP address is:10.0.0.254
• My Puppet  Master internal IP address is:10.0.1.239

First, on the NAT instance make sure IP forwarding is enabled:
[root@ip-10-0-0-254 ~]#cat /proc/sys/net/ipv4/ip_forward
1
[root@ip-10-0-0-254 ~]#
We are good to go....
Next, we will instruct to redirect any requests coming to port 8080 to IP 10.0.1.239 port 8080: 
 
[root@ip-10-0-0-254 ~]# iptables -t nat -i eth0 -I PREROUTING -p tcp --dport 8080 -j DNAT --to 10.0.1.239:8080

Note, that in some cases you will want to limit this function only for incoming traffic, since the above example will forward any requests (even from inside the VPC) destined for port 8080, the best solution is to specify the destination IP address of the NAT instance -

[root@ip-10-0-0-254 ~]# iptables -t nat -d 10.0.0.254 -I PREROUTING -p tcp --dport 8080-j DNAT --to 10.0.1.239:8080

Pay attention that I've specified the NAT internal IP address. The reason for that is because the destination IP of the packet is in fact NAT instance internal IP - that's because Amazon EC2 already use NAT when correlating between elastic IP's and instance internal IP addresses.

Verify the command worked with:

[root@ip-10-0-0-254 ~]#iptables -L -t nat -v


Save your iptables configuration:
[root@ip-10-0-0-254 ~]#iptables-save > fw_conf_`date +%F`
[root@ip-10-0-0-254 ~]#/etc/init.d/iptables save

Make sure the security group your NAT instance is currently using allows relevant incoming traffic.

Finally, test the connection from outside of the VPC (make sure traffic is not blocked by any security group):

>telnet 123.123.123.123 8080

Your request now should be be redirected to the back-end node on private subnet on the VPC.

Cheers.

Checkpoint FW :Failed to load Policy on Module

While not being exactly a security expert once in a while I have to deal with some security appliances especially in test environments (where FW rules need to be adjusted quite frequently).

Most of the time I was extremely pleased with Checkpoint products ,at least for me their products were rock solid - until one day I wasn't able to install new policies on my Checkpoint FW.  The symptom was quite awkward - after saving the policy and verifying it successfully, during the installation process I always got an error saying "Installation failed. Failed to load Policy on Module" no matter what I tried, no additional info was specified which complicated things a bit.

Here is my workaround for the problem:

After you've logged in into the appliance as admin user (either via console or ssh) ,type:
# Expert

In order to get into privileged (Expert) mode (which basically allows you to work as "root" user on the appliance , as it was a regular Linux box).

After you got into expert mode the prompt will change to:
 [Expert@firewall]#

Now, you need to locate the "fwm" process (which is the FW management), kill it and then restart it.

Please note that if your SmartDashboard (or any other Checkpoint applications) are connected to the FW ,it will terminate them, yet the FW traffic (including any established VPN connections) will not be affected, so proceed without worries:

[Expert@firewall]#ps -ef|grep fwm
[Expert@firewall]#kill fwm-pid
[Expert@firewall]#fwm &

After fwm was started successfully on your FW box, try installing the policy again - usually this should do the trick.

If restarting fwm did not help, as a last resort  only, you will need to restart the CP services.
This will of course disconnect any sessions and every established  VPN connections, so think twice before executing it:

[Expert@firewall]#cpstop && cpstart.

The CP restart process takes around ~1 minute during which the FW may seem unresponsive.


This did the trick for me and I hope it helped some one out there too.
If you have more elegant solution for this issue, please let me know.
 
Cheers. 

XenServer 5.6 FP1 VDI Issue

Recently I ran into situation where one of my production VM's  went unresponsive,  the VM console showed various VDI related (xvda) I/O errors and the machine was halted.

It's worth mentioning that my XenServers (5.6 FP1) nodes operate in pool mode, and the problematic VDI resided on iSCSI LUN which seemed to be OK.

There weren't much other options rather than shutting down the VM with:

#xe vm-shutdown vm=vm0001 force=true

However, when I've tried to power on the VM back I got this nasty error:
18-Mar-12 9:42:16 AM Error: Starting VM 'vm0001' - Internal error: Failure("The VDI e17e2406-dbe9-40f6-98c3-af470e8aa91b is already attached in RW mode; it can't be attached in RO mode!")

Here is the workaround which did the job for me:

1. Find the UUID of the Storage Repository and the VM problematic VDI.

#xe sr-list |grep -i -C2 'your LUN name'
#xe vdi-list |grep -i -C2 'vdi name'

2. Next, we need to to remove VDI from the listing:

 #xe vdi-forget uuid=

Do not worry about the contents of the VDI they are fine :)

Verify the VDI is indeed gone:

#xe vdi-list |grep -i 'vdi name'

3. It's time to re-scan the storage repository that hosts the VDI via:
# xe sr-scan sr-uuid=

4. Verify the VDI is back in the listing:
#xe vdi-list sr-uuid=

Please note that the "name" and "description" fields are now empty.

5. Use XenCenter to reattach the VDI to your VM , and start it on different XenServer host inside your pool (right click on the VM, select "storage"->"attach"->).


This should do the magic.


Cheers!

Howto resize XenServer LUN's Online

In the following procedure I will show how to extend iSCSI attached LUN's on XenServer (v 5.6 SP2 in my case) on the fly ,so no service restart or downtime are needed and the VDI's that reside on the resized LUN are not affected, read on!

First of all on the storage side (NetApp filer in my case)list the available LUN's, the LUN I want to resize is  called "my_lun30", with LUN ID of 22, which is currently allocated with 90GB.

For the test I will extend it by another 10GB, making it a 100GB LUN.

filer1> lun show
        /vol/vol1/my_lun10         300.0g (322163441664)  (r/w, online, mapped)
        /vol/vol2/my_lun20    200g (214748364800)  (r/w, online, mapped)
        /vol/vol2/my_lun30       90.0g (107388862464)  (r/w, online, mapped)


filer1> lun resize /vol/vol2/my_lun30 +10g
lun resize: resized to:  100.0g (107388862464)


As you can see, my_lun30 is now 100GB:

filer1> lun show
        /vol/vol1/my_lun10         300.0g (322163441664)  (r/w, online, mapped)
        /vol/vol2/my_lun20    200g (214748364800)  (r/w, online, mapped)
        /vol/vol2/my_lun30        100.0g (107388862464)  (r/w, online, mapped)

We are done with the storage side, let's head to the XenServer side.
 In case you work in pool mode, login to the pool master as root:

I suggest installing "lsscsi" which provides a nice way of viewing SCSI attached disks/LUNs:

[root@xen3]#yum install lsscsi -y

[root@xen3]# lsscsi
[0:0:0:0]    cd/dvd  Optiarc  DVD RW AD-7561S  AH52  /dev/scd0
[2:0:0:0]    disk    NETAPP   LUN              7340  /dev/sda
[2:0:0:22]   disk    NETAPP   LUN              7340  /dev/sdb
[2:0:0:33]   disk    NETAPP   LUN              7340  /dev/sdc
[3:0:0:1]    disk    NETAPP   LUN              7340  /dev/sdd
[3:0:0:3]    disk    NETAPP   LUN              7340  /dev/sde

If YUM traffic to a repository is blocked - It's also possible to see the SCSI id's under /proc via:

# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: Optiarc  Model: DVD RW AD-7561S  Rev: AH52
  Type:   CD-ROM                           ANSI  SCSI revision: 05
Host: scsi2 Channel: 00 Id: 00 Lun: 00
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 22
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi2 Channel: 00 Id: 00 Lun: 33
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 01
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04
Host: scsi3 Channel: 00 Id: 00 Lun: 03
  Vendor: NETAPP   Model: LUN              Rev: 7340
  Type:   Direct-Access                    ANSI  SCSI revision: 04

From the output we can see the SCSI ID of the LUN + it's corresponding device on the system, and check the device physical size, as you can see it is not updated yet:

[root@xen3 backup_scripts]# fdisk -l /dev/sdb

Disk /dev/sdb: 96.6 GB, 96647249920 bytes
255 heads, 63 sectors/track, 11750 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Since XenServer uses LVM, we can see the physical volume info of the disk, as you can see the size is still 90GB.
  
[root@xen3]# pvdisplay /dev/sdb
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               VG_XenStorage-e71389d1-4dc3-2518-aa30-9f5f0c70ba12
  PV Size               90.01 GB / not usable 6.12 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              23039
  Free PE               16624
  Allocated PE          6415
  PV UUID               oSZlnA-VxlA-3qbp-07nI-Ql0b-3cG2-9wo2lC


Now, we will tell XenServer to rescan the SCSI bus, we will provide the SCSI id which we previously got from the "lsscsi" command (2:0:0:22):

[root@xen3]# echo 1 > /sys/class/scsi_disk/2:0:0:22/device/rescan

You can notice the immediate change of /dev/sdb:

[root@xen3]# fdisk -l /dev/sdb

Disk /dev/sdb: 107.3 GB, 107388862464 bytes
255 heads, 63 sectors/track, 13055 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Now, resize the physical volume with:

[root@xen3]# pvresize /dev/sdb

  Physical volume "/dev/sdb" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

Check again the physical volume size, notice the change:

[root@xen3]#  pvdisplay /dev/sdb
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               VG_XenStorage-e71389d1-4dc3-2518-aa30-9f5f0c70ba12
  PV Size               100.01 GB / not usable 6.12 MB
  Allocatable           yes
  PE Size (KByte)       4096
  Total PE              25600
  Free PE               19185
  Allocated PE          6415
  PV UUID               oSZlnA-VxlA-3qbp-07nI-Ql0b-3cG2-9wo2lC

Let's get the SR uuid, I know the LUN ID is 22, so:

[root@xen3]# xe sr-list|grep lun22 -B1
uuid ( RO)                : e71389d1-4dc3-2518-aa30-9f5f0c70ba12
          name-label ( RW): iSCSI_filer1_lun22

Notice that the SR size has yet to be updated:

[root@xen3]#  xe sr-param-list uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12|grep physical-size
           physical-size ( RO): 96632569856

Now, finally update the relevant SR:

[root@xen3]#  xe sr-update uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12

And at last, the SR is updated with the correct new LUN size :

[root@xen3]#  xe sr-param-list uuid=e71389d1-4dc3-2518-aa30-9f5f0c70ba12|grep physical-size
           physical-size ( RO): 107374182400


You are done!

Install & Configure OpenVZ (CentOS 6.2)

OpenVZ is a great tool which offers a great virtualization solution with near zero overhead, thus offering great performance.
In this short tutorial I will show how to install it on CentOS 6.2 machine, read on:


1) Get the OpenVZ repository and update "yum":

#wget -P /etc/yum.repos.d http://download.openvz.org/openvz.repo

#rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ

#yum update

2) Install relevant packages

#yum install openvz-kernel-rhel6 vzctl vzquota bridge-utils -y

3) Modify relevant kernel (networking) settings to allow proper communication with the VPS'es:

#vi /etc/sysctl.conf

#add these lines for sysctl openvz configuration
net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=1
net.ipv4.icmp_echo_ignore_broadcasts=1 

net.ipv4.conf.default.forwarding=1
net.ipv4.conf.default.proxy_arp = 0
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.eth0.proxy_arp=1

Update the new kernel settings:

#sysctl -p

4) Reboot the machine:

#shutdown -r now

A new Kernel should appear (2.6.32-042stab044.17 in my case) in the Grub menu.
Boot into the new Kernel.

5) Check that a new interface (venet0) exists:

# ifconfig venet0
venet0    Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet6 addr: fe80::1/128 Scope:Link
          UP BROADCAST POINTOPOINT RUNNING NOARP  MTU:1500  Metric:1
          RX packets:161 errors:0 dropped:0 overruns:0 frame:0
          TX packets:182 errors:0 dropped:12 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:19255 (18.8 KiB)  TX bytes:15822 (15.4 KiB)

Also, check the the "vz" service is running:

#/etc/init.d/vz status
OpenVZ is running...

6) So far, so good - It's time to get some OS template. Let's get an Ubuntu 11.10 64bit template:

#wget http://download.openvz.org/template/precreated/ubuntu-11.04-x86_64.tar.gz -P /vz/template/cache

All templates come as archives and reside inside /vz/template/cache directory.

As a best practice it's a good idea to keep /vz on a separate partition (or a LUN), the partition needs to be big enough to sustain all the VPS'es that are about to be created, so do the calculation according to your needs.

7) Basic installation is done. You should be able to use the vz* commands and administer your VM's via the CLI.

For example to create a new VM out of the downloaded template use:

#vzctl create 1 --ostemplate ubuntu-11.04-x86_64 --ipadd 10.0.0.12 --hostname vz03

When 1 is the uid of the VPS.

After the creation, initialize the created VPS via:

#vzctl start 1

You can now enter into the VPS by simply SSH'ing into it or via the following command:

#vzctl enter 1




A very cool (and free) web management which I highly recommend is called OpenVZ Web Panel, can be easily installed via this command:

#wget -O – http://ovz-web-panel.googlecode.com/svn/installer/ai.sh | sh

After the installation, check that the OpenVZ web panel is listening on port 3000:

#lsof -i :3000

An initialization script is provided as part of the installation and is located under: /etc/init.d/owp

Once installed the web panel can be accessed from your browser via: 
http://your-ip:3000

The interface is minimalistic but very convenient and user friendly:

Note: Be sure to modify firewall settings on the hosting machine accordingly to allow access to port 3000.

Happy VZ'ing!

How to Install & Configure Cfengine (v3) on CentOS 6

In this short tutorial I will demonstrate how to quickly install and configure Cfengine (version 3.x) on CentOS 6 x64 box and also show a basic example of this great automation tool, let's get started:

First, install the needed pre-requirements (EPEL/RPMforge provide these):

#yum install openssl openssl-devel db4 db4-devel flex pcre pcre-devel openldap gcc -y

Next, download the latest version source (3.2.3 at the moment of writing) and install it:

#wget http://cfengine.com/source-code/download?file=cfengine-3.2.3.tar.gz -O /root/cfengine-3.2.3.tar.gz

#tar xvzf cfengine-3.2.3.tar.gz; cd cfengine-3.2.3/

#./configure

#make

#make install

Now we will create the needed folders for CF under /var to work as should ,we will  also copy all relevant binaries and configuration files:

#mkdir -p  /var/cfengine/{masterfiles,bin,inputs}
#cp -rp /usr/local/share/cfengine/masterfiles/*.cf /var/cfengine/masterfiles/
#cp -rp /usr/local/share/cfengine/masterfiles/*.cf /var/cfengine/inputs/
#cp -rp /usr/local/sbin/cf-* /var/cfengine/bin/

Test the installation with:

#cf-promises -v

On the last line output you should get:
cf3>  -> Inputs are valid

Now, in order to demonstrate Cfengine abilities we will create a new file under /tmp called "cftest" with premissions of 744 and owner root.

In order to do that we will create a special configuration input file under:
/var/cfengine/inputs, called cftest.cf

So, let's see what we got inside of the configuration file:

#cat /var/cfengine/inputs/cftest.cf

body common control
{
# Define a bundle sequence
bundlesequence => { "checkperms" };
# Include cfengine_stdlib.cf
inputs => { "cfengine_stdlib.cf" };
version => "1.0.0";
 }
bundle agent checkperms        
{
files:                   
"/tmp/cftest"
create => "true",                             
perms => m("744");
}

Let's verify it's syntax with cf-promise:
#cf-promise -f /var/cfengine/inputs/cftest.cf

We will see that there is no file called cftest under /tmp ,prior running cf-agent.

# ls -l /tmp/cftest
ls: cannot access /tmp/cftest: No such file or directory

Now for the run:

#cf-agent -f /var/cfengine/inputs/cftest.cf

Lets check what's under /tmp:
# ls -l /tmp/cftest
-rwxr--r-- 1 root root 0 Jan 14 01:14 /tmp/cftest

Great, the file was created with the right permissions, just as we wanted. 
This is just the most basic example, via Cfengine you can do much much more, more practical examples to come so stay tuned!

XenServer - Cannot Shut VM

Have you ever experienced  a situation where you could not shut one of your running XenServer running VM's?   In this brief article I'll present a workaround for this common problem...  From my personal experiences this problem is often caused when the domain of the specific VM is still active (this often happens as a result of inproper shutdown and/or storage cutoffs) - in such case the domain needs to be destroyed manually.  OK, so here's the problem - from the XenServer (v 5.6 SP1) dom0 CLI I'm trying to shut a VM (using the force option): #xe vm-reset-powerstate vm=xxxxxxx force=true
 However the output I get is:  The operation could not be performed because a domain still exists for the specified VM.
vm: f087dxxxxx (xxxxxx)
domid: 5  From the output we will need  two parameters :the domain id (5 in our case) and the UUID of the problematic VM , so write them down somewhere. Now we need to destroy the currently active domain, XenServer includes couple of good tools under /opt/xensource, the command that is relevant is:   #/opt/xensource/debug/destroy_domain -domid 5  Now, finally, we will be able shut the VM with:  #xe vm-reboot uuid=XXXX --force  And we're done.  Hope that helped some one out there. Cheers!

Perl SSH Log-in

Hi Folks, 

In the following post I will demonstrate a password-less login into Cisco appliance via the SSH protocol using a Perl script.

When combined with cron, it can be a great solution for saving your appliance configuration (show run) or checking status without the need to log-in into the appliance each time.

Pre requirements:

• Perl
• SSH client (duh!)
• Net-SSH-Perl module

On Red-Hat (and friends) the module can be obtained via:

#yum install perl-Net-SSH-Perl -y 

Be sure to have a proper repository installed such as rpmforge 

 

Our simple script connects to the Cisco appliance via SSH and runs "show run" command (this account has enabled privilege).
The script itself should look like this:

#!/usr/bin/perl -w
use strict;
use warnings;
use Net::SSH::Perl;
my $ssh = Net::SSH::Perl->new('hostname');
$ssh->login('username', 'password');
my($out) = $ssh->cmd("show run");
print $out;

Since the script contains your appliance username & password don't forget to remove permissions for others:

#chmod o-rwx script.pl

Now, let's run the script:

#./script.pl

Output:

Building configuration...

Current configuration : 8011 bytes
...more output ommited...

Works like charm!

Now, the only thing is left is to synchronize it with cron :)

Enjoy.

Howto configure LDAP with TLS

Configuring LDAP over TLS is a crucial step in order to achieve a secure directory based authentication.

In the following tutorial I'll demonstrate how to configure OpenLDAP with TLS.

For the demonstration both server & client I've used are CentOS 5.5 x86_64

My LDAP server is OpenLDAP v2.3.43

Let's get started.

Step1 - Generating and Signing Certificates:

First of all we need to generate and sign the OpenLDAP server certificates.

In this scenario we will both generate and sign our certificate (self signed), this solution is good for internal and usually small environments.
If you ever plan to use this in a big enterprise production environment you will need a proper CA, like Verisign or Terrena to sign the certificates.

Before continuing please make sure the open SSL package is installed.

Generate your private key and a certificate request form:

#cd /etc/openldap/cacerts

#openssl req -nodes -newkey rsa:2048 -keyout server.key -out server.csr  Sign the certificate with:
#openssl x509 -in server.csr -out server.crt -req -signkey server.key -days 3650
 

Now you got both private and public keys.

After the certificates have been generated, assign the correct permissions and ownerships:

#chown ldap server.*

#chmod 0400 server.key

#chmod 0644 server.crt

Step2 - Server Side:

The following two lines need to be added in your server configuration file
(/etc/openldap/slapd.conf):

TLSCertificateFile      /etc/openldap/cacerts/server.crt
TLSCertificateKeyFile   /etc/openldap/cacerts/server.key

Restart the LDAP server:

#/etc/init.d/ldap restart 

Step3 - Client Side:

Make sure the ssl parameter in your LDAP client configuration file (/etc/ldap.conf) looks like this:

ssl start_tls

Also, make sure /etc/openldap/ldap.conf includes the following parameters:

TLS_CACERTDIR /etc/openldap/cacerts

TLS_REQCERT allow

That's it, from this point your OpenLDAP is ready to be used over TLS.
Stay secure ;)

XenServer Backup Script

This handy bash script I wrote, enables Xen Server Administrators to create On-Line backups (Importing the VM into.xva file) while the backed up virtual machine is up and running.

I found this extremely useful for big production environments - where lots of VM's require an appropriate & efficient back-up solution and downtime is not an option.

The script is very simple and user friendly, it's executed as root from the dom0 machine and used as follows:

In order to list all the available VM's on the system:

#./xen_onlinebackup.sh -l

In order to back up a VM:

#./xen_onlinebackup.sh -p /path/to/backupdir -s vm-name

The script supports both interactive and unattended mode (-u flag), the later designed to be built into scripts as an automation solution for large production environments.

For any help/usage, just run:

#./xen_onlinebackup.sh -h

Tested succesefully on XenServer 5.6 FP1 & XenServer 5.6 SP2.

Any comments, feed-backs or suggestions for improvements are very welcomed.

 #!/bin/bash

#

#This script used to create XenServer VM's online backup #Importing the VM into .xva file #Written by Paul Podolny - July 2011

#Variables

COUNT=0

AUTO=0 #Used for unattended mode

VMLIST=`xe vm-list is-control-domain=false |grep -i name-label|awk -F": " '{print $2}'`

#Functions

list() {

echo "====================================="

echo "Listing available VM's on $HOSTNAME..."

echo "====================================="

for i in $VMLIST;do

      echo $i;let COUNT+=1

done

echo "================================="

echo "Total $COUNT Virtual Machines"

echo "================================="

exit 0

}

usage() {

cat << EOF

This script will backup your Xen VMs on-line:

------

Usage:

------

-h Help/Usage.

-l List VM's.

-u Run in unattended mode (No questions asked).

-s Server Name (Backed Up VM).

-p Path to backup.

--------

Example:

--------

$0 -s server01 -p /nfs/backupdir/ -u

EOF

exit 0

}

check_dir() {

echo "=============================================================="

echo "XenServer Online Backup"

echo "=============================================================="

if  [ -e ${BACKUPPATH}/${VMNAME} ];then

        echo "Directory ${VMNAME} on path ${BACKUPPATH} seem to exist, will use it."

else

      echo "Creating dir. ${VMNAME} on backup path: ${BACKUPPATH}"

      mkdir ${BACKUPPATH}/${VMNAME}

fi

echo "`date +%H:%M` Backing Up ${VMNAME}, please hold on...Do NOT interrupt with CTRL+C !!!"

echo "================================================="

}

################################

#MAIN Prog

################################

while getopts "s:p:lhu" flag ; do

        case $flag in

                l ) list;;

                s ) VMNAME=$OPTARG;;

                p)  BACKUPPATH=$OPTARG;;

                h ) usage;;

                u)  AUTO=1;;

                *)  usage;;

        esac

done

#Sanity check  - Make sure backup path + VM name specificed if [ -z ${VMNAME} ] || [ -z ${BACKUPPATH} ];then echo  "Error:Both backup path AND VM name must be specified, exiting..."

echo  "Use $0 -h  for help"

exit 1

fi

#If running in interactive mode

if [ ${AUTO} != "1" ];then

      echo "Going to back-up ${VMNAME} to ${BACKUPPATH} ,is this OK? [y/n]"

      read ANSWER

      if [ ${ANSWER} != "y" ] && [ ${ANSWER} != "Y" ];then

            echo "Exiting by users request..."

            exit 1
      fi
fi

      #else proceed with main program...

      else

            check_dir

            TEMPUUID=$(xe vm-snapshot vm=$VMNAME new-name-label=snap_tmp_`date +%F`)

            xe template-param-set is-a-template=false ha-always-run=false uuid=$TEMPUUID

            xe vm-export vm=${TEMPUUID} filename=${BACKUPPATH}/${VMNAME}/${VMNAME}_backup_`date +%F`.xva

            echo "Removing Temporary Snapshot....(DO NOT PANIC!)"

            xe vm-uninstall uuid=${TEMPUUID} force=true

            echo "==============================================================="

            echo "### Back Up of ${VMNAME} Completed!###"

            echo  "`date +%H:%M`"

#END OF SCRIPT