Monday, March 3, 2014

Random Useful Stuff

In this short post I'll try to concentrate some useful Linux CLI wisdow I find myself using again and again:

Prepend timestamp before each command and append to log:

ps -U hadoop -opid=,vsize=,rss=,pcpu=,comm=,args| gawk '{ print strftime("%Y-%m-%dT%H:%M:%S"), $0; fflush(); }' >> $logfile

AWK - Check ascending incremental (by 1) order:

cat /tmp/servers| awk 'BEGIN{count=1}{if ($1 != count+1) print "MISS"}; {count=$1; print $0}'

AWK - Compare two columns strings:

cat /tmp/test | awk '{if ($1 != $2) print $1 " <= not equal to => " $2 ;else print $0 }'

Run 'top' in batch mode (useful for debug):

top -p 7459 -b -d 30 | awk -v "date=$(date)" '/7459/ {print date,$0}'

Find processes blocked on IO:

while [ 1 ] ;do ps aux|awk '{if ($8 ~ "D") print }';done

Quick GNUplot:

gnuplot -persist <(echo 'plot "testdata.txt" using 1:xticlabels(2) with boxes')

Clean Memory Cache:

sync && echo 3 > /proc/sys/vm/drop_caches

Share current directory quickly via web server (port 8000):

python -m SimpleHTTPServer

Check URL latency:

curl -L -o /dev/null -s -w {time_connect}:%{time_starttransfer}:%{time_total}\\n ""

Determine external IP:

dig +short

Diff local and remote file via 'Meld':

meld local_file.txt <(ssh host cat remote_file.txt)

Share a terminal:

script| tee -a /dev/pts/1 

Parallel ping (multiple hosts in a file):

echo $(cat hosts-file)|xargs -P0 -n1 ping -c1 -w1 

DIY Parallel SSH:

cat hosts-file |xargs -P0 -I host ssh host uname -r

Create an archive of log files:

find /path/to/dir -name '*.log' | tar -c --files-from=- | bzip2 > logs-`date +%F`.tar.bz2

Compressed MySQL dump over SSH:

mysqldump -u root -h host --all-databases |pigz | ssh user@host "cat > /tmp/dump.sql.pgz" 

Start a script with 'nohup' over SSH:

ssh myhost "nohup /path/to/script/ > /path/to/script/nohup.out 2> /path/to/script/nohup.err < /dev/null &" 

Remove old kernels (Ubuntu):
echo $(dpkg --list | grep linux-image | awk '{ print $2 }' | sort -V \| sed -n '/'`uname -r`'/q;p') $(dpkg --list | grep linux-headers | awk '{ print $2 }' | sort -V | sed -n '/'"$(uname -r | sed "s/\([0-9.-]*\)-\([^0-9]\+\)/\1/")"'/q;p') | xargs sudo apt-get -y purge

Copy all cron's from one machine to another over SSH:

 crontab -l |ssh myhost001 "crontab -"

Unattended SSH key generation:

ssh-keygen -q -t rsa -f ~/.ssh/id_rsa -N ""

Base conversion (Hex to Dec):

echo 'ibase=16;obase=A;FF' | bc

Check XML syntax:

xmllint --noout core-site.xml; echo $?

Pretty XML output:

xmllint --format yourxmlfile.xml

Check Puppet manifest syntax:

puppet parser validate /path/to/mainfest/init.pp

Check ERB template syntax:

erb -x -T '-' mytemplate.erb | ruby -c

Check YAML syntax:

ruby -e "require 'yaml'; YAML.parse('/path/to/file.yaml'))";

Check Python script syntax:

python -m py_compile /path/to/ 

Check Json syntax:

cat file.json | python -m json.tool

Java core diagnosis (stack trace):

jstack -J-d64 $JAVA_HOME/bin/java /opt/cores/java.core.7788 

Java core diagnosis (gdb):

gdb $JAVA_HOME/bin/java /opt/cores/java.core.7788 
(gdb) where

Close specific file descriptor (for example 123):

gdb -p PID

(gdb) call close(123)
$1 = 0