Wednesday, June 26, 2013

Linux partition is still full after freeing up space - Workaround

Have you ever found that a partition was almost full , you've deleted some files that took space but 'df' still showed a fully utilized partition?

Use this workaround:


First, we need to find the PID's responsible for the file descriptors of the deleted files:


root@server01:~# ls -ld /proc/*/fd/* 2>&1 | grep '(deleted)'

lrwx------ 1 root     root     64 Apr 12 09:15 /proc/14249/fd/5 -> /tmp/tmpfBD2mbh (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/14250/fd/1 -> /tmp/tmpfBD2mbh (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/14250/fd/2 -> /tmp/tmpfBD2mbh (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/14252/fd/1 -> /tmp/tmpfBD2mbh (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/14252/fd/2 -> /tmp/tmpfBD2mbh (deleted)
l-wx------ 1 root     root     64 Apr 12 09:15 /proc/14252/fd/3 -> /var/log/migrate.log.2013-04-11 (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/19986/fd/5 -> /tmp/tmpfJIL1po (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/19987/fd/1 -> /tmp/tmpfJIL1po (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/19987/fd/2 -> /tmp/tmpfJIL1po (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/19989/fd/1 -> /tmp/tmpfJIL1po (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/19989/fd/2 -> /tmp/tmpfJIL1po (deleted)
l-wx------ 1 root     root     64 Apr 12 09:15 /proc/19989/fd/3 -> /var/log/migrate.log (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/30066/fd/5 -> /tmp/tmpfosfoa4 (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/30070/fd/1 -> /tmp/tmpfosfoa4 (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/30070/fd/2 -> /tmp/tmpfosfoa4 (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/30071/fd/1 -> /tmp/tmpfosfoa4 (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/30071/fd/2 -> /tmp/tmpfosfoa4 (deleted)
l-wx------ 1 root     root     64 Apr 12 09:15 /proc/30071/fd/3 -> /var/log/migrate.log (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/5728/fd/5 -> /tmp/tmpf1gJt6t (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/5729/fd/1 -> /tmp/tmpf1gJt6t (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/5729/fd/2 -> /tmp/tmpf1gJt6t (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/5731/fd/1 -> /tmp/tmpf1gJt6t (deleted)
lrwx------ 1 root     root     64 Apr 12 09:15 /proc/5731/fd/2 -> /tmp/tmpf1gJt6t (deleted)
l-wx------ 1 root     root     64 Apr 12 09:15 /proc/5731/fd/3 -> /var/log/migrate.log (deleted)
l-wx------ 1 root     root     64 Apr  9 10:20 /proc/967/fd/2 -> /var/log/samba/log.smbd.1 (deleted)
l-wx------ 1 root     root     64 Apr  9 10:20 /proc/967/fd/8 -> /var/log/samba/log.smbd.1 (deleted)

Get the PID's of the processes holding the file descriptors:

root@server01:~# ls -ld /proc/*/fd/* 2>&1 | grep '(deleted)'|awk '{print $9}' |awk -F/ '{print $3}'|sort -u

You can itereave over the PID's you have found and restart (or in this case - kill them):

root@
server01:~# for i in `ls -ld /proc/*/fd/* 2>&1 | grep '(deleted)'|awk '{print $9}' |awk -F/ '{print $3}'|sort -u`;do kill  $i ;done
The partition should be free again.

No comments: