Lost space on root partition

I was recently told by my computer that I'd run out of space on my Mandriva root partition. It's a 26GB partition and shouldn't be anywhere near full but still, this is a bit of a critical state to be in so I immediately investigated.

The first step was to see what the damage was, in this case using df -h:

[james@eastlight ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda9              26G   26G   51K 100% /
/dev/sda5             236M  202M   30M  88% /boot
/dev/sda7             332G  311G  4.6G  99% /home
/dev/sdb2             363G  277G   68G  81% /backups

As you can see, the root partition was indeed full so I tried to investigate where the large files were:

[james@eastlight /]$ sudo du -x -k --max-depth=1 | sort -n
0       ./dev
0       ./proc
0       ./sys
4       ./backups
4       ./home
7       ./boot
8       ./.config
8       ./initrd
8       ./mnt
16      ./lost+found
16      ./.Trash-500
36      ./media
6972    ./bin
14828   ./sbin
15012   ./tmp
17964   ./lib64
18676   ./root
30124   ./opt
41936   ./etc
158504  ./lib
457420  ./var
5342568 ./usr
6104135 .

Unfortunately, this didn't show me any problems and I was a bit stumped at this point. Some threads on the internet suggested using a graphical disk analyser such as xdiskusage which I installed and ran (sudo xdiskusage). Incidentally, if you get an error such as:

[james@eastlight /]$ sudo xdiskusage
No protocol specified
Can't open display: :0

try running:

[james@eastlight /]$ xhost +local:root
non-network local connections being added to access control list

to get it working again.

After running xdiskusage it became clear that there was some fundamental problem with the root partition as there was a large area of the disk (around 18GB) marked out as "(permission denied)". This was clearly not quite right as I was running as root. Investigation in the xdiskusage manual page revealed that the likely cause was some space that df said was used but du did not report. The further examples weren't helpful but I came across someone who was investigating a similar problem and they were having issues with mount points. My backup partition is on a separate disk but is mounted to /backups. As a hunch, I rebooted into single user mode and unmounted my backups folder (unmount /backups). I then changed into that directory and found about 18GB of backup files there!

Only then did I remember that I had disconnected the second hard drive for some time and not disabled the backup script. The script had duly written to /backups and instead of writing to the second hard drive, began filling up my root partition.

I removed the erroneous backup files, rebooted and hey presto:

[james@eastlight /]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda9              26G  6.0G   19G  25% /
/dev/sda5             236M  202M   30M  88% /boot
/dev/sda7             332G  311G  4.6G  99% /home
/dev/sdb2             363G  277G   68G  81% /backups

All back to normal. This was a real head-scratcher!

Published on 24th November 2011.