/ filling up, for the life of me can't see what it might be

[gbh]

Would appreciate any thoughts. Have been trawling posts for 2 days and not having any significant luck.
Only started getting partition full erors approx 12 hours after a reboot, has been fine for months prior to that.

root@tank [/]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             9.7G  9.5G   34M 100% /
tmpfs                 1.9G     0  1.9G   0% /dev/shm
/dev/sda1             291M  138M  138M  51% /boot
/dev/sda8             258G   13G  233G   6% /home
/dev/sda7             3.9G   73M  3.6G   2% /tmp
/dev/sda2             9.9G  6.8G  2.6G  73% /usr
/dev/sda3             9.9G  1.3G  8.2G  14% /var
xxx.xxx.xxx.xxx:/mnt/tank/skunkworks/tank.xxx.com
                      200G   23G  178G  12% /mnt/backups

backup is on / but it's linked to /home/backup but it's empty/not in use
I also have remote /mnt/backup and this is where backups are directed

nothing at all dramatic when doing a ls in /root

fstab looks like this

root@tank [~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Sun May 27 06:06:40 2012
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=a9374b4d-dbea-464d-9203-3cd6af039c46 / ext4 defaults,usrquota   1 1
UUID=dbbe179b-e6b9-460c-921d-a329f2c7ad5e /boot                   ext4    defaults        1 2
UUID=aec5f695-3cd8-4d55-bfa3-faae514e307d /home ext4 defaults,usrquota 1   2
UUID=5b760c4f-2f4b-4f1b-8db0-57fa099dfdbb /tmp                    ext4    loop,noexec,nosuid,rw         1 2
UUID=44a50a4e-bf87-4461-9dd6-5e715cb2a199 /usr  ext4 defaults,usrquota 1   2
UUID=c54bfa24-1af4-42b3-9b10-c613df8134c5 /var  ext4 defaults,usrquota 1   2
UUID=40485749-084c-4d93-9afe-909b302e2944 swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs  noexec,nosuid        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults,noexec,nosuid        0 0

also if I'm in / and do a

ls -lR | less

and then scroll through everything, nothing at all jumps out at me as an especially big file.

have done a

yum clean all

no mention of disk errors in /var/log/messages at all

lost+found is empty

I've been through pretty much everything and removed what I could.
I'm think I'm now getting snow blind

hoping someone here can make further suggestions?
I haven't tried a reboot yet, thinking that's next.

 

[quietFinn]

you can start with this:

du -h --max-depth=1 /

then you will see what directory is taking the most space.
Just ignore those that are mounted, i.e. /home /tmp /usr /var

 

[PenguinInternet]

Also check /home/cpbackuptmp/ as this can sometimes take up considerable working space whilst backups are being created, depending on the size of your sites

 

[gbh]

thanks for the help!

running du -h --max-depth=1 / now it got down to /tmp and then looks like it's either still working or died, will let it run and report back.

I also don't have a /home/cpbackuptmp/ dir at all anywhere on the machine.

 

[gbh]

well du -h --max-depth=1 / has been running for about 9 hours now and just seems to have stopped..
output is currently as below
any further thoughts greatly appreciated
the 15-20 meg of space I cleared up last night by removing a bunch of small stuff seems to have been filled again now
8+(

root@tank [/home]# du -h --max-depth=1 /
83M     /etc
7.6M    /bin
du: cannot access `/proc/29077/task/29077/fd/4': No such file or directory
du: cannot access `/proc/29077/task/29077/fdinfo/4': No such file or directory
du: cannot access `/proc/29077/fd/4': No such file or directory
du: cannot access `/proc/29077/fdinfo/4': No such file or directory
0       /proc
0       /sys
16K     /selinux
16K     /lost+found
1011M   /var
13G     /home
513M    /lib
8.3M    /root
168K    /dev
32M     /lib64
4.0K    /srv
0       /net
128M    /boot
6.7G    /usr
1.1M    /tmp

 

[gbh]

I tried unmounting the nfs mount /mnt/backups in case that had anything to do with it.
no change

I also just rebooted.
no change.


any further thoughts greatly appreciated.