tmp shows 29% full, but there's almost no data there!

[jols]

Running df shows tmp at 29% full:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/hda3 74794656 38328356 32666928 54% /
/dev/hda1 101089 18856 77014 20% /boot
none 513728 0 513728 0% /dev/shm
/dev/hdb1 76920416 54072544 18940464 75% /backupdrive
/usr/tmpDSK 247919 66799 168320 29% /tmp
/tmp 247919 66799 168320 29% /var/tmp


However, when I ls -lh in tmp, I get only this:

[email protected] [/tmp]# ls -lh
total 1.9M
drwxrwxrwt 2 root root 1.8M Nov 6 09:46 ./
drwxr-xr-x 23 root root 4.0K Nov 6 2006 ../
-rw------- 1 mailnull mail 55 Nov 6 09:45 ClamAVBusy.lock
lrwxrwxrwx 1 root root 30 Nov 6 2006 mysql.sock -> ../../var/lib/mysql/mysql.sock=
-rw------- 1 nobody nobody 0 Nov 6 09:43 sess_258f6082f7573d265193981a231b2cc1
-rw------- 1 nobody nobody 0 Nov 6 09:44 sess_2e9ffd6233449b259fb186ca864912b0
-rw------- 1 nobody nobody 0 Nov 6 09:43 sess_310fd4983f94721eb37ffa1bf6a3c750
-rw------- 1 nobody nobody 0 Nov 6 09:42 sess_3b4d78ac587c8f5d865e6ab224fd3a8f
-rw------- 1 nobody nobody 0 Nov 6 09:45 sess_5e3d640dbf61bc6ff359443febd9cb22
-rw------- 1 nobody nobody 0 Nov 6 09:45 sess_784f9c339e78ba4297e52cdf85068e1e
-rw------- 1 nobody nobody 0 Nov 6 09:45 sess_7c633b16707724e60702e7f83ff37bf7
-rw------- 1 nobody nobody 0 Nov 6 09:42 sess_eb0477cea57f0f9e402bbcf205f1bd35


Something is not quite adding up. Does have a clue about this?

 

[chirpy]

It's probably deleted files still open to a running process. See:

lsof | grep /tmp

 

[jols]

Thanks for this, but I am not sure what to do with the info presented here:

[email protected] [/tmp]# lsof | grep /tmp
mysqld 4091 mysql 6u REG 7,0 0 312 /tmp/ibMOK7QT (deleted)
mysqld 4091 mysql 7u REG 7,0 0 313 /tmp/ibHeD8Xp (deleted)
mysqld 4091 mysql 8u REG 7,0 0 314 /tmp/ibU3cLaW (deleted)
mysqld 4091 mysql 12u REG 7,0 0 315 /tmp/ibLVSrdB (deleted)
httpd 4731 root 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
bash 5667 root cwd DIR 7,0 1918976 2 /tmp
top 8633 root cwd DIR 7,0 1918976 2 /tmp
leechprot 12143 root 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15573 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15593 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15681 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15682 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15683 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15722 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15724 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15725 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15764 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15802 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15804 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15805 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15807 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15808 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15809 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
httpd 15810 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
bash 15952 root cwd DIR 7,0 1918976 2 /tmp
httpd 16240 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
lsof 16344 root cwd DIR 7,0 1918976 2 /tmp
grep 16345 root cwd DIR 7,0 1918976 2 /tmp
lsof 16346 root cwd DIR 7,0 1918976 2 /tmp



Should I start killing processes? e.g. kill 15952, kill 16344, kill 16345 etc?

 

[jols]

Okay, this is insteresting, I started just deleting processes. But when I deleted the four mysqld processes (currently under 19466) mysql crashed. Then I restarted mysql, and now we show the prcesses up and running, but with the (deleted) in the entry line, see below:


[email protected] [/tmp]# lsof | grep /tmp
bash 5667 root cwd DIR 7,0 1918976 2 /tmp
MailScann 6913 mailnull 16wR REG 7,0 55 73 /tmp/ClamAVBusy.lock
top 8633 root cwd DIR 7,0 1918976 2 /tmp
bash 15952 root cwd DIR 7,0 1918976 2 /tmp
mysqld 19466 mysql 6u REG 7,0 0 14 /tmp/ibfrkCb8 (deleted)
mysqld 19466 mysql 7u REG 7,0 0 107 /tmp/ibfWB5BJ (deleted)
mysqld 19466 mysql 8u REG 7,0 0 108 /tmp/ibR0ts9k (deleted)
mysqld 19466 mysql 12u REG 7,0 0 109 /tmp/ibd9X7JW (deleted)
MailScann 19862 mailnull 16w REG 7,0 55 73 /tmp/ClamAVBusy.lock
lsof 19863 root cwd DIR 7,0 1918976 2 /tmp
grep 19864 root cwd DIR 7,0 1918976 2 /tmp
lsof 19865 root cwd DIR 7,0 1918976 2 /tmp


And still I am getting 29% full even though I can only "see" about 1.8mb of data in /tmp:

[email protected] [/tmp]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 72G 37G 32G 54% /
/dev/hda1 99M 19M 76M 20% /boot
none 502M 0 502M 0% /dev/shm
/dev/hdb1 74G 52G 19G 75% /backupdrive
/usr/tmpDSK 243M 66M 165M 29% /tmp
/tmp 243M 66M 165M 29% /var/tmp

 

[chirpy]

Try restarting httpd and mysql

 

[jols]

Okay, just restarted mysql and httpd, and no change. I still get:

[email protected] [/tmp]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda3 72G 37G 32G 54% /
/dev/hda1 99M 19M 76M 20% /boot
none 502M 0 502M 0% /dev/shm
/dev/hdb1 74G 52G 19G 75% /backupdrive
/usr/tmpDSK 243M 66M 165M 29% /tmp
/tmp 243M 66M 165M 29% /var/tmp
[email protected] [/tmp]#

 

[chirpy]

It's probably a corrupt file system on /tmp then. You can recreate that partition with:

1. Stop all running processes access /tmp (also stop chkservd)

2. Delete /usr/tmpDSK

3. Run /scripts/securetmp

4. Restart stopped services

 

[jols]

Thanks for the advice chirpy, first however I want to make sure I have the proceedure down. I "think" it would probably go like this:

service chkservd stop
service httpd stop
service mysql stop
unmount /tmp (not sure about this one)
rm -rf /usr/tmpDSK
/scripts/securetmp
service httpd start
service mysql start
service chkservd start


Look correct? Would there be any other services to stop?

 

[chirpy]

Yes, that should be it (I had indeed missed out the /tmp umount - you should check if /var/tmp shows up in df, if it does, umount that as well). If you get an error that /tmp is busy when you try and umount it, redo the lsof command above to find out what has files open on it.