The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

tmp shows 29% full, but there's almost no data there!

Discussion in 'General Discussion' started by jols, Nov 6, 2006.

  1. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    2
    Trophy Points:
    38
    Running df shows tmp at 29% full:

    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/hda3 74794656 38328356 32666928 54% /
    /dev/hda1 101089 18856 77014 20% /boot
    none 513728 0 513728 0% /dev/shm
    /dev/hdb1 76920416 54072544 18940464 75% /backupdrive
    /usr/tmpDSK 247919 66799 168320 29% /tmp
    /tmp 247919 66799 168320 29% /var/tmp


    However, when I ls -lh in tmp, I get only this:

    root@server [/tmp]# ls -lh
    total 1.9M
    drwxrwxrwt 2 root root 1.8M Nov 6 09:46 ./
    drwxr-xr-x 23 root root 4.0K Nov 6 2006 ../
    -rw------- 1 mailnull mail 55 Nov 6 09:45 ClamAVBusy.lock
    lrwxrwxrwx 1 root root 30 Nov 6 2006 mysql.sock -> ../../var/lib/mysql/mysql.sock=
    -rw------- 1 nobody nobody 0 Nov 6 09:43 sess_258f6082f7573d265193981a231b2cc1
    -rw------- 1 nobody nobody 0 Nov 6 09:44 sess_2e9ffd6233449b259fb186ca864912b0
    -rw------- 1 nobody nobody 0 Nov 6 09:43 sess_310fd4983f94721eb37ffa1bf6a3c750
    -rw------- 1 nobody nobody 0 Nov 6 09:42 sess_3b4d78ac587c8f5d865e6ab224fd3a8f
    -rw------- 1 nobody nobody 0 Nov 6 09:45 sess_5e3d640dbf61bc6ff359443febd9cb22
    -rw------- 1 nobody nobody 0 Nov 6 09:45 sess_784f9c339e78ba4297e52cdf85068e1e
    -rw------- 1 nobody nobody 0 Nov 6 09:45 sess_7c633b16707724e60702e7f83ff37bf7
    -rw------- 1 nobody nobody 0 Nov 6 09:42 sess_eb0477cea57f0f9e402bbcf205f1bd35


    Something is not quite adding up. Does have a clue about this?
     
  2. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    It's probably deleted files still open to a running process. See:

    lsof | grep /tmp
     
  3. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    2
    Trophy Points:
    38
    Thanks for this, but I am not sure what to do with the info presented here:

    root@server [/tmp]# lsof | grep /tmp
    mysqld 4091 mysql 6u REG 7,0 0 312 /tmp/ibMOK7QT (deleted)
    mysqld 4091 mysql 7u REG 7,0 0 313 /tmp/ibHeD8Xp (deleted)
    mysqld 4091 mysql 8u REG 7,0 0 314 /tmp/ibU3cLaW (deleted)
    mysqld 4091 mysql 12u REG 7,0 0 315 /tmp/ibLVSrdB (deleted)
    httpd 4731 root 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    bash 5667 root cwd DIR 7,0 1918976 2 /tmp
    top 8633 root cwd DIR 7,0 1918976 2 /tmp
    leechprot 12143 root 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15573 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15593 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15681 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15682 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15683 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15722 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15724 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15725 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15764 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15802 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15804 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15805 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15807 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15808 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15809 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15810 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    bash 15952 root cwd DIR 7,0 1918976 2 /tmp
    httpd 16240 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    lsof 16344 root cwd DIR 7,0 1918976 2 /tmp
    grep 16345 root cwd DIR 7,0 1918976 2 /tmp
    lsof 16346 root cwd DIR 7,0 1918976 2 /tmp



    Should I start killing processes? e.g. kill 15952, kill 16344, kill 16345 etc?
     
  4. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    2
    Trophy Points:
    38
    Okay, this is insteresting, I started just deleting processes. But when I deleted the four mysqld processes (currently under 19466) mysql crashed. Then I restarted mysql, and now we show the prcesses up and running, but with the (deleted) in the entry line, see below:


    root@server [/tmp]# lsof | grep /tmp
    bash 5667 root cwd DIR 7,0 1918976 2 /tmp
    MailScann 6913 mailnull 16wR REG 7,0 55 73 /tmp/ClamAVBusy.lock
    top 8633 root cwd DIR 7,0 1918976 2 /tmp
    bash 15952 root cwd DIR 7,0 1918976 2 /tmp
    mysqld 19466 mysql 6u REG 7,0 0 14 /tmp/ibfrkCb8 (deleted)
    mysqld 19466 mysql 7u REG 7,0 0 107 /tmp/ibfWB5BJ (deleted)
    mysqld 19466 mysql 8u REG 7,0 0 108 /tmp/ibR0ts9k (deleted)
    mysqld 19466 mysql 12u REG 7,0 0 109 /tmp/ibd9X7JW (deleted)
    MailScann 19862 mailnull 16w REG 7,0 55 73 /tmp/ClamAVBusy.lock
    lsof 19863 root cwd DIR 7,0 1918976 2 /tmp
    grep 19864 root cwd DIR 7,0 1918976 2 /tmp
    lsof 19865 root cwd DIR 7,0 1918976 2 /tmp


    And still I am getting 29% full even though I can only "see" about 1.8mb of data in /tmp:

    root@server [/tmp]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/hda3 72G 37G 32G 54% /
    /dev/hda1 99M 19M 76M 20% /boot
    none 502M 0 502M 0% /dev/shm
    /dev/hdb1 74G 52G 19G 75% /backupdrive
    /usr/tmpDSK 243M 66M 165M 29% /tmp
    /tmp 243M 66M 165M 29% /var/tmp
     
  5. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Try restarting httpd and mysql
     
  6. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    2
    Trophy Points:
    38
    Okay, just restarted mysql and httpd, and no change. I still get:

    root@skyline [/tmp]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/hda3 72G 37G 32G 54% /
    /dev/hda1 99M 19M 76M 20% /boot
    none 502M 0 502M 0% /dev/shm
    /dev/hdb1 74G 52G 19G 75% /backupdrive
    /usr/tmpDSK 243M 66M 165M 29% /tmp
    /tmp 243M 66M 165M 29% /var/tmp
    root@skyline [/tmp]#
     
  7. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    It's probably a corrupt file system on /tmp then. You can recreate that partition with:

    1. Stop all running processes access /tmp (also stop chkservd)

    2. Delete /usr/tmpDSK

    3. Run /scripts/securetmp

    4. Restart stopped services
     
  8. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    2
    Trophy Points:
    38
    Thanks for the advice chirpy, first however I want to make sure I have the proceedure down. I "think" it would probably go like this:

    service chkservd stop
    service httpd stop
    service mysql stop
    unmount /tmp (not sure about this one)
    rm -rf /usr/tmpDSK
    /scripts/securetmp
    service httpd start
    service mysql start
    service chkservd start


    Look correct? Would there be any other services to stop?
     
  9. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Yes, that should be it (I had indeed missed out the /tmp umount - you should check if /var/tmp shows up in df, if it does, umount that as well). If you get an error that /tmp is busy when you try and umount it, redo the lsof command above to find out what has files open on it.
     
Loading...

Share This Page