Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

tmp shows 29% full, but there's almost no data there!

Discussion in 'General Discussion' started by jols, Nov 6, 2006.

  1. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    3
    Trophy Points:
    168
    Running df shows tmp at 29% full:

    Filesystem 1K-blocks Used Available Use% Mounted on
    /dev/hda3 74794656 38328356 32666928 54% /
    /dev/hda1 101089 18856 77014 20% /boot
    none 513728 0 513728 0% /dev/shm
    /dev/hdb1 76920416 54072544 18940464 75% /backupdrive
    /usr/tmpDSK 247919 66799 168320 29% /tmp
    /tmp 247919 66799 168320 29% /var/tmp


    However, when I ls -lh in tmp, I get only this:

    root@server [/tmp]# ls -lh
    total 1.9M
    drwxrwxrwt 2 root root 1.8M Nov 6 09:46 ./
    drwxr-xr-x 23 root root 4.0K Nov 6 2006 ../
    -rw------- 1 mailnull mail 55 Nov 6 09:45 ClamAVBusy.lock
    lrwxrwxrwx 1 root root 30 Nov 6 2006 mysql.sock -> ../../var/lib/mysql/mysql.sock=
    -rw------- 1 nobody nobody 0 Nov 6 09:43 sess_258f6082f7573d265193981a231b2cc1
    -rw------- 1 nobody nobody 0 Nov 6 09:44 sess_2e9ffd6233449b259fb186ca864912b0
    -rw------- 1 nobody nobody 0 Nov 6 09:43 sess_310fd4983f94721eb37ffa1bf6a3c750
    -rw------- 1 nobody nobody 0 Nov 6 09:42 sess_3b4d78ac587c8f5d865e6ab224fd3a8f
    -rw------- 1 nobody nobody 0 Nov 6 09:45 sess_5e3d640dbf61bc6ff359443febd9cb22
    -rw------- 1 nobody nobody 0 Nov 6 09:45 sess_784f9c339e78ba4297e52cdf85068e1e
    -rw------- 1 nobody nobody 0 Nov 6 09:45 sess_7c633b16707724e60702e7f83ff37bf7
    -rw------- 1 nobody nobody 0 Nov 6 09:42 sess_eb0477cea57f0f9e402bbcf205f1bd35


    Something is not quite adding up. Does have a clue about this?
     
  2. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,460
    Likes Received:
    21
    Trophy Points:
    463
    Location:
    Go on, have a guess
    It's probably deleted files still open to a running process. See:

    lsof | grep /tmp
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    3
    Trophy Points:
    168
    Thanks for this, but I am not sure what to do with the info presented here:

    root@server [/tmp]# lsof | grep /tmp
    mysqld 4091 mysql 6u REG 7,0 0 312 /tmp/ibMOK7QT (deleted)
    mysqld 4091 mysql 7u REG 7,0 0 313 /tmp/ibHeD8Xp (deleted)
    mysqld 4091 mysql 8u REG 7,0 0 314 /tmp/ibU3cLaW (deleted)
    mysqld 4091 mysql 12u REG 7,0 0 315 /tmp/ibLVSrdB (deleted)
    httpd 4731 root 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    bash 5667 root cwd DIR 7,0 1918976 2 /tmp
    top 8633 root cwd DIR 7,0 1918976 2 /tmp
    leechprot 12143 root 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15573 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15593 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15681 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15682 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15683 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15722 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15724 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15725 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15764 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15802 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15804 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15805 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15807 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15808 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15809 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    httpd 15810 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    bash 15952 root cwd DIR 7,0 1918976 2 /tmp
    httpd 16240 nobody 4u REG 7,0 0 55 /tmp/ZCUDE2PHga (deleted)
    lsof 16344 root cwd DIR 7,0 1918976 2 /tmp
    grep 16345 root cwd DIR 7,0 1918976 2 /tmp
    lsof 16346 root cwd DIR 7,0 1918976 2 /tmp



    Should I start killing processes? e.g. kill 15952, kill 16344, kill 16345 etc?
     
  4. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    3
    Trophy Points:
    168
    Okay, this is insteresting, I started just deleting processes. But when I deleted the four mysqld processes (currently under 19466) mysql crashed. Then I restarted mysql, and now we show the prcesses up and running, but with the (deleted) in the entry line, see below:


    root@server [/tmp]# lsof | grep /tmp
    bash 5667 root cwd DIR 7,0 1918976 2 /tmp
    MailScann 6913 mailnull 16wR REG 7,0 55 73 /tmp/ClamAVBusy.lock
    top 8633 root cwd DIR 7,0 1918976 2 /tmp
    bash 15952 root cwd DIR 7,0 1918976 2 /tmp
    mysqld 19466 mysql 6u REG 7,0 0 14 /tmp/ibfrkCb8 (deleted)
    mysqld 19466 mysql 7u REG 7,0 0 107 /tmp/ibfWB5BJ (deleted)
    mysqld 19466 mysql 8u REG 7,0 0 108 /tmp/ibR0ts9k (deleted)
    mysqld 19466 mysql 12u REG 7,0 0 109 /tmp/ibd9X7JW (deleted)
    MailScann 19862 mailnull 16w REG 7,0 55 73 /tmp/ClamAVBusy.lock
    lsof 19863 root cwd DIR 7,0 1918976 2 /tmp
    grep 19864 root cwd DIR 7,0 1918976 2 /tmp
    lsof 19865 root cwd DIR 7,0 1918976 2 /tmp


    And still I am getting 29% full even though I can only "see" about 1.8mb of data in /tmp:

    root@server [/tmp]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/hda3 72G 37G 32G 54% /
    /dev/hda1 99M 19M 76M 20% /boot
    none 502M 0 502M 0% /dev/shm
    /dev/hdb1 74G 52G 19G 75% /backupdrive
    /usr/tmpDSK 243M 66M 165M 29% /tmp
    /tmp 243M 66M 165M 29% /var/tmp
     
  5. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,460
    Likes Received:
    21
    Trophy Points:
    463
    Location:
    Go on, have a guess
    Try restarting httpd and mysql
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    3
    Trophy Points:
    168
    Okay, just restarted mysql and httpd, and no change. I still get:

    root@skyline [/tmp]# df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/hda3 72G 37G 32G 54% /
    /dev/hda1 99M 19M 76M 20% /boot
    none 502M 0 502M 0% /dev/shm
    /dev/hdb1 74G 52G 19G 75% /backupdrive
    /usr/tmpDSK 243M 66M 165M 29% /tmp
    /tmp 243M 66M 165M 29% /var/tmp
    root@skyline [/tmp]#
     
  7. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,460
    Likes Received:
    21
    Trophy Points:
    463
    Location:
    Go on, have a guess
    It's probably a corrupt file system on /tmp then. You can recreate that partition with:

    1. Stop all running processes access /tmp (also stop chkservd)

    2. Delete /usr/tmpDSK

    3. Run /scripts/securetmp

    4. Restart stopped services
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. jols

    jols Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,111
    Likes Received:
    3
    Trophy Points:
    168
    Thanks for the advice chirpy, first however I want to make sure I have the proceedure down. I "think" it would probably go like this:

    service chkservd stop
    service httpd stop
    service mysql stop
    unmount /tmp (not sure about this one)
    rm -rf /usr/tmpDSK
    /scripts/securetmp
    service httpd start
    service mysql start
    service chkservd start


    Look correct? Would there be any other services to stop?
     
  9. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,460
    Likes Received:
    21
    Trophy Points:
    463
    Location:
    Go on, have a guess
    Yes, that should be it (I had indeed missed out the /tmp umount - you should check if /var/tmp shows up in df, if it does, umount that as well). If you get an error that /tmp is busy when you try and umount it, redo the lsof command above to find out what has files open on it.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice