The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

/dev/loop0 (/tmp) is 100% full

Discussion in 'Workarounds and Optimization' started by SoN9ne, Apr 20, 2010.

  1. SoN9ne

    SoN9ne Member

    Joined:
    Apr 20, 2010
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    I have found many articles about this but have yet to see one that solves this problem...

    I have been receiving this message almost every month now.
    HTML:
    /dev/loop0 (/tmp) is 100% full
    I have already increased my /tmp directory to 1GB and am thinking about increasing it to 2GB... I am a little concerned since I noticed there are a few files that are unusually large and I have no clue as to what they are, I am hoping someone here will know.

    The files that are unusually large are: (filename - size)
    34bd161eccfacc7d-odbJw2 - 75582
    getID37NK1qg.7&brs\=32745 - 566079488 (This is huge...)
    getID3IUn0Af.7&brs\=6529 - 263851490
    getID3ljXWoI.6&brs\=4470 - 137348853

    Does anyone know what these files are from? Are these cpanel updates? Is it safe to delete these files?

    I hope someone can give me some insight and can help me isolate the issue and prevent it from happening again...

    Thanks,
    Jeremy
     
    #1 SoN9ne, Apr 20, 2010
    Last edited: Apr 20, 2010
  2. SoN9ne

    SoN9ne Member

    Joined:
    Apr 20, 2010
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Nvm I had been able to isolate the issue. It was getID3 from a wordpress install.
     
  3. vanessa

    vanessa Well-Known Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    817
    Likes Received:
    22
    Trophy Points:
    18
    Location:
    Virginia Beach, VA
    cPanel Access Level:
    DataCenter Provider
    One of your users is doing this. If you're running suPHP and it's a PHP script, the ownership of the files should tell you who is causing the problem. You can band-aid it all you want by increasing the tmp disk size, but you need to figure out who on your server is filling it up.
     
  4. chapsrulez

    chapsrulez Member

    Joined:
    Apr 3, 2003
    Messages:
    13
    Likes Received:
    0
    Trophy Points:
    0
    Jeremy.

    I have the same problem, i mean im running out of space in /tmp but havent found a way to unmount it add more space and mount it again.

    Could you please tell me how you did this.
    Thanks a lot
    /cas

    Nevermind, i already found how.
    http://adamstechblog.com/2009/11/03/resize-tmp-partition-on-cpanel/
     
    #4 chapsrulez, Jun 19, 2010
    Last edited: Jun 19, 2010
  5. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    620
    Likes Received:
    6
    Trophy Points:
    18
    I have same problem Getting these messages quite frequently. Not sure what is the cause and how to find it?
     
  6. Sparrow-Sean

    Sparrow-Sean Member

    Joined:
    Jul 15, 2008
    Messages:
    9
    Likes Received:
    0
    Trophy Points:
    0
    Location:
    Queanbeyan, Australia
    You can remove the files alternatively whilst you manage a mount:

    rm -rf *

    That should remove everything in the TMP file, you can always manage a CRONTAB -E command to delete the files of this directory, the TMP will eventually fill up and this is a possible solution to size increase.
     
  7. GaryT

    GaryT Well-Known Member

    Joined:
    May 19, 2010
    Messages:
    321
    Likes Received:
    3
    Trophy Points:
    16
    I would not recommend that as if the user ends up in the wrong directory.. then... well I can only imagen the outcome :eek: you do know MySql is there ?

    /tmp directory is more of less your session files and such, You can pretty much remove all, If you do the following:

    then
    you may see tons of files like sess_fea3b004782e544393d9453550ced004

    if you do then just run this:

     
  8. JeffP.

    JeffP. Well-Known Member

    Joined:
    Sep 28, 2010
    Messages:
    164
    Likes Received:
    9
    Trophy Points:
    18
    There is a utility called "tmpwatch" that should exist in a default install of CentOS. tmpwatch exists to handle the removal of "old" files from /tmp and other locations on the server. From the manpage for tmpwatch:

    Code:
    DESCRIPTION
    
           tmpwatch  recursively  removes files which haven’t been accessed for a given number of hours. Normally, it’s 
           used to clean up directories which are used for temporary holding space such as /tmp.
    
    tmpwatch runs via cron:

    Code:
    [root@cpj ~]# cat /etc/cron.daily/tmpwatch 
    flags=-umc
    
    /usr/sbin/tmpwatch "$flags" -x /tmp/.X11-unix -x /tmp/.XIM-unix \
            -x /tmp/.font-unix -x /tmp/.ICE-unix -x /tmp/.Test-unix 240 /tmp
    
    /usr/sbin/tmpwatch "$flags" 720 /var/tmp
    
    for d in /var/{cache/man,catman}/{cat?,X11R6/cat?,local/cat?}; do
        if [ -d "$d" ]; then
            /usr/sbin/tmpwatch "$flags" -f 720 "$d"
        fi
    done
    
    The first command without the excluded paths essentially looks like this:

    Code:
    /usr/sbin/tmpwatch -umc 240 /tmp
    
    The "240" is the number of hours, which would be 10 days. If you want to have files removed from /tmp more frequently, you may try changing 240 to a lesser value. This would be more preferred than running a cron job that runs "rm -rf" on everything inside /tmp, as that could interfere with the MySQL and PostgreSQL sockets, and will interfere with EasyApache while it's running.
     
  9. GaryT

    GaryT Well-Known Member

    Joined:
    May 19, 2010
    Messages:
    321
    Likes Received:
    3
    Trophy Points:
    16
    Or you can use your custom one and set it to run when you like which also cleans other things, Runs very good.

    Do the following:

    Add:

    Save and close:


    run this:
    run:
    add:
    save and close:

    reload / restart crond service:

    thats it

    This cron will then run the script every day and clear out the /tmp folder.

    Hope this helps ;)
     
    #9 GaryT, Dec 24, 2010
    Last edited: Dec 24, 2010
    lobo92 likes this.
  10. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    620
    Likes Received:
    6
    Trophy Points:
    18
    Thank You for nice example and tutorial.

    I realised that issue of quick flooding space of this /dev/loop0 is caused by sending high amount of emails using SMTP protocol.

    It reaches value like 99%+- and i get email, but its quickyl cleaned back to 3% usage..

    Can be affected anyohow webserver or websites hosted on this server?
    How to prevent it when we know SMTP is the cause?
     
  11. twhiting9275

    twhiting9275 Well-Known Member

    Joined:
    Sep 26, 2002
    Messages:
    538
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    cPanel's minimal tmp setup is horrific at best. With stuff climbing up there constantly, it should be increased from 512mb to at least a 2-3 gig partition. Here's how to do this:

    Step 1: Remove the old tmp space
    Code:
    service httpd stop
    service mysql stop
    rm -rf /usr/tmpDSK
    
    Step 2, edit the script used to create tmp space
    Code:
    nano /scripts/securetmp
    
    In there, look for
    Code:
    my $tmpdsksize     = 512000;    # Must be larger than 250000
    and replace it with
    Code:
    my $tmpdsksize     = 3072000;    # Must be larger than 250000
    
    Save (ctrl-X , then y when asked), and reboot the server
    Code:
    shutdown -r now
    
    The system will adjust temp sizes when it comes back up.

    For whatever reason, unmounting tmp is close to impossible, even if mysql and http aren't running. Your best bet is to follow the above (which I've done many times and never had a problem with)
     
  12. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    620
    Likes Received:
    6
    Trophy Points:
    18
    I have just followed above guide to resize /temp size from 512MB to 4GB. My server has 8 core cpu and 8GB RAM.

    After restart, it is 5 minutes and load of the server is very high and websites show very slow:

    LOAD: 45.89 (8 CPUs)

    After 10 minutes load is 8.12 (8 CPUs) and goes down......

    What can be the cause? PS: Is it good idea to have such big /tmp on this configuration? (Server is used for webhosting, there is plenty of free RAM, but requires alot of CPU)
     
    #12 postcd, Jan 20, 2011
    Last edited: Jan 20, 2011
  13. twhiting9275

    twhiting9275 Well-Known Member

    Joined:
    Sep 26, 2002
    Messages:
    538
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    The first reboot will cause your server to have to create the new partition, which is going to mean that it's going to be loaded down a bit. This will pass, just ride it out.
     
  14. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    620
    Likes Received:
    6
    Trophy Points:
    18
    Thanks. What happend is that LOAD went down under value 8, then i deduplicated one large file using shell. After it was done there was running full cpanel backup, that load is 40-50 more than one hour!

    Then i retrminated cpbackup, but loand do not went down.

    Restart of Apache and Mysql dont help, load is up. High was 80.

    What should i do please?

    Considering making smaller /tmp ?

    [​IMG]

    it gets better

    usually before change CPU was 3.00 (8) and RAM 12% used

    Edit: Load is back at 70 :((((
     
    #14 postcd, Jan 20, 2011
    Last edited: Jan 20, 2011
  15. twhiting9275

    twhiting9275 Well-Known Member

    Joined:
    Sep 26, 2002
    Messages:
    538
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    following the steps I posted above won't lead to massive increased load repeatedly, on a normally functioning hard drive. If, however your hard drive has issues, you're going to see that, no matter what. Check your server logs
     
  16. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    620
    Likes Received:
    6
    Trophy Points:
    18
    Cpu Was at 50.00 (8) more than hour.. so i changed size back to 512MB, CPU is high after restart, im going to sleep now. If anyone can help, please post there, i will be gratefull.
     
  17. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    620
    Likes Received:
    6
    Trophy Points:
    18
    Thank You, im total newbie, i must go sleep now... But load is now returning down..... at 12.00 (8) :D Thank you

    //EDIT 5minutes later: Server load 5.11 (8 CPUs) Im going to sleep peacefully, sweet dreams and long night :)

    //EDIT 8 minutes later, im good back at 1.5 (8) load.

    [​IMG]

    So this means, be carefull doing this change, choosing high value.. maybe

    //EDIT 20 minutes later: load is normal - Server load 0.70 (8 CPUs)
    Would be interested if anyone have idea where is the problem.
     
    #17 postcd, Jan 20, 2011
    Last edited: Jan 20, 2011
  18. twhiting9275

    twhiting9275 Well-Known Member

    Joined:
    Sep 26, 2002
    Messages:
    538
    Likes Received:
    15
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    Wrong. This is one of the first changes I make on all servers, because /tmp should never be stuck at 512mb. NEVER has this caused a problem, ever. Like I said, the only way this would cause repeatedly high loads is if you've got bad hardware , which will inevitably cause problems anywhere, anyways.

    Then don't attack and insult people trying to help you out, or their methods of doing things. I've already told you where the problem most likely lies, you need to evaluate your logs and go from there. The method I posted a couple posts back is not going to cause any problems, or high load on your server at all.
     
  19. postcd

    postcd Well-Known Member

    Joined:
    Oct 22, 2010
    Messages:
    620
    Likes Received:
    6
    Trophy Points:
    18
    Insulting: It is your point of view.

    PS: If anyone can tell or give a link to guide how to find out cause of filling /tmp, i will be very gratefull.
    Thank you
     
  20. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Hello postcd,

    You would need to post what you are seeing in /tmp when it is filling up. We cannot guess why it's filling up in your instance, since it can be entirely dependent on the server and what is running on the server. When you get the alerts, what files are taking up the most space in /tmp and who owns those files?

    It's possible that MySQL might be doing it. If you look in /var/lib/mysql/hostname.err location where hostname is the server's hostname and see the following type of entries for MySQL:

    Code:
    Incorrect key file for table '/tmp/#sql_4108_0.MYI'; try to repair it
    This would be indicative of MyISAM filling up /tmp with temporary large binary table date, and then running out of space. This is just one possible way /tmp could be filling up constantly (and it would then cause MySQL to have issues during those times as well).

    It could alternatively be something like EAccelerator data filling it up if you are using EAccelerator on the machine, since it writes large files to /tmp for users.

    It could be some user with a lot of activity on the site with a large number of user session files (not as frequently the case, but possible).

    I could go on and on about the possible causes. The way to find out the cause is to check /tmp when it is full with a simple "ls -lah" or sorting by size with "du -sk .[!.]* *|sort -n" for the commands.

    Thanks.
     
Loading...

Share This Page