The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

/tmp fills up crashes apache (mod_gzip) need script

Discussion in 'EasyApache' started by rpmws, Mar 31, 2002.

Page 1 of 2
  1. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    I run mod_gzip.

    about once a month someone will run a screwed up script that will send mod_gzip into a .wrk file building loop and in about 4 minues /tmp fills up and then apache crashes becuase it needs /tmp with mod_gzip.

    Is there a way to make the chkservd (when it checks /tmp) drive to automatically delete all *.wrk files in /tmp and restart apache after that? It checks every 8 minutes. If it sees it .. it does warn us via email but does nothing else. In my case /tmp will be fine for 6 months untill these /wrk files go haywire. I have a script (php) I can run right now that will crash any mod_gzip server in 10 minutes.

    What can be done ??.. I had my first downtime in 2 years today that was over 1 hour. I was away when it happened.
     
    #1 rpmws, Mar 31, 2002
  2. feanor

    feanor Well-Known Member

    Joined:
    Aug 13, 2001
    Messages:
    836
    Likes Received:
    0
    Trophy Points:
    316
    Why don't you write that script and put it in the root cron? This doesn't sound like something under the &chkservd& realm.

    It sounds a bit more tailored to the specific setup of apache that you have there... that is, offering mod_gzip in itself is a whole soap opera of its own.
    :p

    Make a shell script that does what you mentioned and put it in the root cron for every 10 minutes, or whatever interval you'd like.
     
    #2 feanor, Mar 31, 2002
  3. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    I thought of that BUT those .wrk files fly in and out of /tmp all the time. It's only when one goes nuts and fills up /tmp (and it doesn't matter what size your /tmp is).. it fills it up and then apache crashes. So what I was thinking ..since chkservd already warns us about drive space problems every 8 minutes it could also clean it up for us. I don't want to just delete all the time these files. especially every 10 minutes. It would drive people crazy. You should see those thing getting written and deleted all the time in there. But I have to say man ... mod_zip is the best thing ever happened to my box.(other than this issue) :)
     
    #3 rpmws, Mar 31, 2002
  4. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    Nick you are the programming GOD ... what do you think? can you make it when a /tmp 85% full warning message goes out just on /tmp ..can you make it so it will clean house in /tmp and restart apache? or make that option available in WHM? I know of 5 or 6 servers this happens on at times...and the bad thing is port 80 remote checks do not catch it. Apache shows as &up& but no pages serve to the browser so if you don't watch it .. it can be a costly downtime for apache. Other than that problem I love mod_gzip. We would only need this for /tmp .
     
    #4 rpmws, Mar 31, 2002
  5. bdraco

    bdraco Guest

    If I can get access to a server running mod_gzip (I personally don't use it), I can have cpanellogd nuke any stable .wrk files for you.
     
    #5 bdraco, Mar 31, 2002
  6. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    How would you go about doing that?? I mean it would have to be fast . When these loops happen one of these .wrk files can grow to 1500mb in about 3 minutes when a loop happens. AS soon as /tmp fills up apache stops serving (well mod_gzip does) graphics will still work from apache.

    I don't mind you doing this on my box. I have a php script I can run that will surely test it.

    When you want to mess with it? ..or maybe send me the changes and directions and I will try it for you and let you know.

    If you would just have it hunt for overgrown (growing) .wrk files and rm them that would work ..OR check for drive space there would work. Good idea to restart apache if it gets close to full. That's what I have to do.

    Note about mod_gzip .. It saves me about $200 month in bandwidth and I have people that have signed up with me simply becuase they can see a difference. It is much more noticable on a 56K connection. Example. This same Open Bulletin Board I had tested on my box and we had a fair amount of content and similar posts (displayed) to browsers. On my box I could have a full page of posts on the screen in less than 1 second on 56k. When I visit here on this board each time I view a page it takes about 5 seconds to see a page. It is a huge difference man. I dare you to try it!!! :) also your board box is 10ms faster in network speed from here so it wasn't that either. I love it man :)
     
    #6 rpmws, Mar 31, 2002
  7. bdraco

    bdraco Guest

    [quote:51cd0018ac][i:51cd0018ac]Originally posted by rpmws[/i:51cd0018ac]

    How would you go about doing that?? I mean it would have to be fast . When these loops happen one of these .wrk files can grow to 1500mb in about 3 minutes when a loop happens. AS soon as /tmp fills up apache stops serving (well mod_gzip does) graphics will still work from apache.

    I don't mind you doing this on my box. I have a php script I can run that will surely test it.

    When you want to mess with it? ..or maybe send me the changes and directions and I will try it for you and let you know.

    If you would just have it hunt for overgrown (growing) .wrk files and rm them that would work ..OR check for drive space there would work. Good idea to restart apache if it gets close to full. That's what I have to do.

    Note about mod_gzip .. It saves me about $200 month in bandwidth and I have people that have signed up with me simply becuase they can see a difference. It is much more noticable on a 56K connection. Example. This same Open Bulletin Board I had tested on my box and we had a fair amount of content and similar posts (displayed) to browsers. On my box I could have a full page of posts on the screen in less than 1 second on 56k. When I visit here on this board each time I view a page it takes about 5 seconds to see a page. It is a huge difference man. I dare you to try it!!! :) also your board box is 10ms faster in network speed from here so it wasn't that either. I love it man :)[/quote:51cd0018ac]

    Do you have any idea _how_ a loop gets started ?
     
    #7 bdraco, Apr 1, 2002
  8. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    not sure if it is really a loop but the script I have is designed to delete (PHP) a set of .jpg files that are thumbnails for a shopping cart. So you can reset the image sizes. It basically looks in a folder in the users public_html and deletes all files in there. As it does this it prints on the screen each file I deletes. Well this script did this .wrk problem when he moved his script to another folder. The path to teh images folder was set ../../thumbs and it needed to be ../thumbs. Basically it couldn't find the folder. The browser will just hang on you..the server stays up for other pages for about 5 minutes while in teh background loads go up to about 3 and a .wrk file starts growing into a monster. Then /tmp fills up and mod_gzip quits.. no pages from apache. Delete the .wrk files and restart apache and it's fixed. Happened to me again by accident last night. But I have seen this happen with other scripts trying to run.
     
    #8 rpmws, Apr 1, 2002
  9. bdraco

    bdraco Guest

    [quote:4a7259cf72][i:4a7259cf72]Originally posted by rpmws[/i:4a7259cf72]

    not sure if it is really a loop but the script I have is designed to delete (PHP) a set of .jpg files that are thumbnails for a shopping cart. So you can reset the image sizes. It basically looks in a folder in the users public_html and deletes all files in there. As it does this it prints on the screen each file I deletes. Well this script did this .wrk problem when he moved his script to another folder. The path to teh images folder was set ../../thumbs and it needed to be ../thumbs. Basically it couldn't find the folder. The browser will just hang on you..the server stays up for other pages for about 5 minutes while in teh background loads go up to about 3 and a .wrk file starts growing into a monster. Then /tmp fills up and mod_gzip quits.. no pages from apache. Delete the .wrk files and restart apache and it's fixed. Happened to me again by accident last night. But I have seen this happen with other scripts trying to run.[/quote:4a7259cf72]

    Can you get an strace of this happening ?
     
    #9 bdraco, Apr 1, 2002
  10. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    sure...never used strace before ..looks like a cool tool.

    The user would be nobody ..what would I do to get that data for you..what syntax do you like to see when I run strace.

    I will crash it for you just for the hell of it :) it's kinda fun but not when you are 2 hours from the office.
     
    #10 rpmws, Apr 1, 2002
  11. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    actually I just did it and learned something. If root deletes the .wrk (growing) file from /tmp the process dies .. mod_Gzip still runs and so does apache ..as long as /tmp doesn't fill completely up first. But everytime this has happened to me that same file will grow till it can't anymore. In 21 seconds it made it to 221MB just now. I closed the browser that was calling the bad script and did a &rm *.wrk& and it didn't return(the .wrk file). Apache and mod_gzip still working fine. It's only when it gets full when you need to restart apache.
     
    #11 rpmws, Apr 1, 2002
  12. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    Hey Nick?? how about it man?
     
    #12 rpmws, Apr 2, 2002
  13. xnull

    xnull Well-Known Member

    Joined:
    Sep 9, 2001
    Messages:
    156
    Likes Received:
    0
    Trophy Points:
    316
    Not sure if it is any help, but we configure mod_gzip on a per domain basis. Not all clients want it.

    And in our setup, we told it to not save work/temp files. We also have it using each person's own /home/user/tmp folder instead of the main /temp so it is using their disk space and doesn't interfere with other work files.
     
    #13 xnull, Apr 2, 2002
  14. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    Humm... interesting :) I like saving the $300 per month in bandwidth costs per server. I can't imagine why someone wouldn't want it but you are on to something with that /home/user/tmp thing :) thanks for the tip. What do I have to do to change all this?
     
    #14 rpmws, Apr 2, 2002
  15. carperman

    carperman Well-Known Member

    Joined:
    Feb 7, 2002
    Messages:
    150
    Likes Received:
    0
    Trophy Points:
    316
    Bit off topic but what setting do you use for mod_gzip? i could never get it to work, well it would work sort of but would never compress files just leave error messages in log, i tried to find out what eror was but never did.

    details of your setup would be handy
     
    #15 carperman, Apr 3, 2002
  16. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    Hey Nick??

    This just happened to me today again only this time it killed teh entire server somehow. .gradually all tcp/ip services stopped and I had to physically reoot a machine that had 195 days on it. I didn't clean out /tmp fast enough and restart apache. Have you gotten anywhere on this?
     
    #16 rpmws, Apr 8, 2002
  17. bert

    bert Well-Known Member

    Joined:
    Aug 21, 2001
    Messages:
    602
    Likes Received:
    0
    Trophy Points:
    316
    [quote:c2c55d0646][i:c2c55d0646]Originally posted by xnull[/i:c2c55d0646]

    And in our setup, we told it to not save work/temp files. We also have it using each person's own /home/user/tmp folder instead of the main /temp so it is using their disk space and doesn't interfere with other work files.[/quote:c2c55d0646]
    Could you tell us how you do that? We have problems with a particular server. It has happened three times in three days. I have disabled mod_gzip now on that server. I wonder which scripts screw everything up and what could cause it :(
     
    #17 bert, Apr 12, 2002
  18. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    In my case it was a php script that runs through a folder and deletes files (cleans out everything) the output on the page is like:
    Removing 101412.jpg...REMOVED
    Removing 101743.jpg...REMOVED
    Removing 101985.jpg...REMOVED
    Removing 101556.JPG...REMOVED
    Removing no_img.jpg...REMOVED
    Removing 101216.jpg...REMOVED
    Removing 101824.jpg...REMOVED
    Removing 102110.jpg...REMOVED
    Removing 101475.jpg...REMOVED
    Removing 101784.jpg...REMOVED
    Removing 101757.jpg...REMOVED
    Removing 101982.jpg...REMOVED

    Completed successfully...




    --------------------------------------------------------------

    In my case if the path to the folder was wrong this script would cause mod_gzip to fill up /tmp in about 4 minutes. I told the user if he did it again I would shoot him in the head.
     
    #18 rpmws, Apr 12, 2002
  19. Karl

    Karl Well-Known Member
    PartnerNOC

    Joined:
    Aug 10, 2001
    Messages:
    84
    Likes Received:
    1
    Trophy Points:
    308
    It has happen with us too. Its when a user writes a poor script with no error checking. In your case the script will hav elooped and printed a message such as &Invalid file handle& or &Invalid directory handle& etc. in an infinite loop because they didn't check if the directory existed first in their script. I have a cron job running to delete .wrk files every 10 mins at the moment to stop this happening until I can be bothered to write a shell script to check the drive space first then delete if needed.
     
    #19 Karl, Apr 17, 2002
  20. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    7
    Trophy Points:
    318
    Location:
    back woods of NC, USA
    I am doing the same thing right now..only problem is if /tmp fills up completely apache will need to be restarted for pages to serve again. Glad to see I am not the only one :)
     
    #20 rpmws, Apr 17, 2002
(You must log in or sign up to post here.)
Page 1 of 2
Loading...
Similar Threads - fills crashes apache
  1. Sam12

    What to do when /usr/local/cpanel fills up?

    Sam12, Aug 4, 2015, in forum: General Discussion
    Replies:
    3
    Views:
    349
    cPanelMichael
    Aug 4, 2015
  2. ElviCities

    SOLVED cpanel-dovecot-solr times out/crashes

    ElviCities, Mar 30, 2017, in forum: E-mail Discussions
    Replies:
    6
    Views:
    2,627
    craigedmonds
    Apr 13, 2017
  3. mdspidle

    Apache Crashes during account transfer

    mdspidle, Feb 4, 2017, in forum: Data Protection
    Replies:
    2
    Views:
    404
    cPanelMichael
    Feb 6, 2017
  4. joem

    Apache crashes often on new server

    joem, Dec 9, 2016, in forum: EasyApache
    Replies:
    1
    Views:
    416
    cPanelMichael
    Dec 14, 2016
  5. nhadsall

    mysql cpu usage high, and crashes vps

    nhadsall, Jan 22, 2016, in forum: Workarounds and Optimization
    Replies:
    2
    Views:
    816
    cPanelMichael
    Jan 25, 2016

Share This Page