/tmp fills up crashes apache (mod_gzip) need script

[rpmws]

I run mod_gzip.

about once a month someone will run a screwed up script that will send mod_gzip into a .wrk file building loop and in about 4 minues /tmp fills up and then apache crashes becuase it needs /tmp with mod_gzip.

Is there a way to make the chkservd (when it checks /tmp) drive to automatically delete all *.wrk files in /tmp and restart apache after that? It checks every 8 minutes. If it sees it .. it does warn us via email but does nothing else. In my case /tmp will be fine for 6 months untill these /wrk files go haywire. I have a script (php) I can run right now that will crash any mod_gzip server in 10 minutes.

What can be done ??.. I had my first downtime in 2 years today that was over 1 hour. I was away when it happened.

 

[feanor]

Why don't you write that script and put it in the root cron? This doesn't sound like something under the &chkservd& realm.

It sounds a bit more tailored to the specific setup of apache that you have there... that is, offering mod_gzip in itself is a whole soap opera of its own.
:p

Make a shell script that does what you mentioned and put it in the root cron for every 10 minutes, or whatever interval you'd like.

 

[rpmws]

I thought of that BUT those .wrk files fly in and out of /tmp all the time. It's only when one goes nuts and fills up /tmp (and it doesn't matter what size your /tmp is).. it fills it up and then apache crashes. So what I was thinking ..since chkservd already warns us about drive space problems every 8 minutes it could also clean it up for us. I don't want to just delete all the time these files. especially every 10 minutes. It would drive people crazy. You should see those thing getting written and deleted all the time in there. But I have to say man ... mod_zip is the best thing ever happened to my box.(other than this issue)

 

[rpmws]

Nick you are the programming GOD ... what do you think? can you make it when a /tmp 85% full warning message goes out just on /tmp ..can you make it so it will clean house in /tmp and restart apache? or make that option available in WHM? I know of 5 or 6 servers this happens on at times...and the bad thing is port 80 remote checks do not catch it. Apache shows as &up& but no pages serve to the browser so if you don't watch it .. it can be a costly downtime for apache. Other than that problem I love mod_gzip. We would only need this for /tmp .

 

[bdraco]

If I can get access to a server running mod_gzip (I personally don't use it), I can have cpanellogd nuke any stable .wrk files for you.

 

[rpmws]

How would you go about doing that?? I mean it would have to be fast . When these loops happen one of these .wrk files can grow to 1500mb in about 3 minutes when a loop happens. AS soon as /tmp fills up apache stops serving (well mod_gzip does) graphics will still work from apache.

I don't mind you doing this on my box. I have a php script I can run that will surely test it.

When you want to mess with it? ..or maybe send me the changes and directions and I will try it for you and let you know.

If you would just have it hunt for overgrown (growing) .wrk files and rm them that would work ..OR check for drive space there would work. Good idea to restart apache if it gets close to full. That's what I have to do.

Note about mod_gzip .. It saves me about $200 month in bandwidth and I have people that have signed up with me simply becuase they can see a difference. It is much more noticable on a 56K connection. Example. This same Open Bulletin Board I had tested on my box and we had a fair amount of content and similar posts (displayed) to browsers. On my box I could have a full page of posts on the screen in less than 1 second on 56k. When I visit here on this board each time I view a page it takes about 5 seconds to see a page. It is a huge difference man. I dare you to try it!!! also your board box is 10ms faster in network speed from here so it wasn't that either. I love it man

 

[bdraco]

[quote:51cd0018ac][i:51cd0018ac]Originally posted by rpmws[/i:51cd0018ac]

How would you go about doing that?? I mean it would have to be fast . When these loops happen one of these .wrk files can grow to 1500mb in about 3 minutes when a loop happens. AS soon as /tmp fills up apache stops serving (well mod_gzip does) graphics will still work from apache.

I don't mind you doing this on my box. I have a php script I can run that will surely test it.

When you want to mess with it? ..or maybe send me the changes and directions and I will try it for you and let you know.

If you would just have it hunt for overgrown (growing) .wrk files and rm them that would work ..OR check for drive space there would work. Good idea to restart apache if it gets close to full. That's what I have to do.

Note about mod_gzip .. It saves me about $200 month in bandwidth and I have people that have signed up with me simply becuase they can see a difference. It is much more noticable on a 56K connection. Example. This same Open Bulletin Board I had tested on my box and we had a fair amount of content and similar posts (displayed) to browsers. On my box I could have a full page of posts on the screen in less than 1 second on 56k. When I visit here on this board each time I view a page it takes about 5 seconds to see a page. It is a huge difference man. I dare you to try it!!! also your board box is 10ms faster in network speed from here so it wasn't that either. I love it man [/quote:51cd0018ac]

Do you have any idea _how_ a loop gets started ?

 

[rpmws]

not sure if it is really a loop but the script I have is designed to delete (PHP) a set of .jpg files that are thumbnails for a shopping cart. So you can reset the image sizes. It basically looks in a folder in the users public_html and deletes all files in there. As it does this it prints on the screen each file I deletes. Well this script did this .wrk problem when he moved his script to another folder. The path to teh images folder was set ../../thumbs and it needed to be ../thumbs. Basically it couldn't find the folder. The browser will just hang on you..the server stays up for other pages for about 5 minutes while in teh background loads go up to about 3 and a .wrk file starts growing into a monster. Then /tmp fills up and mod_gzip quits.. no pages from apache. Delete the .wrk files and restart apache and it's fixed. Happened to me again by accident last night. But I have seen this happen with other scripts trying to run.

 

[bdraco]

[quote:4a7259cf72][i:4a7259cf72]Originally posted by rpmws[/i:4a7259cf72]

not sure if it is really a loop but the script I have is designed to delete (PHP) a set of .jpg files that are thumbnails for a shopping cart. So you can reset the image sizes. It basically looks in a folder in the users public_html and deletes all files in there. As it does this it prints on the screen each file I deletes. Well this script did this .wrk problem when he moved his script to another folder. The path to teh images folder was set ../../thumbs and it needed to be ../thumbs. Basically it couldn't find the folder. The browser will just hang on you..the server stays up for other pages for about 5 minutes while in teh background loads go up to about 3 and a .wrk file starts growing into a monster. Then /tmp fills up and mod_gzip quits.. no pages from apache. Delete the .wrk files and restart apache and it's fixed. Happened to me again by accident last night. But I have seen this happen with other scripts trying to run.[/quote:4a7259cf72]

Can you get an strace of this happening ?

 

[rpmws]

sure...never used strace before ..looks like a cool tool.

The user would be nobody ..what would I do to get that data for you..what syntax do you like to see when I run strace.

I will crash it for you just for the hell of it it's kinda fun but not when you are 2 hours from the office.

 

[rpmws]

actually I just did it and learned something. If root deletes the .wrk (growing) file from /tmp the process dies .. mod_Gzip still runs and so does apache ..as long as /tmp doesn't fill completely up first. But everytime this has happened to me that same file will grow till it can't anymore. In 21 seconds it made it to 221MB just now. I closed the browser that was calling the bad script and did a &rm *.wrk& and it didn't return(the .wrk file). Apache and mod_gzip still working fine. It's only when it gets full when you need to restart apache.

 

[rpmws]

Hey Nick?? how about it man?

 

[xnull]

Not sure if it is any help, but we configure mod_gzip on a per domain basis. Not all clients want it.

And in our setup, we told it to not save work/temp files. We also have it using each person's own /home/user/tmp folder instead of the main /temp so it is using their disk space and doesn't interfere with other work files.

 

[rpmws]

Humm... interesting I like saving the $300 per month in bandwidth costs per server. I can't imagine why someone wouldn't want it but you are on to something with that /home/user/tmp thing thanks for the tip. What do I have to do to change all this?

 

[carperman]

Bit off topic but what setting do you use for mod_gzip? i could never get it to work, well it would work sort of but would never compress files just leave error messages in log, i tried to find out what eror was but never did.

details of your setup would be handy

 

[rpmws]

Hey Nick??

This just happened to me today again only this time it killed teh entire server somehow. .gradually all tcp/ip services stopped and I had to physically reoot a machine that had 195 days on it. I didn't clean out /tmp fast enough and restart apache. Have you gotten anywhere on this?

 

[bert]

[quote:c2c55d0646][i:c2c55d0646]Originally posted by xnull[/i:c2c55d0646]

And in our setup, we told it to not save work/temp files. We also have it using each person's own /home/user/tmp folder instead of the main /temp so it is using their disk space and doesn't interfere with other work files.[/quote:c2c55d0646]
Could you tell us how you do that? We have problems with a particular server. It has happened three times in three days. I have disabled mod_gzip now on that server. I wonder which scripts screw everything up and what could cause it

 

[rpmws]

In my case it was a php script that runs through a folder and deletes files (cleans out everything) the output on the page is like:
Removing 101412.jpg...REMOVED
Removing 101743.jpg...REMOVED
Removing 101985.jpg...REMOVED
Removing 101556.JPG...REMOVED
Removing no_img.jpg...REMOVED
Removing 101216.jpg...REMOVED
Removing 101824.jpg...REMOVED
Removing 102110.jpg...REMOVED
Removing 101475.jpg...REMOVED
Removing 101784.jpg...REMOVED
Removing 101757.jpg...REMOVED
Removing 101982.jpg...REMOVED

Completed successfully...




--------------------------------------------------------------

In my case if the path to the folder was wrong this script would cause mod_gzip to fill up /tmp in about 4 minutes. I told the user if he did it again I would shoot him in the head.

 

[Karl]

It has happen with us too. Its when a user writes a poor script with no error checking. In your case the script will hav elooped and printed a message such as &Invalid file handle& or &Invalid directory handle& etc. in an infinite loop because they didn't check if the directory existed first in their script. I have a cron job running to delete .wrk files every 10 mins at the moment to stop this happening until I can be bothered to write a shell script to check the drive space first then delete if needed.

 

[rpmws]

I am doing the same thing right now..only problem is if /tmp fills up completely apache will need to be restarted for pages to serve again. Glad to see I am not the only one