/tmp full! Load 3.50+! Apache down!

[SHSaeed]

Hello,

The wierdest thing happened today.. Suddently I noticed all sites on this server were down. I checked the status page in WHM and it showes that the server load was on 3.50+ and that /tmp was 100% full. I did a 'ls -al' in /tmp and this is part of what I got.. http://www.antinetscape.org/temp/tmp_full01.gif

What are those files? What program created those? Anyone know? As soon as I deleted all files in /tmp that started with &_&, the problem was solved. Sites were working and load went back down to normal.

 

[ecoutez]

I would check my Apache logs

If nobody owns the files, and nobody would normally have write permission in /tmp, I'd be concerned that you have a script going wild. Maybe grep through your httpd logs for &wrk&? And do the same for any other logs whose daemons run as nobody.

And yell with what you find - this might be something we all need to keep an eye out for.

- Jason

 

[SHSaeed]

The /tmp directory is available for everyone to write in. Apache uses it, PHP users it for sessions, uploads, etc..

 

[rpmws]

Often what happens is a script loops and th eoutput just continues to grow and grow which makes mod_gzip make a wrk file that grows and grows as well. Since it issn't apache that is down but mod_gzip busy here is what you need to do.

go in /tmp

rm * -f

Go in WHM restart Apache.

Do your best to find the user that crashed it. Often they will run a script and nothing comes back on the screen. He or she knows who he is on your server.

 

[SHSaeed]

That's true. I checked the mod_gzip configuration on the server and noticed there is no min/max limit, so now I've added these lines to it:

mod_gzip_minimum_file_size 1024
mod_gzip_maximum_file_size 20971520
mod_gzip_maximum_inmem_size 2097152

First line sets the minimum file size that is compressed to 1 KiloByte.
Second line set the maximum file size that is compressed to 20 MegaBytes (maybe a little too high, but still a lot lower than 1 GB what do you think?).
Third line sets the maximum in-memory file size compression to 2 MegaBytes.

Good configuration for a shared server?

 

[moronhead]

You can also create a directory in a non-busy disk partition for *.wrk files. You need to inform apache about this:

mod_gzip_temp_dir /home/gzip --& restart apache

Then a cron job in /etc/crontab removes any instances of exponentially growing *.wrk files:

0-58/2 * * * * root touch /home/gzip/tmp56.wrk
1-59/2 * * * * root rm /home/gzip/*.wrk

The first line will create a dummy file every 2 minutes in the temp directory. The second line makes sure any *.wrk files are removed with the same frequency.

Just ensure the dummy file creation and the file removal times don't overlap as in the example.

 

[HostIt]

Originally posted by moronhead
Just ensure the dummy file creation and the file removal times don't overlap as in the example.

Sorry to be a dummy, but why do you need to create the dummy file? And would it work just as well to have a cron job delete .wrk files every 2 minutes from /tmp?