Our servers keep crashing

[Myacen]

Upon investigating what was using the cpu, it was this: /usr/local/apache/bin/httpd -DSSL

Anyone have any idea what it is? How we can disable it (if possible).

Processes keep locking up (httpd) causing the server to crash.

 

[jumpdomain]

Have you setup the resource limits for Apache?

 

[Myacen]

Yes we have, yet it still keeps hanging and eventually causing the server to crash. Is their anyway to remove ssl support completly

 

[shaun]

service httpd stop
service httpd start

if you want ssl support do service httpd startssl

/usr/local/apache/bin/httpd -DSSL does mean that it is a ssl process either. It just means that apache was started with ssl support. Go into the WHM and look at the apache status and see what domains have alot of connections to the server. Also codered/nimda have generated alot of load on server's here in the past, even though apache isnt affected by it, it still responds and sends the error page which can cause extra load. Add these lines to your httpd.conf

RedirectMatch ^.*\.(dll|ida).* & dev/null
RedirectMatch ^.*\cmd\.* & dev/null
RedirectMatch ^.*\root\.* & dev/null

 

[Myacen]

Unfortunatley, it's not working.

 

[shaun]

whats not working? the load isnt droping or some of the commands i gave you dont work?

 

[Myacen]

Processes are still hanging, and /usr/local/apache/bin/httpd -DSSL is still using all of both cpu's up.

if you want ssl support do service httpd startssl
service httpd stop
service httpd start

You gave me that command to start httpd with ssl support, can you start httpd with no ssl support?

 

[shaun]

yes

service httpd stop
service httpd start

that shoudl not start it with ssl support.

If it does try this

/usr/local/apache/bin/apachectl stop
/usr/local/apache/bin/apachectl start


the service httpd startssl was to start it back up with ssl support not to stop it.

sorry if i wasnt clear.

 

[Myacen]

This:

If it does try this

/usr/local/apache/bin/apachectl stop
/usr/local/apache/bin/apachectl start

Has seemed to fix it....

 

[Myacen]

It fixed it for five minutes then started hanging again. (httpd processes)

 

[ecoutez]

I know the topic is old, but...

I know the topic is old, but here are some thoughts:

1 - check your logs. Are any of them surprisingly large? Perhaps an access log or SSL_Engine log grew way too large, and Apache can't write to the log. Generally this shouldn't cause HTTPd to consume large amounts of CPU time, but it's worth checking.

2 - look at the tail of your Apache error_log. If it keeps failing, maybe something useful is being written there.

3 - Sanity check - is there a partition that's maxed out on the system? 'df -h' will show your partition allocation.

Hopefully this is long since solved!

- Jason

 

[shaun]

the reason it stoped for 5 min and started again is because that is your httpd server. sombody's site on your server must have alot of traffic or just has alot of disk activity.

Like i said before httpd -dssl just means that apache is started with ssl support and it does not mean that ssl is whats causing it. Add those lines i pasted above you your httpd.conf file and run

/usr/local/apache/bin/apachectl stop
/usr/local/apache/bin/apachectl startssl

also do this tail -f /usr/local/apache/logs/error_log if you see alot of codered activity then the stuff i pasted above will help.

Some times people have Bulitin board systems running that are written in perl or a perl script or somthing running that have the -W on the top line with the perl descriptor. ex: /usr/bin/perl -W this can cause a hell of alot of traffic. A client here did this to his BBS and caused a dedicated 1ghz server running 512 ram to crash.

If you want me to look at your server send you your root password and ip address to [email protected]

I can also add those lines if you'd like.

 

[rpmws]

I bet it started back up SSL becuase chkservd hit it and restarted it. I would go in and search for new certs that had a new timestamp on it and see if removing that call in httpd.conf for the newest cert would fix it.

 

[hst]

try this

try this
/etc/rc.d/init.d/httpd configtest
This should show if you have any error in your httpd.conf file that may be causing the problem.

You also will want to look at you /var/log/messages file to see if there may be an error in you named.conf file Maybe a slight problem there. You should see some errors comming up when you restart your services.

 

[Myacen]

In my messages folder I get alot of:

May 26 05:18:38 sydney kernel: APIC error on CPU1: 04(04)
May 26 05:18:38 sydney kernel: APIC error on CPU0: 02(02)

 

[Myacen]

Yep, just before the server crashes is:

Jun 1 00:27:33 sydney kernel: APIC error on CPU0: 08(02)
Jun 1 00:27:33 sydney kernel: APIC error on CPU1: 02(04)
Jun 1 00:27:33 sydney kernel: APIC error on CPU0: 02(02)
Jun 1 00:27:33 sydney kernel: APIC error on CPU1: 04(04)
Jun 1 00:27:39 sydney kernel: APIC error on CPU1: 04(02)

 

[shaun]

what distro are you using? also what kernel you using?


cat /etc/redhat-release
cat /proc/version

 

[Myacen]

The problem was filenames believe it or not. Someone runs a fan site and has filenames like this:

JC\ Chasez\ -\ Bring\ It\ All\ To\ Me-2.mp3

We suspended the users account, and all went fine.

Thanks for your help guys.

 

[shaun]

i really doubt it was the filenames... the client was probably running a mp3 site and everybody and their mother was hitting the site and downloading the songs. You should/could have easily seen this by going into the WHM and clicking on apache status.


Glad you got the problem solved.

 

[rinty]

[quote:0991161ff0][i:0991161ff0]Originally posted by shaun[/i:0991161ff0]

i really doubt it was the filenames... the client was probably running a mp3 site and everybody and their mother was hitting the site and downloading the songs.

[/quote:0991161ff0]

It was the file names. We removed all files (included jpg files) that had a \ in them. Unsuspend the website and server load is at normal levels. With the files (with \ in them) on the server - it made it crash.