High load/process and named problems

[rsutc]

I have a CentOS 4 server with dual Opteron 246 processors that I keep on cPanel release. It also has the ConfigServer security package, suPHP, suhosin, latest PHP and Apache installed, and is fairly locked down.

Problem 1: Everything runs tickety-boo for weeks, even sometimes months at a relatively low load (<1 except when cpup running). Then out of the blue the number of processes suddenly shoots from 130 to 800+ and the load to 75+. Munin tells a tale afterward of VMStat and/or IOStat and/or memory usage also shooting up (not always exactly the same combo. The only way to recover is a reboot. Neither I nor the DC people see anything obvious in the logs. They think it may be a rogue script, but I have exercised most of the sites running scripts and don't see any multiplication of processes. Any suggestions where to look?

Problem 2: Every once in a while, the named files get corrupted. So far this has always been shortly after a major cPanel update. Any suggestions?

I did a look around here, but don't see anything specific to these issues.
Rick

[sanjuabrahamk]

Try to find the top memory usage process with top command and use the following command tp find the database if mysql is the top one.

mysqladmin -i2 processlist

[rsutc]

Thanks for the tip. It happened again. This time, I was barely able to ssh into the box, and on a whim I used
pkill httpd
to kill off all children of httpd. This axed some 600 of the 800 sleeping processes, telling me (I think) that one or more web sites are responsible. The load came down, but Apache was hooped, and I had to reboot twice to get everything running again. Can anyone recommend a next step in view of this new data?

Rick

[jols]

Sounds like it may be massive bot attacks. Try determining which domains are being hit with:

mysqladmin processlist

Then take the high load candidates and do this:

tail -1000 /usr/local/apache/domlogs/thedomain.com

Find the bot hits and start blocking their IPs using your firewall.

Then afterward, add some rules to your mod_security ruleset like this:

SecRule HTTP_User-Agent "Baiduspider"
SecRule HTTP_User-Agent "BecomeBot"
SecRule HTTP_User-Agent "ScoutJet"
SecRule HTTP_User-Agent "Gigabot"

and so on.

I have noticed a very large increase in the number of different bots hitting our servers over the past few months. When these things start hitting large WordPress sites with large numbers of page links, look out! I've seen a multi-processor server go from (load average) 5 to 150 or more in just a few minutes, strictly due to bot hits, specifically hitting MySQL driven pages.

[rsutc]

Sounds like it may be massive bot attacks.

This sounds like a promising approach. I do have a few domains with WordPress that periodically report medium numbers of running processes. However, in the logs, I see no bot explorations of note around the time these processes are reported.

Would you be willing to share your list of the bots you have restricted? Perhaps if I search for some of them by name, I might come up with something illuminating.

Is there also a way to limit the number of processes for any one site, or perhaps for Apache as a whole?

Does anybody think that increasing the memory would help? It's of course when the number of processes eats the main memory and gets into swap that we really get problems. But would increased memory merely delay the inevitable by a few seconds?

Rick

[rsutc]

With still more on this.

I have now discovered that suhosin is being blamed as a memory hog by people with similar problems. Moreover, php 5.2.10 has a leak that makes things worse. I have downgraded temporarily and find the system using far less memory and ramping up much more slowly. It seems that with suhosin each Apache child takes much more memory than without. Stress the system with lots of http accesses (as you would when a bot visits) and memory use spirals out of control. Once it hits swap, you're toast as all the indicators fly off the scale.

OK, now... For someone familiar with these issues:
1. does the current version of php 5.2.10 supplied by cpanel have the patch to fix the memory leak in strtotime?
2. Is there a way to configure suhosin to limit the memory it consumes without generating other errors?

Rick

Rick

[Snowman30]

With still more on this.

I have now discovered that suhosin is being blamed as a memory hog by people with similar problems. Moreover, php 5.2.10 has a leak that makes things worse. I have downgraded temporarily and find the system using far less memory and ramping up much more slowly. It seems that with suhosin each Apache child takes much more memory than without. Stress the system with lots of http accesses (as you would when a bot visits) and memory use spirals out of control. Once it hits swap, you're toast as all the indicators fly off the scale.

OK, now... For someone familiar with these issues:
1. does the current version of php 5.2.10 supplied by cpanel have the patch to fix the memory leak in strtotime?
2. Is there a way to configure suhosin to limit the memory it consumes without generating other errors?

Rick

Rick

other than downgrading or turning off suhosin have you found a solution and do you have any links to discussions or info on the suhosin leak issue?