Server Load and Runaway processes

[pguy]

Hello Folks,

Although we have had this issue before, this time it has had me stumped. The server load seems to spike to 150 or so and renders all unusable. Sites are not served and all goes down until a reboot or waiting for 10 - 15 minutes tends to bring the server load back to some usable level.


I've gone through all the logs and have found absolutely no clue. No formmails, no spam etc

At times I have noticed that httpd tends to start growing in size. Via the top process it can be seen taking up sizes above 200MB or so. I have checked access_logs , CPU usage from WHM and there is nothing weird at all !

Even the httpd, if restarted when the size starts growing, tends to function normally and brings down loads to around 1 or so.

Where else can i see ? I seriously am stumped and need some clues as to how I get about isolating the problem.

Warmest Regards,

cPanel.net Support Ticket Number:

 

[pguy]

Found the issue and boy was it tough :D

One of our clients had put in a Rewrite rule in .htacess which stated a certain page to load if a 404 occured. Here the page which was to be loaded was not found and this pushed apache into a loop. If you just hit any 404 page, it could take the server down ! <sheesh>


I really had thought apache would have been able to handle such an issue..

cPanel.net Support Ticket Number:

 

[hostultra]

How did you find it?

cPanel.net Support Ticket Number:

 

[Website Rob]

Nice catch pguy, and hat's off to someone's troubleshooting ability. I too, would be interested in knowing how it was done as runaway scripts can be a real 'bug-a-boo' for any ServerAdmin.

cPanel.net Support Ticket Number:

 

[sexy_guy]

Well im sure he doesnt have 400 sites on his box so finding something like that is doeable. Try finding something like that 417 sites on the box.

cPanel.net Support Ticket Number:

 

[ciphervendor]

Originally posted by pguy
Hello Folks,

Although we have had this issue before, this time it has had me stumped. The server load seems to spike to 150 or so and renders all unusable. Sites are not served and all goes down until a reboot or waiting for 10 - 15 minutes tends to bring the server load back to some usable level.


I've gone through all the logs and have found absolutely no clue. No formmails, no spam etc

At times I have noticed that httpd tends to start growing in size. Via the top process it can be seen taking up sizes above 200MB or so. I have checked access_logs , CPU usage from WHM and there is nothing weird at all !

Even the httpd, if restarted when the size starts growing, tends to function normally and brings down loads to around 1 or so.

Where else can i see ? I seriously am stumped and need some clues as to how I get about isolating the problem.

Warmest Regards,

cPanel.net Support Ticket Number:

What kernel / OS are you running?

cPanel.net Support Ticket Number:

 

[pguy]

I actually have 800 sites on that specific box. So that made the job that more tough.

I am sure the way I did it would help someone out there. TOP had to be constantly running. I for one simply had no clue as to howto lay the one on this issue.

I started to see httpd's size grow upto 400MB and increasing at a rate of around a 100 MB every minute. I saw the PID and then in WHM clicked on "Current Processes" , then clicked on the right PID and there was a clue to the website name.

To make it all the worse, it was a robots.txt ! Damn... one seek by a search engine and the server would go down ! Now thats news isn't it !

I tried to browse to that robots.txt , and found that it wasn't coming up. I was here thinking that may be it was a spider trapped in a loop or something. But that wasn't meant to be.

Since robots.txt didn't come up, it meant that .htaccess was setup with some values. So checked that and voila a Rewrite rule was put in stating that if a 404 was found, it should load a specific php file. Now this php file was missing and this caused apache to go into tizzy everytime someone hit a 404 on that page !

I could call it an offending piece of code, but the client was actually using the same piece of code on other websites rather successfully. The only difference - the referenced php file was available on those sites and here it wasn't.


Funny, it wasn't a runaway script was it ? That's where I was looking all this while ! Just a simple issue. Even after setting up WHM to ensure Apache's size not to exceed 196 MB or so, that seems to have no effect on issues like these.

But yes, one 404 and the server down .. that sums this case up.

Cheers
:D

cPanel.net Support Ticket Number:

 

[pguy]

We're running RH 7.3 with 2.4.9-34enterprise Kernel.

cPanel.net Support Ticket Number:

 

[ciphervendor]

Any particular reason why you're using the enterprise kernel? Have you considered manually compiling 2.4.21 on your box?

cPanel.net Support Ticket Number:

 

[pguy]

No particular reasons. The kernel was updated quite a while ago by the DC guys, when the previous kernel had some RAM usage issues. Haven't tried an updat e to the latest version yet.

Any specific reason I need to look into for a kernel upgrade ?

cPanel.net Support Ticket Number:

 

[ciphervendor]

Well the new kernel certainly has a lot of vuln plugs; one of the most important is ptrace. The most recent version also includes IDE speed improvements. Memory management isn't an issue anymore and 2.4.21 has been incredibly stable for me on SCSI/RAID & IDE machines.

You'll notice reliability & speed increases in your system after this kernel upgrade.

 

[ciphervendor]

Just kind of further to the above....are you still running ext2? If so, you would also benefit from an upgrade to the ext3 file system.

cPanel.net Support Ticket Number:

 

[pguy]

With 800 onboard sites, I am a definitely queasy about trying a kernel update for now. I've read quite a bit of "how-to"s on it, but never tried to update one before.

I might have the DC staff take a look at it. Mighty thanks for the tip and the heads up on the vulnerability issue.

Anything to get the machine go faster and remain stable over periods of time :D

cPanel.net Support Ticket Number:

 

[pguy]

Didn't they have an issue with ext3 and quotas working only on RH 7.2 ?

cPanel.net Support Ticket Number:

 

[ciphervendor]

That was fixed in kernel release 2.4.17

cPanel.net Support Ticket Number:

 

[pguy]

Thanks mate. You have been very helpful. I guess its time to look at a kernel update for the box. Anything to keep em secure and fast

Thanks for your time and concern

Cheers,

cPanel.net Support Ticket Number:

 

[ciphervendor]

If you recompile your kernel you may also want to consider patching 2.4.21 with grsecurity (grsecurity.net). That will keep your box tighter than a frogs ass under water.

cPanel.net Support Ticket Number:

 

[pguy]

Point Noted and Taken. You're good about server security. I haven't really looked into all the other areas of security, but the kernel and GRsecurity seem to be the way to go.

Just got APF installed and it seems to be working pretty well. I've got lots more to learn and I am sure with folks like you around, I know I have support and hope :D

cPanel.net Support Ticket Number: