Apaching consuming large amounts of CPU

[asofrank]

Recently we pushed out a new Easyapache build on all of our servers. Everything appears to have went fine, and I believe the only change was an upgrade to ioncube and zend. We kept the original template, so I know that no changes were made in that regard.

Our servers are running CentOS 5.4, with Apache 2.2.14 and Cpanel::Easy::Apache v3.2.0 rev4899. The PHP configuration varies a bit depending on the server, but we run both PHP4 and PHP5, one as a DSO, the other as suPHP.

Everything will run fine for a random amount of time, then suddenly (within a couple minutes) the load will spike up to over 100. After ssh'ing in and looking, there are many apache processes running, and quite a few of them are using ~ 10% CPU. What I gather is the CPU usage spikes from apache, then the requests just start backing up which causes more child processes to spawn. This eventually leads to the server eating all its available CPU, RAM, and swap space if not dealt with.

Has anyone else seen this happen recently? The last time we saw something like this was when wordpress pushed out an update with some bad code and cause apache to eat 100% CPU, though I'm not sure if thats the case here.

Please let me know if you need any additional diagnostic information. We are seeing this on around 80 or so servers at the moment.

[timdorr]

To add some information, we're seeing this in Wordpress during it's internal cron run. It has something to do with it calling wp_remote_post() and doing a bunch of stuff related to fsockopen(). I haven't been able to find the exact trigger, but this is stuff that was working fine just a week ago.

[fi77i]

Try /scripts/apachelimits

[asofrank]

The apache limit have been set and unchanged for quite some time.

[Infopro]

To add some information, we're seeing this in Wordpress during it's internal cron run. It has something to do with it calling wp_remote_post() and doing a bunch of stuff related to fsockopen(). I haven't been able to find the exact trigger, but this is stuff that was working fine just a week ago.

Searching for wp_remote_post() gives me this page:
Function Reference/trackback « WordPress Codex
mentioning wp_remote_post() for trackback URLS. Not sure if that helps or not.

The apache limit have been set and unchanged for quite some time.

Of course you've gone over your settings again anyway even though they've not been touched for some time, right?

What version of cPanel?
Mod security installed via EA3?
Have your settings been double checked in WHM > Service Configuration > Apache Configuration (global config and memory usage in particular)
Is CSF installed? This can help point more directly at a particular account causing problems, in my experience.
How about running the latest EA3 again and hand select from the exhaustive options list and build a new profile just to be sure your options are all as you want on one server to see if it helps?

cPanel has made lots of new changes as I'm sure you know. If this issue is something out of the blue across 80 servers my first thought would be to go over all those settings I hadn't changed for quite some time.

[asobj]

All are running 11.24.5-RELEASE_38506
Mod_security is installed via EA3
Settings are set via script, not through whm but they still are correct.
CSF is installed.
According to the changelog not much was changed since the last time we ran EA3. It's ran fairly often to keep things current (about 3 weeks from last run) and we use one EA3 template file across all servers.

Easyapache 4899 - 2009-11-27 13:11:34
Updated IonCube Loaders to 3.3.7

Easyapache 4895 - 2009-11-23 12:44:39
Add CVE-2009-3555 patch to Apache 1.3

Easyapache 4860 - 2009-11-18 11:47:35
Updated SourceGuardian to 8.0
Updated IonCube to 3.3.5
Updated EAccelerator to 0.9.6-RC1

The servers that have been affected have not all shown issues at the same time.