I'm running cPanel 11.25.0-S46156 on Centos 5.5 x86_64 on VMware. The virtual server has 4 CPUs and 8Gb of RAM and is hosting just under 200 sites.
It runs like a dream 99% of the time, but once every week day, the load skyrockets bringing the server to it's knees. (I've seen the load average reach 160)
Once I restart the machine everything returns to normal until the next day when at a some random time the load will skyrocket again.
The logs reveal nothing unusual. I've been able to watch top a few times while the load has been going through the ceiling and there were no visible processes contributing to the load in any significant way.
The MySQL slow query log shows whatever queries happen to be running at the time of the high load, these are totally random and appear to be running slowly *because* of the high load, rather than being the cause of the high load.
I currently have two separate server management teams monitoring the server as well as my own techs. Copious amounts of logging have been added to the system, but none of the logs have revealed anything significant. (When the server is under load many of the logs are not even written)
I've added a bash script to PT_LOAD in CSF which restarts apache if the load average reaches 6, but the load spikes so rapidly that this doesn't even have enough resources to fire.
On those rare occasions where I've been able to shell into the box, I've sometimes been able to reduce the load by restarting apache manually, but interestingly if I do this, another load spike will occur within a few hours. Only rebooting the server prevents a re-occurrence (until the next day).
The host server does not appear to be causing the load either, during times when the Guest machine was under extreme load, the host machine was running at about 25% capacity.
So I'm posting this in the hope that someone else out there may have had a similar experience and might be able to offer some insights as to the cause of these "phantom" loads.
It runs like a dream 99% of the time, but once every week day, the load skyrockets bringing the server to it's knees. (I've seen the load average reach 160)
Once I restart the machine everything returns to normal until the next day when at a some random time the load will skyrocket again.
The logs reveal nothing unusual. I've been able to watch top a few times while the load has been going through the ceiling and there were no visible processes contributing to the load in any significant way.
The MySQL slow query log shows whatever queries happen to be running at the time of the high load, these are totally random and appear to be running slowly *because* of the high load, rather than being the cause of the high load.
I currently have two separate server management teams monitoring the server as well as my own techs. Copious amounts of logging have been added to the system, but none of the logs have revealed anything significant. (When the server is under load many of the logs are not even written)
I've added a bash script to PT_LOAD in CSF which restarts apache if the load average reaches 6, but the load spikes so rapidly that this doesn't even have enough resources to fire.
On those rare occasions where I've been able to shell into the box, I've sometimes been able to reduce the load by restarting apache manually, but interestingly if I do this, another load spike will occur within a few hours. Only rebooting the server prevents a re-occurrence (until the next day).
The host server does not appear to be causing the load either, during times when the Guest machine was under extreme load, the host machine was running at about 25% capacity.
So I'm posting this in the hope that someone else out there may have had a similar experience and might be able to offer some insights as to the cause of these "phantom" loads.
Last edited: