High Apache loads, long processes

Hi,

For the last week or so, I've been experiencing server loads much higher than usual. When checking with top, it is always httpd consuming most of the cpu, followed by php-fpm. It's not uncommon 3 or 4 httpd processes using 100-300% of the CPU each. Sometimes these processes will have been running 8 or 10 or more minutes. I don't think that's normal. At times, there also seems to be extremely high numbers of packets per second, sometimes inbound, sometimes out.

I've checked the disks themselves in the server and they are fine. I've run several malware checks and they seem fine*. Overall bandwidth use seems fairly normal. I've switched from using mpm_prefork to mpm_worker as it seemed to help with lag being caused by the load. I'm not sure where to go from here.

Any suggestions would be welcome.

* maldet was finding what seems to be false positives in the various domlogs. It looked like it was reacting to POST requests that just 404'd anyways. It found nothing in any public_html dirs.

 

Hi Thanks,

Yes. Mostly WP. When I did a little checking with netstat, it seemed things were pretty distributed. I'm finding it getting harder and harder to meaningfully block this sort of stuff by IP as the pokes are increasingly distributed... even brute force attempts. Tools that have helped for years are getting less and less effective.

 

Shouldn't an httpd process that's consuming 300% of CPU and been running for 19+ minutes kill itself eventually?

 

Thanks @GOT

I appreciate the tips! The WP sites are all diligently kept up to date and pretty pristine. They are pretty much all from one developer, who runs a tight ship. Wordfence was running on all of them, but not providing much help and in fact increasing load itself. We've temporarily disabled it. I am using CSF/LFD with WP Fail2Ban as well as some custom mod_sec rules for random pokes at wp-admin and xmlprc. I'll follow that link through to their forum though, always something new to learn

I appreciate the advice.

 

I've been thinking about moving away from apache for a little while now... so much knowledge invested in it though, I'm a little afraid to pull the trigger ;-)

 

@GOT

You're right. LiteSpeed's performance is pretty impressive! The extra $45/month looks like it will be well spent. I've been running it for a couple days now and am monitoring for any issues... so far, so good.

I also use the script at this post Tutorial - Troubleshooting high server loads on Linux servers to monitor loads and behaviour. The reports now are mostly pretty good, but I still seem to get a lot of reports of high Packets Per Second, especially outbound. That said, I'm having a hard time getting consensus/knowledge on what is actually a reasonable number.

Does anyone have any thoughts as to what a reasonable number is for PPS in/out? Any suggestions where to find what is causing the the high numbers outbound? I'm not seeing anything obvious in domlogs and such, but I'm really poking around blind and ignorant.

 

Thanks @cPanelMichael