High Apache loads, long processes

[verdon]

Hi,

For the last week or so, I've been experiencing server loads much higher than usual. When checking with top, it is always httpd consuming most of the cpu, followed by php-fpm. It's not uncommon 3 or 4 httpd processes using 100-300% of the CPU each. Sometimes these processes will have been running 8 or 10 or more minutes. I don't think that's normal. At times, there also seems to be extremely high numbers of packets per second, sometimes inbound, sometimes out.

I've checked the disks themselves in the server and they are fine. I've run several malware checks and they seem fine*. Overall bandwidth use seems fairly normal. I've switched from using mpm_prefork to mpm_worker as it seemed to help with lag being caused by the load. I'm not sure where to go from here.

Any suggestions would be welcome.

* maldet was finding what seems to be false positives in the various domlogs. It looked like it was reacting to POST requests that just 404'd anyways. It found nothing in any public_html dirs.

 

[GOT]

Are you hosting wordpress? If so the most likely scenario is that your wp sites are getting hammered.

When you see php fpm in top it should show you the user associated with it. I would look through their domlogs to see what is getting hammered. Also it's often just one or two ips causing the mayhem and using netstat or the logs you should be able to determine those and block them in your firewall.

 

[verdon]

Hi Thanks,

Yes. Mostly WP. When I did a little checking with netstat, it seemed things were pretty distributed. I'm finding it getting harder and harder to meaningfully block this sort of stuff by IP as the pokes are increasingly distributed... even brute force attempts. Tools that have helped for years are getting less and less effective.

 

[verdon]

Shouldn't an httpd process that's consuming 300% of CPU and been running for 19+ minutes kill itself eventually?

 

[GOT]

Biggest potential problem areas are outdated wp and plugins, xmlrpc.php and wp-longin.php attacks. I can't really tell you how to best parse the logs, though there are plugins <cough cough> that can help identify attacking IPs. There are also modifications you can make to CSF to block thinks like IPs that hit xmlrpc.php XX times

Blocking Wordpress Login and xmlprc attacks with LFD - ConfigServer Community Forum

There really isn't a single solution, its about identifying the specific problem at the time and mitigating that case.

Maldet really shouldn't be run on anything other than site date in public_html really.

It would probably be worthwhile to review your processlist too to make sure you don't have malware bots running. If you do, look at the owning user and consider that site a real problem.

 

[verdon]

Thanks @GOT

I appreciate the tips! The WP sites are all diligently kept up to date and pretty pristine. They are pretty much all from one developer, who runs a tight ship. Wordfence was running on all of them, but not providing much help and in fact increasing load itself. We've temporarily disabled it. I am using CSF/LFD with WP Fail2Ban as well as some custom mod_sec rules for random pokes at wp-admin and xmlprc. I'll follow that link through to their forum though, always something new to learn

I appreciate the advice.

 

[GOT]

Worse comes to worst, I'd suggest using Litespeed.

 

[verdon]

I've been thinking about moving away from apache for a little while now... so much knowledge invested in it though, I'm a little afraid to pull the trigger ;-)

 

[GOT]

In cPanel its going to use the apache configs for the most part, though there are separate settings for some things located in the Litespeed console, though in most cases these rarely have to be adjusted.

You'll be pretty impressed with the performance.

 

[verdon]

@GOT

You're right. LiteSpeed's performance is pretty impressive! The extra $45/month looks like it will be well spent. I've been running it for a couple days now and am monitoring for any issues... so far, so good.

I also use the script at this post Tutorial - Troubleshooting high server loads on Linux servers to monitor loads and behaviour. The reports now are mostly pretty good, but I still seem to get a lot of reports of high Packets Per Second, especially outbound. That said, I'm having a hard time getting consensus/knowledge on what is actually a reasonable number.

Does anyone have any thoughts as to what a reasonable number is for PPS in/out? Any suggestions where to find what is causing the the high numbers outbound? I'm not seeing anything obvious in domlogs and such, but I'm really poking around blind and ignorant.

 

[cPanelMichael]

Hello @verdon,

On the topic of WordPress, we recently published some guides on optimizing WordPress that you may also find helpful:

Tutorial - Basic WordPress Optimization Steps
Tutorial - Advanced WordPress Optimization Steps

Thanks!

 

[verdon]

Thanks @cPanelMichael