verdon

Well-Known Member
Nov 1, 2003
917
10
168
Northern Ontario, Canada
cPanel Access Level
Root Administrator
Hi,

For the last week or so, I've been experiencing server loads much higher than usual. When checking with top, it is always httpd consuming most of the cpu, followed by php-fpm. It's not uncommon 3 or 4 httpd processes using 100-300% of the CPU each. Sometimes these processes will have been running 8 or 10 or more minutes. I don't think that's normal. At times, there also seems to be extremely high numbers of packets per second, sometimes inbound, sometimes out.

I've checked the disks themselves in the server and they are fine. I've run several malware checks and they seem fine*. Overall bandwidth use seems fairly normal. I've switched from using mpm_prefork to mpm_worker as it seemed to help with lag being caused by the load. I'm not sure where to go from here.

Any suggestions would be welcome.

* maldet was finding what seems to be false positives in the various domlogs. It looked like it was reacting to POST requests that just 404'd anyways. It found nothing in any public_html dirs.
 

GOT

Get Proactive!
PartnerNOC
Apr 8, 2003
1,543
208
343
Chesapeake, VA
cPanel Access Level
DataCenter Provider
Are you hosting wordpress? If so the most likely scenario is that your wp sites are getting hammered.

When you see php fpm in top it should show you the user associated with it. I would look through their domlogs to see what is getting hammered. Also it's often just one or two ips causing the mayhem and using netstat or the logs you should be able to determine those and block them in your firewall.
 

verdon

Well-Known Member
Nov 1, 2003
917
10
168
Northern Ontario, Canada
cPanel Access Level
Root Administrator
Hi Thanks,

Yes. Mostly WP. When I did a little checking with netstat, it seemed things were pretty distributed. I'm finding it getting harder and harder to meaningfully block this sort of stuff by IP as the pokes are increasingly distributed... even brute force attempts. Tools that have helped for years are getting less and less effective.
 

GOT

Get Proactive!
PartnerNOC
Apr 8, 2003
1,543
208
343
Chesapeake, VA
cPanel Access Level
DataCenter Provider
Biggest potential problem areas are outdated wp and plugins, xmlrpc.php and wp-longin.php attacks. I can't really tell you how to best parse the logs, though there are plugins <cough cough> that can help identify attacking IPs. There are also modifications you can make to CSF to block thinks like IPs that hit xmlrpc.php XX times

Blocking Wordpress Login and xmlprc attacks with LFD - ConfigServer Community Forum

There really isn't a single solution, its about identifying the specific problem at the time and mitigating that case.

Maldet really shouldn't be run on anything other than site date in public_html really.

It would probably be worthwhile to review your processlist too to make sure you don't have malware bots running. If you do, look at the owning user and consider that site a real problem.
 

verdon

Well-Known Member
Nov 1, 2003
917
10
168
Northern Ontario, Canada
cPanel Access Level
Root Administrator
Thanks @GOT

I appreciate the tips! The WP sites are all diligently kept up to date and pretty pristine. They are pretty much all from one developer, who runs a tight ship. Wordfence was running on all of them, but not providing much help and in fact increasing load itself. We've temporarily disabled it. I am using CSF/LFD with WP Fail2Ban as well as some custom mod_sec rules for random pokes at wp-admin and xmlprc. I'll follow that link through to their forum though, always something new to learn :)

I appreciate the advice.
 

GOT

Get Proactive!
PartnerNOC
Apr 8, 2003
1,543
208
343
Chesapeake, VA
cPanel Access Level
DataCenter Provider
In cPanel its going to use the apache configs for the most part, though there are separate settings for some things located in the Litespeed console, though in most cases these rarely have to be adjusted.

You'll be pretty impressed with the performance.
 
  • Like
Reactions: verdon

verdon

Well-Known Member
Nov 1, 2003
917
10
168
Northern Ontario, Canada
cPanel Access Level
Root Administrator
@GOT

You're right. LiteSpeed's performance is pretty impressive! The extra $45/month looks like it will be well spent. I've been running it for a couple days now and am monitoring for any issues... so far, so good.

I also use the script at this post Tutorial - Troubleshooting high server loads on Linux servers to monitor loads and behaviour. The reports now are mostly pretty good, but I still seem to get a lot of reports of high Packets Per Second, especially outbound. That said, I'm having a hard time getting consensus/knowledge on what is actually a reasonable number.

Does anyone have any thoughts as to what a reasonable number is for PPS in/out? Any suggestions where to find what is causing the the high numbers outbound? I'm not seeing anything obvious in domlogs and such, but I'm really poking around blind and ignorant.
 
  • Like
Reactions: cPanelMichael