Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

High Apache loads, long processes

Discussion in 'Workarounds and Optimization' started by verdon, Jun 18, 2019.

  1. verdon

    verdon Well-Known Member

    Joined:
    Nov 1, 2003
    Messages:
    902
    Likes Received:
    8
    Trophy Points:
    168
    Location:
    Northern Ontario, Canada
    cPanel Access Level:
    Root Administrator
    Hi,

    For the last week or so, I've been experiencing server loads much higher than usual. When checking with top, it is always httpd consuming most of the cpu, followed by php-fpm. It's not uncommon 3 or 4 httpd processes using 100-300% of the CPU each. Sometimes these processes will have been running 8 or 10 or more minutes. I don't think that's normal. At times, there also seems to be extremely high numbers of packets per second, sometimes inbound, sometimes out.

    I've checked the disks themselves in the server and they are fine. I've run several malware checks and they seem fine*. Overall bandwidth use seems fairly normal. I've switched from using mpm_prefork to mpm_worker as it seemed to help with lag being caused by the load. I'm not sure where to go from here.

    Any suggestions would be welcome.

    * maldet was finding what seems to be false positives in the various domlogs. It looked like it was reacting to POST requests that just 404'd anyways. It found nothing in any public_html dirs.
     
  2. GOT

    GOT Get Proactive! PartnerNOC

    Joined:
    Apr 8, 2003
    Messages:
    1,484
    Likes Received:
    187
    Trophy Points:
    193
    Location:
    Chesapeake, VA
    cPanel Access Level:
    DataCenter Provider
    Are you hosting wordpress? If so the most likely scenario is that your wp sites are getting hammered.

    When you see php fpm in top it should show you the user associated with it. I would look through their domlogs to see what is getting hammered. Also it's often just one or two ips causing the mayhem and using netstat or the logs you should be able to determine those and block them in your firewall.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. verdon

    verdon Well-Known Member

    Joined:
    Nov 1, 2003
    Messages:
    902
    Likes Received:
    8
    Trophy Points:
    168
    Location:
    Northern Ontario, Canada
    cPanel Access Level:
    Root Administrator
    Hi Thanks,

    Yes. Mostly WP. When I did a little checking with netstat, it seemed things were pretty distributed. I'm finding it getting harder and harder to meaningfully block this sort of stuff by IP as the pokes are increasingly distributed... even brute force attempts. Tools that have helped for years are getting less and less effective.
     
  4. verdon

    verdon Well-Known Member

    Joined:
    Nov 1, 2003
    Messages:
    902
    Likes Received:
    8
    Trophy Points:
    168
    Location:
    Northern Ontario, Canada
    cPanel Access Level:
    Root Administrator
    Shouldn't an httpd process that's consuming 300% of CPU and been running for 19+ minutes kill itself eventually?
     
  5. GOT

    GOT Get Proactive! PartnerNOC

    Joined:
    Apr 8, 2003
    Messages:
    1,484
    Likes Received:
    187
    Trophy Points:
    193
    Location:
    Chesapeake, VA
    cPanel Access Level:
    DataCenter Provider
    Biggest potential problem areas are outdated wp and plugins, xmlrpc.php and wp-longin.php attacks. I can't really tell you how to best parse the logs, though there are plugins <cough cough> that can help identify attacking IPs. There are also modifications you can make to CSF to block thinks like IPs that hit xmlrpc.php XX times

    Blocking Wordpress Login and xmlprc attacks with LFD - ConfigServer Community Forum

    There really isn't a single solution, its about identifying the specific problem at the time and mitigating that case.

    Maldet really shouldn't be run on anything other than site date in public_html really.

    It would probably be worthwhile to review your processlist too to make sure you don't have malware bots running. If you do, look at the owning user and consider that site a real problem.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. verdon

    verdon Well-Known Member

    Joined:
    Nov 1, 2003
    Messages:
    902
    Likes Received:
    8
    Trophy Points:
    168
    Location:
    Northern Ontario, Canada
    cPanel Access Level:
    Root Administrator
    Thanks @GOT

    I appreciate the tips! The WP sites are all diligently kept up to date and pretty pristine. They are pretty much all from one developer, who runs a tight ship. Wordfence was running on all of them, but not providing much help and in fact increasing load itself. We've temporarily disabled it. I am using CSF/LFD with WP Fail2Ban as well as some custom mod_sec rules for random pokes at wp-admin and xmlprc. I'll follow that link through to their forum though, always something new to learn :)

    I appreciate the advice.
     
  7. GOT

    GOT Get Proactive! PartnerNOC

    Joined:
    Apr 8, 2003
    Messages:
    1,484
    Likes Received:
    187
    Trophy Points:
    193
    Location:
    Chesapeake, VA
    cPanel Access Level:
    DataCenter Provider
    Worse comes to worst, I'd suggest using Litespeed.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. verdon

    verdon Well-Known Member

    Joined:
    Nov 1, 2003
    Messages:
    902
    Likes Received:
    8
    Trophy Points:
    168
    Location:
    Northern Ontario, Canada
    cPanel Access Level:
    Root Administrator
    I've been thinking about moving away from apache for a little while now... so much knowledge invested in it though, I'm a little afraid to pull the trigger ;-)
     
  9. GOT

    GOT Get Proactive! PartnerNOC

    Joined:
    Apr 8, 2003
    Messages:
    1,484
    Likes Received:
    187
    Trophy Points:
    193
    Location:
    Chesapeake, VA
    cPanel Access Level:
    DataCenter Provider
    In cPanel its going to use the apache configs for the most part, though there are separate settings for some things located in the Litespeed console, though in most cases these rarely have to be adjusted.

    You'll be pretty impressed with the performance.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    verdon likes this.
  10. verdon

    verdon Well-Known Member

    Joined:
    Nov 1, 2003
    Messages:
    902
    Likes Received:
    8
    Trophy Points:
    168
    Location:
    Northern Ontario, Canada
    cPanel Access Level:
    Root Administrator
    @GOT

    You're right. LiteSpeed's performance is pretty impressive! The extra $45/month looks like it will be well spent. I've been running it for a couple days now and am monitoring for any issues... so far, so good.

    I also use the script at this post Tutorial - Troubleshooting high server loads on Linux servers to monitor loads and behaviour. The reports now are mostly pretty good, but I still seem to get a lot of reports of high Packets Per Second, especially outbound. That said, I'm having a hard time getting consensus/knowledge on what is actually a reasonable number.

    Does anyone have any thoughts as to what a reasonable number is for PPS in/out? Any suggestions where to find what is causing the the high numbers outbound? I'm not seeing anything obvious in domlogs and such, but I'm really poking around blind and ignorant.
     
    cPanelMichael likes this.
  11. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    verdon likes this.
  12. verdon

    verdon Well-Known Member

    Joined:
    Nov 1, 2003
    Messages:
    902
    Likes Received:
    8
    Trophy Points:
    168
    Location:
    Northern Ontario, Canada
    cPanel Access Level:
    Root Administrator
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice