High CPU load and lots of '/usr/sbin/httpd -k start processes'

FabianNL

Registered
Nov 29, 2016
4
0
1
Arnhem, The Netherlands
cPanel Access Level
Root Administrator
Almost everyday my server 'hangs' for about 2 hours, because there are 20+ processes '/usr/sbin/httpd -k start', owned by nobody. If I kill these processes, everything returns to normal. I checked all the logs and also the Apache connections to see if a specific site is causing higher than normal traffic, but that's not the case.
I've searched Google and these forums, but I can't find anyone with the exact same problems.

This is my configuration:
Code:
/etc/redhat-release:CentOS release 6.8 (Final)
/usr/local/cpanel/version:11.60.0.26
/var/cpanel/envtype:kvm
CPANEL=release
Server version: Apache/2.4.23 (cPanel)
Server built:   Nov  8 2016 16:57:01
ea-php-cli Copyright 2016 cPanel, Inc.
PHP 5.6.28 (cli) (built: Nov 14 2016 15:20:37)
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with the ionCube PHP Loader v4.7.5, Copyright (c) 2002-2014, by ionCube Ltd., and
    with Zend Guard Loader v3.3, Copyright (c) 1998-2014, by Zend Technologies
    with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2015, by Zend Technologies
mysql  Ver 15.1 Distrib 10.0.28-MariaDB, for Linux (x86_64) using readline 5.1
Does this problem sound familiair to someone?
 

FabianNL

Registered
Nov 29, 2016
4
0
1
Arnhem, The Netherlands
cPanel Access Level
Root Administrator
Hello,

It's normal to see "/usr/sbin/httpd -k start" processes owned by the "nobody" user. That's the standard username Apache runs as. The following thread is a good place to start if you want to determine what's causing the high load average:

Troubleshooting high server loads on Linux servers

Thank you.
Hi Michael,

Thanks for your answer. I already found that thread before posting this question. I've tried the solutions mentioned there, but with no luck. Can I somehow see what these processes are doing? It now occurs multiple times each day and everytime I have to manually stop multiple processes (30+) in order to make my server responsive again. Killing those processes doesn't seem to have any negative impact on the running sites.
It looks as if these httpd processes are running something outside of the normal websites. Is there someway I can check this?
 

inveress

Registered
Apr 8, 2014
4
0
1
cPanel Access Level
Root Administrator
Hey all,

Just wondering if there was any cause established with this as I'm having the same issue.
Very few visitors on the server, multiple "/usr/sbin/httpd -k start" tasks showing under Process Manager, high load on server (10-15 on a 6 CPU server).
If I restart Apache, it's fine.
Seems to happen most/every morning, not exactly at the same time.

Curiously (per FabianNL's comment), I also recently added the OWASP vendor ruleset to ModSecurity (during a recent cPanel/WHM update - 64.0.17, I believe).

I've tried disabling the automatic updates for the ruleset and this hasn't helped, so next step might be disabling the ruleset altogether, but of course I'd rather not...

Any further info?

Thanks!
Peter.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,913
2,203
363
Very few visitors on the server, multiple "/usr/sbin/httpd -k start" tasks showing under Process Manager, high load on server (10-15 on a 6 CPU server).
If I restart Apache, it's fine.
Hello,

You may want to try some of the investigation tips referenced on the following thread to help determine why the server load is high:

Troubleshooting high server loads on Linux servers

There's a script you can setup that will run on a cron job and allow you to see results from when the server load is high.

Thank you.
 

zuronam

Member
Aug 2, 2016
9
2
3
Zimbabwe
cPanel Access Level
Root Administrator
Hi CpanelMichael

I've just experienced this - after an update to 64.0.24, with OWASP Ruleset 3, my server because completely unstable. I was seeing load averages over 15mintes as high and 6.8 and over 1 minute as high as 17 at times!... and on a 6 core VPS - with over 400 customers - meant lots of disgruntled users.

Upon reading this forum (and after trying virtually everything, cPanel support, CloudLinux Support) I've disabled OWASP core ruleset, and v3.0 on my server - lo and behold my server load is back around 0.48 0.67 1.95

As much as I dislike disabling Mod Security - high load and unhappy customers are worse to deal with, esp when you have a server loading with 6-7 nobody processes using 100% cpu time - all the while websites are unreachable, no Webmail access for customers - basically a server admin's worst nightmare - stretched over a number of days

Any ideas on what the next step is now from here - as I've identified the ruleset to be the cause of load on the server
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,913
2,203
363
Upon reading this forum (and after trying virtually everything, cPanel support, CloudLinux Support) I've disabled OWASP core ruleset, and v3.0 on my server - lo and behold my server load is back around 0.48 0.67 1.95
Could you verify the size of the /var/cpanel/secdatadir/ip.pag file on the system? EX:

Code:
du -sh /var/cpanel/secdatadir/ip.pag
Thanks.
 

zuronam

Member
Aug 2, 2016
9
2
3
Zimbabwe
cPanel Access Level
Root Administrator
Could you verify the size of the /var/cpanel/secdatadir/ip.pag file on the system? EX:

Code:
du -sh /var/cpanel/secdatadir/ip.pag
Thanks.
Hi cPpanelMichael,

I had checked the size of the file as I saw when I would check the apache status

httpd status

I'd get a lot of L's for logging


WLLLL..LLLLLL..LLLLLL - something like that

When I checked the file size, it was 26mb if memory serves. I even cleared it out and tailed the file as it grew to around 400kb, then quit as the load situation was not improving.

I took the suggestion offered by popeye, and after disabling a few rules that did not apply - my server has been performing as expected


upload_2017-6-22_15-46-18.png

mind the low uptime is due to the recent kernel updates

Regards
 
  • Like
Reactions: cPanelMichael

SamAdu

Member
May 8, 2018
7
3
3
Germany
cPanel Access Level
Root Administrator
I am posting my solution with the hope that it can help somebody (or myself) in the future.

I was experiencing the same issue on a 24 core, 64GB RAM server. The load would suddenly rise to more than 100 within few minutes after I manually restart apache. After spending two weeks troubleshooting following the link posted by cPanelMichael and support from cpanel staff the issue was not resolved. I even installed nginx (engintron) and even after paying for optimisation ( I thought I was missing something) the issue was still not resolved.

I chanced upon ip.pag file on a post, which turned out to be responsible for the high load

The Fix (without disabling OWASP)
After monitoring the /usr/bin/httpd processes with htop, I noticed that /tmp/ip.pag (about 368MB) was appearing in a lot of the files opened by the process.

However when I googled that file, the results were referring to /var/cpanel/secdatadir/ip.pag but the size was 0 and evidently it wasn't being used.

What fixed the issue for me was using the script described here ModSecurity SDBM Utility - EasyApache 4 - cPanel Documentation

I had to replace references to /var/cpanel/secdatadir with /tmp.

Code:
/usr/sbin/modsec-sdbm-util -D /tmp -v -n /tmp/ip.pag &&\
rm /tmp/ip.pag &&\
rm /tmp/ip.dir &&\
mv /tmp/new_db.pag /tmp/ip.pag &&\
mv /tmp/new_db.dir /tmp/ip.dir
After the script executed, the load stabilized and dropped to normal below 10.

If you use cloudlinux, consider mod_lsapi also as the php handler. Without mod_lsapi my normal load was above 10, now it is below 10 after using mod_lsapi.
 
  • Like
Reactions: cPanelMichael