/usr/bin/php High CPU Usage

[James.Simpson]

Hi All,

These past couple of days i've been having issues with my WHM server with a high CPU usage. I get an alert from CFS stating high CPU usage, and the process list is showing usr/bin/php using the most usage, but I am unable to see what process within PHP that is causing this load.

It has only happened over these last couple of days, and no server settings have been changed for months, so i'm a little puzzled.

The daily process log for today shows the following: - as you can see the top 3 entries at the most CPU usage is php, followed by sa-compile - which is supposed to be spam assasin, but we dont have mail enabled on the server, but we do have ClamAV on.
- Removed -

Last week, I did get another one of these high load emails, but this was becuase it looked like the server was getting a Ddos attack, with 104 connections from the same IP address in France, linked to a BOT server.

The apachestatus.html file that gets emailed, is pretty useless this time, as it does not show much on there, so i'm not sure where i can look to see what is causing the issue. The server CPU usage is still higher than normal at the moment too.

 

[James.Simpson]

Some more information

PHP Version

PHP 5.5.33 (cli) (built: Apr 3 2016 21:13:38)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2015 Zend Technologies
with the ionCube PHP Loader v4.7.5, Copyright (c) 2002-2014, by ionCube Ltd., and
with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies

/usr/local/cpanel/bin/rebuild_phpconf --current

Available handlers: suphp dso fcgi cgi none
DEFAULT PHP: 5
PHP4 SAPI: none
PHP5 SAPI: fcgi
SUEXEC: enabled
RUID2: not installed

The server is running OPCache and is set up and running. Only serving one main website.

It is strange as it only seems to happen during the day (that i know of).

 

[Infopro]

Please attach images to your posts using the "Upload a File" button. It's just to the right of the Post Reply button.

I get an alert from CFS

What does the alert email say exactly?

 

[James.Simpson]

What does the alert email say exactly?

Email states

Subject: High 5 minute load average alert - 6.14

Body:

Time: Tue Jul 5 15:25:57 2016 +0100
1 Min Load Avg: 7.82
5 Min Load Avg: 6.14
15 Min Load Avg: 4.74
Running/Total Processes: 8/401

Apache Status Attachment: pastebin.com/Lew5v3tM
Netstat Attachemnt: pastebin.com/0Y98AD1C
VMStat Attachment: pastebin.com/A0XXazk1
Process Attachment: pastebin.com/hxtZF84F

Thanks in advanced

 

[Infopro]

This thread may be of some use to you:
https://forums.cpanel.net/threads/troubleshooting-high-server-loads-on-linux-servers.319352/

 

[James.Simpson]

Would this suggest that FCGI is crashing?

mod_fcgid status:

Total FastCGI processes: 14
Process: php5  (/usr/local/cpanel/cgi-sys/php5)
Pid    Active    Idle    Accesses    State
30725    78    17    1    Working
30826    38    38    0    Working
30810    43    38    1    Working
30823    39    38    1    Working
30820    39    39    1    Working
30818    40    40    1    Exiting(communication error)
30816    42    42    1    Exiting(communication error)
30813    42    42    3    Exiting(communication error)
30806    45    43    5    Exiting(communication error)
30803    46    45    2    Exiting(communication error)
30800    46    46    2    Exiting(communication error)
30797    48    48    1    Exiting(communication error)
30794    48    48    1    Exiting(communication error)
30791    49    49    3    Exiting(communication error)

 

[cPanelMichael]

Hello,

It doesn't indicate that FCGI is crashing, however you can review /usr/local/apache/logs/error_log for FCGI-related error messages if you think it's the culprit. You may also need to consult with a qualified system administrator if you are unable to determine the source of the CPU usage:

System Administration Services

Thank you.

 

[James.Simpson]

Yea the log file seems to be full of the following (even though i have tweaked the settings)

[Wed Jul 13 21:50:57.049041 2016] [fcgid:warn] [pid 27807] (104)Connection reset by peer: [client XX.XX.XXX.226:59144] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed Jul 13 21:50:57.232405 2016] [fcgid:warn] [pid 27818] (104)Connection reset by peer: [client XX.XX.XXX.226:59146] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed Jul 13 21:51:03.656572 2016] [fcgid:warn] [pid 27787] (104)Connection reset by peer: [client XX.XX.XXX.226:59148] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed Jul 13 21:51:03.844369 2016] [fcgid:warn] [pid 27799] (104)Connection reset by peer: [client XX.XX.XXX.226:59150] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed Jul 13 21:52:46.519882 2016] [fcgid:warn] [pid 27788] (104)Connection reset by peer: [client XX.XX.XXX.226:59180] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed Jul 13 21:52:46.713144 2016] [fcgid:warn] [pid 27972] (104)Connection reset by peer: [client XX.XX.XXX.226:59182] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[Wed Jul 13 21:52:46.890704 2016] [fcgid:warn] [pid 27806] (104)Connection reset by peer: [client XX.XX.XXX.226:59184] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function

It seems these are added to the log file when a user clicks another link without the page fully loading, and even pressing the stop button the the browser.

 

[cPanelMichael]

You may also find the following thread on StackOverflow helpful:

http://stackoverflow.com/questions/...brigade-failed-in-handle-request-ipc-function

Thank you.

 

[James.Simpson]

Well it happened again this morning, and I think I've finally figured it out... Here is the error right before the crash

[Fri Jul 15 09:38:11.002802 2016] [fcgid:warn] [pid 11463:tid 139680914208704] mod_fcgid: process 4118 graceful kill fail, sending SIGKILL
[Fri Jul 15 09:38:11.002865 2016] [fcgid:warn] [pid 11463:tid 139680914208704] mod_fcgid: process 4115 graceful kill fail, sending SIGKILL
[Fri Jul 15 09:38:11.002897 2016] [fcgid:warn] [pid 11463:tid 139680914208704] mod_fcgid: process 4099 graceful kill fail, sending SIGKILL
[Fri Jul 15 09:38:11.002959 2016] [fcgid:warn] [pid 11463:tid 139680914208704] mod_fcgid: process 4113 graceful kill fail, sending SIGKILL
[Fri Jul 15 09:38:11.003003 2016] [fcgid:warn] [pid 11463:tid 139680914208704] mod_fcgid: process 4111 graceful kill fail, sending SIGKILL
[Fri Jul 15 09:38:15.007279 2016] [fcgid:warn] [pid 11463:tid 139680914208704] mod_fcgid: process 4121 graceful kill fail, sending SIGKILL
[Fri Jul 15 09:38:23.697946 2016] [fcgid:warn] [pid 14988:tid 139680704091904] [client xx.xx.xxx.xxx:57566] mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5
[Fri Jul 15 09:38:28.295928 2016] [fcgid:warn] [pid 20274:tid 139680819787520] [client xx.xx.xxx.xxx:57262] mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5
[Fri Jul 15 09:38:29.145629 2016] [fcgid:warn] [pid 15087:tid 139680683112192] [client xx.xx.xxx.xxx:57672] mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5
[Fri Jul 15 09:38:29.204536 2016] [fcgid:warn] [pid 14988:tid 139680830277376] [client xx.xx.xxx.xxx:57674] mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5

That would assume that the process is hitting the limit, but the limit is set as follows:

##Apache Event Default Settings (overrides main httpd.conf)
KeepAlive On
KeepAliveTimeout 2
MaxKeepAliveRequests 500

<IfModule event.c>
ThreadsPerChild 25
ServerLimit 16
MaxRequestWorkers 400
StartServers 6
MinSpareThreads 150
MaxSpareThreads 400
MaxRequestsPerChild 0
</IfModule>

<IfModule mod_fcgid.c>
FcgidMaxRequestsPerProcess 10000
FcgidMinProcessesPerClass 0
FcgidMaxProcessesPerClass 120
FcgidMaxProcesses 2000
FcgidProcessLifeTime 300
FcgidIOTimeout 80
FcgidIdleTimeout 60
FcgidIdleScanInterval 60
FcgidBusyTimeout 3600
FcgidConnectTimeout 10
FcgidBusyScanInterval 100
FcgidErrorScanInterval 3
FcgidZombieScanInterval 3
FcgidOutputBufferSize 0
FcgidZombieScanInterval 5
FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 10000
FcgidMaxRequestLen 20468982
</IfModule>

 

[cPanelMichael]

You may find this thread helpful for that particular error message:

mod_fcgid: can't apply process slot for /usr/local/cpanel/cgi-sys/php5

However, as mentioned on that thread, you may want to consider switching to a different PHP handler if you are not comfortable with customizing the values. Per our documentation, we only recommend the mod_fcgid Apache module for advanced system administrators who understand how to modify the performance of the mod_fcgid Apache module.

Thank you.