HTTPD service appears to be down

[iTecan]

Hello,

our company had a lot of problems related to the httpd service, sometimes all the websites are unable to load and the only solution is to restart the PHP-FPM service.

Opening tickets to cPanel support team give us the direction to check MaxRequestWorkers, ServerLimit and MaxChildren directive.

We have the following server using MPM_Event:

* VPS 8Cores and 12GB RAM.
* Immunify360
* Centos 7.9
* CPANEL last version v100.0.4
* PHP 5.6, 7.2, 7.3, 7.4, 8.0 (cloudlinux version added by Inmunify360)
* PHP-FPM enabled
* Mod-Security Inmunify360 (disabled Apache security vendor)

The configuration related to our Apache Service is the following:



Using the following command:

echo;echo "Count of Times MaxChildren Was Hit Per Domain";for log in $(find /opt/cpanel/ea-php*/root/usr/var/log/php-fpm/ -type f -name error.log);do echo;echo $log; startdate=$( head -1 $log | awk -F"[][]" '{print $2}' );enddate=$( tail -1 $log | awk -F"[][]" '{print $2}' );if [ -z "$startdate" ];then echo "EMPTY";else printf "Log-Start: $startdate\nLog-End: $enddate" | column -t;fi;awk '/server reached max_children/{gsub("_",".",$5);split($5,d,"]");arr[d[1]]++}END{for (i in arr) print arr[i],i|"sort -nr"}' $log ;done

We have the outpout:

Count of Times MaxChildren Was Hit Per Domain

/opt/cpanel/ea-php56/root/usr/var/log/php-fpm/error.log
Log-Start:  21-Nov-2021  03:41:21
Log-End:    24-Nov-2021  17:22:55
4 turismoruralcantabria.com
2 turismorc.itecan.es
2 casondelamarquesa.com
1 rojoyvegas.com
1 cirugiadominguez.es

/opt/cpanel/ea-php70/root/usr/var/log/php-fpm/error.log
Log-Start:  21-Nov-2021  03:41:21
Log-End:    24-Nov-2021  14:21:23
2 ojebar.itecan.es

/opt/cpanel/ea-php72/root/usr/var/log/php-fpm/error.log
Log-Start:  21-Nov-2021  03:41:21
Log-End:    24-Nov-2021  14:21:28
1 lebecuesta.com

/opt/cpanel/ea-php73/root/usr/var/log/php-fpm/error.log
Log-Start:  21-Nov-2021  03:41:21
Log-End:    24-Nov-2021  15:59:19
10 clinicadentalmartinriva.es
2 restauranteelredoble.com
1 itecan.es
1 cirugiacantabria.es

/opt/cpanel/ea-php74/root/usr/var/log/php-fpm/error.log
Log-Start:  21-Nov-2021  03:41:21
Log-End:    24-Nov-2021  16:18:41
28 grupopinta.com
2 electricidadgutierrezsl.com
1 sobremazas.com
1 pensionjade.com
1 palacioguevara.com
1 nuevacasamadrazo.com
1 laiginvest.com
1 lacasadelanavidaddenoja.com
1 garantiasanitaria.es
1 galernamarketing.es
1 fincaartienza.es
1 fincaalegranza.com
1 ergia.es
1 enphys.com
1 casonaelarral.com

This is the usual outpout of the command with only a few domains hitting a lot of times the MaxChildren directive.

Also the times we get notification emails related to this issue we get logs related to Imunify vendor:

Notification    The service “httpd” is now operational.
Startup Log    Nov 17 21:08:58 cloud.itecan.es systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Nov 17 21:09:00 cloud.itecan.es restartsrv_httpd[3995]: AH00513: WARNING: MaxRequestWorkers of 256 is not an integer multiple of ThreadsPerChild of 25, decreasing to nearest multiple 250, for a maximum of 10 servers.
Nov 17 21:09:01 cloud.itecan.es systemd[1]: Can't open PID file /run/apache2/httpd.pid (yet?) after start: No such file or directory
Nov 17 21:09:01 cloud.itecan.es systemd[1]: Started Apache web server managed by cPanel EasyApache.
Log Messages    [Wed Nov 17 21:09:01.496135 2021] [mpm_event:notice] [pid 4016:tid 47711650686016] AH00489: Apache/2.4.51 (cPanel) OpenSSL/1.1.1l mod_bwlimited/1.4 configured -- resuming normal operations
[Wed Nov 17 21:09:01.135989 2021] [:notice] [pid 4013:tid 47711650686016] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Wed Nov 17 20:52:29.076298 2021] [:error] [pid 4996:tid 47399851431680] [client 13.235.161.168:49216] [client 13.235.161.168] ModSecurity: Warning. Pattern match "^POST$" at REQUEST_METHOD. [file "/etc/apache2/conf.d/modsec_vendor_configs/imunify360-full-apache/004_i360_2_bruteforce.conf"] [line "66"] [id "33339"] [msg "IM360 WAF: WordPress XML-RPC access attempt||laposadadeltenor.com"] [severity "NOTICE"] [tag "service_bruteforce"] [tag "service_i360"] [tag "noshow"] [hostname "laposadadeltenor.com"] [uri "/xmlrpc.php"] [unique_id "YZVdfQhEiM8_i_gyomAEfQAAAJA"]
[Wed Nov 17 20:50:59.860881 2021] [:error] [pid 30916:tid 47399944873728] [client 192.210.201.215:36105] [client 192.210.201.215] ModSecurity: Warning. Pattern match "^POST$" at REQUEST_METHOD. [file "/etc/apache2/conf.d/modsec_vendor_configs/imunify360-full-apache/004_i360_2_bruteforce.conf"] [line "66"] [id "33339"] [msg "IM360 WAF: WordPress XML-RPC access attempt||enphys.com"] [severity "NOTICE"] [tag "service_bruteforce"] [tag "service_i360"] [tag "noshow"] [hostname "enphys.com"] [uri "/xmlrpc.php"] [unique_id "YZVdIzCpaVDjcR3yWdXEOgABhCI"]
[Wed Nov 17 20:48:27.136739 2021] [:error] [pid 4995:tid 47399934367488] [client 128.199.161.145:54744] [client 128.199.161.145] ModSecurity: Warning. Pattern match "^POST$" at REQUEST_METHOD. [file "/etc/apache2/conf.d/modsec_vendor_configs/imunify360-full-apache/004_i360_2_bruteforce.conf"] [line "66"] [id "33339"] [msg "IM360 WAF: WordPress XML-RPC access attempt||www.casondelamarquesa.com"] [severity "NOTICE"] [tag "service_bruteforce"] [tag "service_i360"] [tag "noshow"] [hostname "www.casondelamarquesa.com"] [uri "/xmlrpc.php"] [unique_id "YZVciyfWR8bh8RIL8T7q3AAAAFc"]

We have some questions related to this:

Is the system entering in swap and thats the reason the webs can't load?
The ServerLimit and MaxRequestWokers is not configured properly for MPM_Event?
The Imunify Vendor causes some problems and thats the reason to httpd service to fail?

 

[cPanelAnthony]

Hello! Some of the users here might be able to point you in a direction regarding your server's settings. However, it may also be worth speaking with your systems administrator or web hosting company for advice as well.

If a server is using a large amount of page swap, that generally indicates the server is running out of memory. So, while the page swap itself won't cause service failure, it can be a symptom of not having enough RAM, which can lead to a variety of issues.

Regarding the configuration of MaxRequestWorkers and ServerLimit, the following Apache documentation might help.

In regards to Imunify, those log entries are unrelated to the Apache issues. These entries are just describing how Imunify is protecting your server and doing its job; you can see those entries were a long time after the Apache downtime as well based on the time stamps.

 

[kennysamuerto]

I think you are confusing the parameters of apache, with those of FPM. I understand that what you have done, is to adjust the apache values in WHM.

But... Have you adjusted the FPM values for each domain? I think the problem is that one of your domains may be exceeding the FPM limits, and cause the rest to stop working. This can also cause memory consumption, and therefore swape.

I would recommend you to check the FPM values.

 

[iTecan]

I think you are confusing the parameters of apache, with those of FPM. I understand that what you have done, is to adjust the apache values in WHM.

But... Have you adjusted the FPM values for each domain? I think the problem is that one of your domains may be exceeding the FPM limits, and cause the rest to stop working. This can also cause memory consumption, and therefore swape.

I would recommend you to check the FPM values.

Hello Kenny,

We have modify the values checking the domains everyday using the next command:

ls /opt/cpanel|grep ea-php|while read i; do find /opt/cpanel/$i -type f -name error.log; done| while read log; do echo $log; grep max_children $log|tail -5; echo; done

The outpout gives the domains that hit the MaxChildren that is assigned per day and hour so we can modify those values in case. This week we have modify this values with domains that hitted the most times and right now we have very few exceptions.

We also have created a script to restart the php-fpm service in case that it detectes that any web is down.