Apache does not recover from SIGUSR1

jsparks · May 30, 2018

Hi folks,

I've been using php-fpm / apach2.4 / engintron (nginx reverse proxy) for quite some time now and noticed every few days apache dies and does not recover until I manually intervene and restart the process.

The server has has no resource limitation issues that I am aware of, and from what I can tell, everything is configured as it should.

I see in the apache error.log the following:

[Wed May 30 13:13:45.018854 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00297: SIGUSR1 received. Doing graceful restart
[Wed May 30 13:13:48.157406 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00292: Apache/2.4.33 (cPanel) OpenSSL/1.0.2o mod_bwlimited/1.4 mod_fcgid/2.3.9 configured -- resuming normal operations
[Wed May 30 13:13:48.157437 2018] [core:notice] [pid 30722:tid 140261823477888] AH00094: Command line: '/usr/sbin/httpd'

followed by many:

[Wed May 30 13:18:42.782001 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5377 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782033 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5561 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782042 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 8576 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782051 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 9776 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782059 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 11882 still did not exit, sending a SIGTERM

I checked the php-fpm logs in both /opt/cpanel/ea-php70/root/usr/var/log/php-fpm/ & /opt/cpanel/ea-php56/root/usr/var/log/php-fpm/ and seen no errors or warnings.. What else could be controlling this process?

Thanks in advance.

cPanelLauren · May 30, 2018

Hi @jsparks

Does the issue persist when using only Apache? Can you run the following on the server and post the output:

Code:

cat /etc/redhat-release

Code:

cat /usr/local/cpanel/version

Code:

sar -q

Code:

sar -p

Code:

grep -i "Out of Memory" /var/log/messages

Thanks!

jsparks · May 30, 2018

I never actually configured this server to run apache only, it's always been apache/php-fpm/nginx, but I can try re-configuring it if you think it's worth testing.

Please see output:

cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

cat /usr/local/cpanel/version
11.70.0.46

Code:

sar -q
Linux 4.9.87-xxxx-std-ipv6-64 ()      05/30/2018      _x86_64_        (32 CPU)

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
12:10:01 AM        13      1249      5.79      4.77      4.89         0
12:20:01 AM        12      1235      3.99      4.62      4.79         1
12:30:01 AM        10      1240      3.85      4.41      4.60         1
12:40:01 AM        14      1222      3.94      4.23      4.48         1
12:50:01 AM         9      1258      4.85      5.03      4.74         1
01:00:01 AM        16      1266      5.06      4.79      4.65         0
01:10:01 AM         7      1237      4.62      4.49      4.53         0
01:20:01 AM        10      1216      3.70      3.61      4.04         1
01:30:01 AM        11      1248      4.50      4.79      4.50         1
01:40:01 AM        15      1176      8.32     23.20     15.08         1
01:50:01 AM        12      1073      3.34      6.15      9.57         0
02:00:01 AM        11      1082      3.25      3.73      6.74         2
02:10:01 AM        18      1080      6.62      5.69      6.22         3
02:20:01 AM        18      1053      7.10      6.92      6.65         1
02:30:01 AM        17      1073      5.99      5.81      6.05         1
02:40:01 AM        20      1094      5.92      6.17      6.13         5
02:50:01 AM        12      1140      6.06     15.61     12.17         6
03:00:01 AM         9      1033      5.05      5.92      8.40         1
03:10:01 AM        16      1053      6.12      5.41      6.81         1
03:20:01 AM        13      1070      3.35      4.50      5.81         1
03:30:01 AM        11      1064      7.99      6.18      5.89         3
03:40:01 AM        12      1035      6.25      5.85      5.86         1
03:50:01 AM        10      1059      6.23      6.41      6.07         1
04:00:01 AM        14      1043      5.43      5.42      5.68         2
04:10:01 AM        16      1046      4.73      5.57      5.76         2
04:20:01 AM        13      1025      5.93      5.55      5.59         2
04:30:01 AM        15      1034      4.28      4.41      4.94         5
04:40:01 AM         8      1028      6.00      5.35      5.05         2
04:50:01 AM        11      1179      5.90      5.70      5.27         1
05:00:01 AM        10      1174      5.05      4.45      4.66         2
05:10:01 AM        12      1174      4.26      4.93      5.13         0
05:20:01 AM        12      1181      3.86      4.49      4.91         1
05:30:01 AM         9      1182      4.71      5.30      5.23         2
05:40:01 AM        10      1190      4.29      5.02      5.12         1
05:50:01 AM        13      1020      3.73      4.02      4.59         3
06:00:01 AM        14      1060      5.41      5.76      5.22         3
06:10:01 AM        16      1021      3.62      4.36      4.73         2
06:20:01 AM        15      1048      6.63      5.45      5.15         2
06:30:01 AM        12      1051      5.24      5.37      5.17         2
06:40:01 AM         8      1067      3.59      3.93      4.51         3
06:50:01 AM        12      1044      4.97      4.47      4.47         4
07:00:01 AM        10      1043      5.57      5.68      5.13         2
07:10:01 AM         7      1020      4.36      4.80      5.03         1
07:20:01 AM        17      1024      5.27      5.22      5.18         2
07:30:01 AM        14      1060      6.16      6.80      6.25         2
07:40:01 AM         9      1041      5.12      6.04      6.16         3
07:50:01 AM        15      1019      4.54      4.78      5.42         3
08:00:01 AM        12      1047      5.46      5.33      5.45         5
08:10:01 AM        13      1047      5.76      5.51      5.47         1
08:20:01 AM         9      1036      4.70      5.63      5.61         2
08:30:01 AM        15      1027      3.80      4.52      5.14         2
08:40:01 AM        14      1216      7.50      5.40      5.11         1
08:50:01 AM        11      1223      6.28      6.47      5.98         1
09:00:01 AM        11      1232      6.45      6.08      5.98         1
09:10:01 AM        12      1345      6.02      6.68      6.48         2
09:20:01 AM         9      1222      5.41      5.60      6.10         2
09:30:01 AM        10      1040      6.80      6.57      6.42         2
09:40:01 AM        11      1053      5.96      6.06      6.27         2
09:50:01 AM        27      1087      6.54      6.59      6.49         1
10:00:01 AM        15      1093      7.38      8.12      7.55         3
10:10:01 AM        10      1066      8.34      8.17      7.82         3

10:10:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
10:20:01 AM        12      1127      5.99      7.29      7.51         2
10:30:01 AM         8      1124      7.55      6.32      6.71         3
10:40:01 AM        18      1094      5.70      6.03      6.38         1
10:50:01 AM        10      1102      5.06      5.59      6.01         2
11:00:01 AM        10      1074      6.46      5.73      5.79         3
11:10:01 AM        13      1087      4.14      5.02      5.63         1
11:20:01 AM        16      1070      5.39      5.60      5.69         2
11:30:01 AM        20      1071      5.85      5.98      5.91         2
11:40:01 AM        12      1367      5.65      6.23      6.14         0
11:50:01 AM        10      1688      4.56      5.28      5.79         2
12:00:01 PM        15      2019      4.42      4.54      5.17         3
12:10:01 PM        10      2434      4.22      4.70      4.97         2
12:20:01 PM        12      3001      4.70      5.10      5.07         0
12:30:01 PM        12      3376      3.97      4.19      4.57         1
12:40:01 PM        10      3777      2.62      3.87      4.39         0
12:50:01 PM         9      4176      4.63      4.87      4.71         0
01:00:01 PM         9      4465      4.29      3.99      4.28         0
01:10:01 PM         6      4342      1.24      1.68      2.90         0
01:20:01 PM        10      1207      2.04      1.56      2.21         0
01:30:01 PM         9      1518      3.30      3.57      3.09         0
01:40:01 PM        11      1855      4.10      3.76      3.45         0
01:50:01 PM        12      2185      5.17      4.45      3.90         0
02:00:01 PM        11      2501      4.52      3.84      3.79         1
02:10:01 PM        10      2749      3.70      3.66      3.78         0
02:20:01 PM        13      3170      3.85      4.03      3.91         0
02:30:01 PM        12      3537      3.57      4.04      4.09         0
02:40:01 PM        11      3868      4.22      4.54      4.43         0
02:50:01 PM        11      4292      5.13      5.13      4.74         0
03:00:01 PM        17      4551      5.51      5.34      5.06         0
03:10:01 PM        12      1469      5.67      5.07      4.93         0
03:20:01 PM         9      1736      6.38      5.99      5.41         4
03:30:01 PM        15      1629      5.32      7.29      6.82         1
03:40:01 PM        15      1204      5.87      6.63      6.86         2
03:50:01 PM        11      1182      7.18      6.59      6.60         1
04:00:01 PM        14      1181      5.85      5.72      6.12         4
04:10:01 PM        17      1178      6.03      6.22      6.22         1
04:20:01 PM        14      1030      6.80      6.29      6.20         1
04:30:01 PM        13      1025      5.28      5.44      5.79         1
04:40:01 PM        10      1042      4.95      5.28      5.59         2
04:50:01 PM        11      1018      8.34      6.94      6.13         1
05:00:01 PM        13      1022      5.78      6.25      6.29         1
05:10:01 PM        11      1031      3.67      4.50      5.43         1
05:20:01 PM        12      1019      6.53      5.97      5.63         1
05:30:01 PM        10      1245      6.89      5.57      5.37         2
05:40:01 PM        15      1218      5.93      6.28      5.83         1
05:50:01 PM         9      1202      3.18      4.11      5.04         0
06:00:01 PM        12      1154      4.33      3.60      4.19         1
06:10:01 PM        10      1148      2.15      3.07      3.75         0
06:20:01 PM        12      1151      2.81      3.00      3.37         0
06:30:01 PM        11      1135      2.90      2.89      3.20         0
06:40:01 PM         8      1140      2.99      2.79      3.00         0
06:50:01 PM         9      1145      3.21      3.18      3.05         0
07:00:01 PM         8      1145      2.94      2.66      2.74         0
07:10:01 PM         9      1140      1.65      2.41      2.69         0
07:20:01 PM        11      1146      2.60      2.58      2.59         0
07:30:01 PM         8      1137      2.85      2.68      2.66         0
07:40:01 PM        11      1148      2.17      2.74      2.75         0
07:50:01 PM        14      1143      2.27      2.49      2.60         0
08:00:01 PM        13      1149      3.46      3.07      2.79         0
08:10:01 PM        10      1135      2.18      2.63      2.72         0
08:20:01 PM        11      1136      2.77      2.71      2.72         1

08:20:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
08:30:01 PM        10      1140      1.91      2.27      2.58         2
08:40:01 PM         9      1138      2.29      2.42      2.56         0
08:50:01 PM        11      1144      3.05      3.22      2.91         1
09:00:01 PM        11      1146      5.32      3.94      3.35         1
Average:           12      1434      4.86      5.15      5.20         1

sar -p
Linux 4.9.87-xxxx-std-ipv6-64 ()      05/30/2018      _x86_64_        (32 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all     12.33      0.00      1.92      0.72      0.00     85.04
12:20:01 AM     all     12.46      0.00      1.95      0.61      0.00     84.99
12:30:01 AM     all     11.09      0.00      1.80      0.57      0.00     86.53
12:40:01 AM     all     12.23      0.00      1.89      0.57      0.00     85.31
12:50:01 AM     all     13.21      0.00      2.03      0.58      0.00     84.18
01:00:01 AM     all     11.91      0.00      1.90      0.69      0.00     85.49
01:10:01 AM     all     11.30      0.00      1.70      0.55      0.00     86.45
01:20:01 AM     all      9.37      0.01      1.63      0.63      0.00     88.36
01:30:01 AM     all     12.05      0.00      2.20      0.56      0.00     85.18
01:40:01 AM     all     12.03      0.00      4.70     15.69      0.00     67.58
01:50:01 AM     all      7.71      0.00      1.18      1.10      0.00     90.00
02:00:01 AM     all      9.17      0.00      1.63      0.79      0.00     88.41
02:10:01 AM     all      7.43      5.26      1.72      1.99      0.00     83.60
02:20:01 AM     all      7.79      8.05      2.24      2.64      0.00     79.28
02:30:01 AM     all      7.88      3.07      2.06      3.78      0.00     83.21
02:40:01 AM     all      8.28      2.71      1.95      4.68      0.00     82.37
02:50:01 AM     all     12.48      1.62      4.57     12.05      0.00     69.27
03:00:01 AM     all      7.01      1.11      1.25      4.19      0.00     86.45
03:10:01 AM     all      8.00      0.92      1.83      3.93      0.00     85.32
03:20:01 AM     all      6.47      0.98      1.54      4.74      0.00     86.27
03:30:01 AM     all      6.63      3.19      1.45      3.13      0.00     85.60
03:40:01 AM     all      6.64      6.45      1.81      2.95      0.00     82.15
03:50:01 AM     all      8.89      3.58      2.35      3.43      0.00     81.74
04:00:01 AM     all      6.55      3.99      1.85      3.34      0.00     84.26
04:10:01 AM     all      8.52      2.91      2.21      4.19      0.00     82.18
04:20:01 AM     all      7.58      1.99      1.90      4.10      0.00     84.43
04:30:01 AM     all      6.66      1.05      1.27      3.88      0.00     87.14
04:40:01 AM     all      7.06      1.36      1.73      3.90      0.00     85.96
04:50:01 AM     all      7.40      0.31      1.31      5.67      0.00     85.31
05:00:01 AM     all      7.59      0.03      1.13      3.59      0.00     87.66
05:10:01 AM     all     10.70      0.08      2.03      3.80      0.00     83.39
05:20:01 AM     all      8.30      0.52      1.57      3.76      0.00     85.84
05:30:01 AM     all      9.23      0.03      1.66      4.09      0.00     84.99
05:40:01 AM     all      7.11      0.28      1.21      4.75      0.00     86.65
05:50:01 AM     all      6.33      1.29      1.25      3.62      0.00     87.52
06:00:01 AM     all      8.13      1.75      1.88      3.71      0.00     84.54
06:10:01 AM     all      5.74      1.79      1.26      3.85      0.00     87.37
06:20:01 AM     all      8.26      1.84      1.55      3.90      0.00     84.44
06:30:01 AM     all      7.75      2.03      1.85      3.74      0.00     84.63
06:40:01 AM     all      5.68      1.41      1.18      3.62      0.00     88.11
06:50:01 AM     all      6.61      1.70      1.61      3.59      0.00     86.49
07:00:01 AM     all      7.19      1.97      1.69      3.66      0.00     85.49
07:10:01 AM     all      6.75      1.52      1.55      3.68      0.00     86.49
07:20:01 AM     all      7.44      1.64      1.77      3.80      0.00     85.35
07:30:01 AM     all     11.35      1.52      3.03      4.79      0.00     79.30
07:40:01 AM     all      9.46      1.78      1.71      4.05      0.00     83.01
07:50:01 AM     all      6.87      1.46      1.34      3.99      0.00     86.33
08:00:01 AM     all      9.21      1.48      1.59      3.85      0.00     83.88
08:10:01 AM     all      9.35      1.74      1.70      4.11      0.00     83.10
08:20:01 AM     all      9.82      1.01      1.87      3.96      0.00     83.33
08:30:01 AM     all      8.04      0.01      1.26      4.35      0.00     86.34
08:40:01 AM     all      8.16      0.05      1.79      3.80      0.00     86.20
08:50:01 AM     all      9.62      3.14      2.32      3.77      0.00     81.15
09:00:01 AM     all      8.41      2.20      2.05      4.37      0.00     82.97
09:10:01 AM     all     11.35      2.39      2.65      4.67      0.00     78.95
09:20:01 AM     all      8.66      3.42      2.36      3.88      0.00     81.69
09:30:01 AM     all      9.49      4.10      2.58      3.86      0.00     79.97
09:40:01 AM     all      8.09      3.70      1.97      4.03      0.00     82.21
09:50:01 AM     all      9.96      4.46      1.98      3.37      0.00     80.24
10:00:01 AM     all     13.09      6.68      2.98      3.21      0.00     74.03
10:10:01 AM     all     11.08      5.78      2.45      3.74      0.00     76.95

10:10:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:20:01 AM     all     10.55      4.94      2.89      3.07      0.00     78.55
10:30:01 AM     all      8.08      3.31      2.01      3.16      0.00     83.45
10:40:01 AM     all      8.71      3.15      2.13      4.05      0.00     81.95
10:50:01 AM     all      8.89      1.40      1.94      3.90      0.00     83.87
11:00:01 AM     all      8.70      3.29      1.74      3.54      0.00     82.73
11:10:01 AM     all      8.96      2.89      1.74      3.44      0.00     82.97
11:20:01 AM     all     10.62      0.88      1.93      3.13      0.00     83.44
11:30:01 AM     all     10.88      2.19      2.40      2.33      0.00     82.21
11:40:01 AM     all     13.85      0.81      2.34      1.31      0.00     81.70
11:50:01 AM     all     13.16      0.15      2.06      0.81      0.00     83.83
12:00:01 PM     all     11.92      0.00      1.58      0.58      0.00     85.92
12:10:01 PM     all     11.76      0.00      1.68      0.61      0.00     85.96
12:20:01 PM     all     13.54      0.00      2.09      0.50      0.00     83.87
12:30:01 PM     all      8.27      0.00      2.63      0.45      0.00     88.64
12:40:01 PM     all      9.44      0.00      2.93      0.42      0.00     87.21
12:50:01 PM     all     10.99      0.00      3.31      0.45      0.00     85.25
01:00:01 PM     all      8.26      0.00      2.94      0.53      0.00     88.27
01:10:01 PM     all      2.49      0.00      2.20      0.16      0.00     95.15
01:20:01 PM     all      2.01      0.01      2.17      0.15      0.00     95.67
01:30:01 PM     all      8.97      0.00      2.76      0.69      0.00     87.58
01:40:01 PM     all      9.18      0.00      2.56      0.49      0.00     87.77
01:50:01 PM     all      9.83      0.00      2.67      0.54      0.00     86.97
02:00:01 PM     all      8.29      0.00      2.53      0.57      0.00     88.61
02:10:01 PM     all      9.48      0.00      1.34      0.55      0.00     88.63
02:20:01 PM     all      9.36      0.00      1.41      0.48      0.00     88.74
02:30:01 PM     all     11.04      0.00      1.61      0.43      0.00     86.92
02:40:01 PM     all     12.46      0.00      1.83      0.48      0.00     85.22
02:50:01 PM     all     10.51      0.00      3.81      0.59      0.00     85.08
03:00:01 PM     all     11.98      0.00      4.10      0.54      0.00     83.38
03:10:01 PM     all     10.25      0.00      3.99      0.69      0.00     85.07
03:20:01 PM     all     12.44      0.01      4.39      0.82      0.00     82.35
03:30:01 PM     all     10.62      1.95      3.91      4.67      0.00     78.85
03:40:01 PM     all     10.53      2.46      3.28      4.01      0.00     79.72
03:50:01 PM     all     10.43      0.93      2.37      4.61      0.00     81.66
04:00:01 PM     all      8.33      0.88      2.10      4.14      0.00     84.55
04:10:01 PM     all     10.75      2.28      1.98      3.38      0.00     81.61
04:20:01 PM     all      9.59      2.63      1.86      3.57      0.00     82.35
04:30:01 PM     all      8.44      1.75      1.65      3.82      0.00     84.35
04:40:01 PM     all      9.38      1.38      1.74      3.68      0.00     83.83
04:50:01 PM     all      9.57      3.48      2.02      3.54      0.00     81.38
05:00:01 PM     all      8.95      5.17      1.96      3.33      0.00     80.58
05:10:01 PM     all      7.99      0.97      1.49      3.50      0.00     86.06
05:20:01 PM     all      7.47      4.44      2.24      2.94      0.00     82.91
05:30:01 PM     all      8.25      1.19      2.09      4.00      0.00     84.47
05:40:01 PM     all      9.35      2.30      2.20      3.97      0.00     82.18
05:50:01 PM     all      7.91      0.93      1.84      1.74      0.00     87.57
06:00:01 PM     all      7.91      0.00      1.41      0.42      0.00     90.27
06:10:01 PM     all      8.54      0.00      1.53      0.38      0.00     89.55
06:20:01 PM     all      6.27      0.00      1.21      0.38      0.00     92.15
06:30:01 PM     all      8.09      0.00      1.34      0.36      0.00     90.22
06:40:01 PM     all      7.53      0.00      1.32      0.35      0.00     90.79
06:50:01 PM     all      7.31      0.00      1.31      0.35      0.00     91.04
07:00:01 PM     all      6.62      0.00      1.23      0.34      0.00     91.81
07:10:01 PM     all      8.24      0.01      1.39      0.42      0.00     89.94
07:20:01 PM     all      7.17      0.00      1.29      0.26      0.00     91.28
07:30:01 PM     all      6.86      0.00      1.46      0.29      0.00     91.39
07:40:01 PM     all      7.33      0.00      1.25      0.37      0.00     91.05
07:50:01 PM     all      5.70      0.00      1.23      0.31      0.00     92.75
08:00:01 PM     all      6.67      0.00      1.36      0.26      0.00     91.71
08:10:01 PM     all      7.51      0.00      1.35      0.26      0.00     90.87
08:20:01 PM     all      6.51      0.00      1.27      0.28      0.00     91.93

08:20:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:30:01 PM     all      6.74      0.00      1.27      0.30      0.00     91.69
08:40:01 PM     all      6.72      0.00      1.22      0.27      0.00     91.79
08:50:01 PM     all      8.97      0.00      1.49      0.30      0.00     89.24
09:00:01 PM     all      9.99      0.00      1.70      0.27      0.00     88.04
Average:        all      8.92      1.33      1.99      2.55      0.00     85.22

grep -i "Out of Memory" /var/log/messages
May 30 01:34:26 kernel: Out of memory: Kill process 5314 (mysqld) score 179 or sacrifice child
May 30 01:35:14 kernel: Out of memory: Kill process 11083 (mysqld) score 38 or sacrifice child
May 30 01:35:37 kernel: Out of memory: Kill process 11690 (mysqld) score 19 or sacrifice child
May 30 01:36:32 kernel: Out of memory: Kill process 5941 (named) score 8 or sacrifice child
May 30 01:36:39 kernel: Out of memory: Kill process 11878 (mysqld) score 11 or sacrifice child
May 30 02:45:12 kernel: Out of memory: Kill process 12259 (mysqld) score 46 or sacrifice child
May 30 02:45:40 kernel: Out of memory: Kill process 13100 (named) score 6 or sacrifice child
May 30 02:45:56 kernel: Out of memory: Kill process 29233 (mysqld) score 9 or sacrifice child

I'm tracing the OOM issue but don't think it's related as the timestamps don't match.

cPanelLauren · May 31, 2018

Hi @jsparks

You are correct that the timestamps don't match and it's MySQL being killed so they may not be directly related but was I am gathering from this output is you do have what appears to be consistently medium/high load on the server as well as some IO wait spikes. Are you on a VPS or standard environment?

I'd also like to know what's in the logs between/before/after these warnings:

jsparks said:
[Wed May 30 13:18:42.782001 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5377 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782033 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5561 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782042 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 8576 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782051 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 9776 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782059 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 11882 still did not exit, sending a SIGTERM

Thanks!

jsparks · May 31, 2018

Hi Lauren,

This is a dedicated machine which is running very low load (typically). What you are seeing in the sar output is likely a result of the cpbackup that started May 30 early in the AM, which I believe is what caused SQL to run OOM. I don't think this issue that I'm seeing with Apache is directly correlated with cpbackup, as it did happen a few times outside of backup windows.

Output before warning:

[y 30 13:13:45.018854 2018Wed May 30 13:11:18.422118 2018] [proxy_fcgi:error] [pid 4420:tid 140261573338880] (70007)The timeout specified has expired: [client X:59826] AH01075: Error dispatching request to : (polling), referer: Google
[Wed Ma 30 13:13:45.018854 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00297: SIGUSR1 received. Doing graceful restart
[Wed May 30 13:13:48.114379 2018] [ssl:warn] [pid 30722:tid 140261823477888] AH01909: proxy-subdomains-ssl-default-vhost.localhost:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 30 13:13:48.118733 2018] [ssl:warn] [pid 30722:tid 140261823477888] AH01909: X:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 30 13:13:48.157406 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00292: Apache/2.4.33 (cPanel) OpenSSL/1.0.2o mod_bwlimited/1.4 mod_fcgid/2.3.9 configured -- resuming normal operations
[Wed May 30 13:13:48.157437 2018] [core:notice] [pid 30722:tid 140261823477888] AH00094: Command line: '/usr/sbin/httpd'

Output after:

Truncated, >100 child process errors.
[Wed May 30 13:18:48.822710 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3185 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.823154 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3452 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.833602 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3543 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.833863 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3845 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.834489 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4203 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.834952 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4420 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835105 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4629 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835363 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4841 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835598 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4923 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835768 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 5141 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835954 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 8263 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.836204 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 8191 still did not exit, sending a SIGKILL

This is where I manually restarted Apache via WHM CP.
[Wed May 30 13:19:38.468224 2018] [ssl:warn] [pid 9786:tid 139695678261376] AH01909: proxy-subdomains-ssl-default-vhost.localhost:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 30 13:19:38.472819 2018] [ssl:warn] [pid 9786:tid 139695678261376] AH01909: X.com:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 30 13:19:38.497125 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity for Apache/2.9.2 (ModSecurity: Open Source Web Application Firewall) configured.
[Wed May 30 13:19:38.497133 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: APR compiled version="1.6.3"; loaded version="1.6.3"
[Wed May 30 13:19:38.497136 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Wed May 30 13:19:38.497138 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: LUA compiled version="Lua 5.1"
[Wed May 30 13:19:38.497140 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: LIBXML compiled version="2.9.7"
[Wed May 30 13:19:38.497142 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.

cPanelLauren · May 31, 2018

Hi @jsparks

Could you please open a ticket using the link in my signature? I feel like it might be better to troubleshoot this issue with access to the system. Once the ticket is open please reply to this thread with the ticket ID so we can update the thread with the outcome.

Thanks!

Thread starter	Similar threads	Forum	Replies	Date
	SOLVED Does this Apache status look right?	Web Servers and Applications	4	Mar 7, 2022
	SOLVED /usr/local/cpanel/scripts/restartsrv_httpd doesn't restart Apache	Web Servers and Applications	5	May 2, 2020
D	EasyApache 4 does not support the requested Apache version error	Web Servers and Applications	2	Dec 26, 2018
R	migirate to EA_4 error “Cpanel::Easy::Apache::Ident” ignored since it does not have an RPM	Web Servers and Applications	1	Dec 12, 2018
G	SOLVED EA4 Migration Fails, So Does Apache 2.4 Build	Web Servers and Applications	2	Aug 28, 2017

Search

Search

Apache does not recover from SIGUSR1

jsparks

Member

cPanelLauren

Product Owner II

jsparks

Member

cPanelLauren

Product Owner II

jsparks

Member

cPanelLauren

Product Owner II