Apache does not recover from SIGUSR1

jsparks

Member
Sep 16, 2015
7
0
1
Halifax
cPanel Access Level
Root Administrator
Hi folks,

I've been using php-fpm / apach2.4 / engintron (nginx reverse proxy) for quite some time now and noticed every few days apache dies and does not recover until I manually intervene and restart the process.

The server has has no resource limitation issues that I am aware of, and from what I can tell, everything is configured as it should.

I see in the apache error.log the following:

[Wed May 30 13:13:45.018854 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00297: SIGUSR1 received. Doing graceful restart
[Wed May 30 13:13:48.157406 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00292: Apache/2.4.33 (cPanel) OpenSSL/1.0.2o mod_bwlimited/1.4 mod_fcgid/2.3.9 configured -- resuming normal operations
[Wed May 30 13:13:48.157437 2018] [core:notice] [pid 30722:tid 140261823477888] AH00094: Command line: '/usr/sbin/httpd'

followed by many:

[Wed May 30 13:18:42.782001 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5377 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782033 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5561 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782042 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 8576 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782051 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 9776 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782059 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 11882 still did not exit, sending a SIGTERM

I checked the php-fpm logs in both /opt/cpanel/ea-php70/root/usr/var/log/php-fpm/ & /opt/cpanel/ea-php56/root/usr/var/log/php-fpm/ and seen no errors or warnings.. What else could be controlling this process?

Thanks in advance.
 

jsparks

Member
Sep 16, 2015
7
0
1
Halifax
cPanel Access Level
Root Administrator
I never actually configured this server to run apache only, it's always been apache/php-fpm/nginx, but I can try re-configuring it if you think it's worth testing.

Please see output:

cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)

cat /usr/local/cpanel/version
11.70.0.46
Code:
sar -q
Linux 4.9.87-xxxx-std-ipv6-64 ()      05/30/2018      _x86_64_        (32 CPU)

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
12:10:01 AM        13      1249      5.79      4.77      4.89         0
12:20:01 AM        12      1235      3.99      4.62      4.79         1
12:30:01 AM        10      1240      3.85      4.41      4.60         1
12:40:01 AM        14      1222      3.94      4.23      4.48         1
12:50:01 AM         9      1258      4.85      5.03      4.74         1
01:00:01 AM        16      1266      5.06      4.79      4.65         0
01:10:01 AM         7      1237      4.62      4.49      4.53         0
01:20:01 AM        10      1216      3.70      3.61      4.04         1
01:30:01 AM        11      1248      4.50      4.79      4.50         1
01:40:01 AM        15      1176      8.32     23.20     15.08         1
01:50:01 AM        12      1073      3.34      6.15      9.57         0
02:00:01 AM        11      1082      3.25      3.73      6.74         2
02:10:01 AM        18      1080      6.62      5.69      6.22         3
02:20:01 AM        18      1053      7.10      6.92      6.65         1
02:30:01 AM        17      1073      5.99      5.81      6.05         1
02:40:01 AM        20      1094      5.92      6.17      6.13         5
02:50:01 AM        12      1140      6.06     15.61     12.17         6
03:00:01 AM         9      1033      5.05      5.92      8.40         1
03:10:01 AM        16      1053      6.12      5.41      6.81         1
03:20:01 AM        13      1070      3.35      4.50      5.81         1
03:30:01 AM        11      1064      7.99      6.18      5.89         3
03:40:01 AM        12      1035      6.25      5.85      5.86         1
03:50:01 AM        10      1059      6.23      6.41      6.07         1
04:00:01 AM        14      1043      5.43      5.42      5.68         2
04:10:01 AM        16      1046      4.73      5.57      5.76         2
04:20:01 AM        13      1025      5.93      5.55      5.59         2
04:30:01 AM        15      1034      4.28      4.41      4.94         5
04:40:01 AM         8      1028      6.00      5.35      5.05         2
04:50:01 AM        11      1179      5.90      5.70      5.27         1
05:00:01 AM        10      1174      5.05      4.45      4.66         2
05:10:01 AM        12      1174      4.26      4.93      5.13         0
05:20:01 AM        12      1181      3.86      4.49      4.91         1
05:30:01 AM         9      1182      4.71      5.30      5.23         2
05:40:01 AM        10      1190      4.29      5.02      5.12         1
05:50:01 AM        13      1020      3.73      4.02      4.59         3
06:00:01 AM        14      1060      5.41      5.76      5.22         3
06:10:01 AM        16      1021      3.62      4.36      4.73         2
06:20:01 AM        15      1048      6.63      5.45      5.15         2
06:30:01 AM        12      1051      5.24      5.37      5.17         2
06:40:01 AM         8      1067      3.59      3.93      4.51         3
06:50:01 AM        12      1044      4.97      4.47      4.47         4
07:00:01 AM        10      1043      5.57      5.68      5.13         2
07:10:01 AM         7      1020      4.36      4.80      5.03         1
07:20:01 AM        17      1024      5.27      5.22      5.18         2
07:30:01 AM        14      1060      6.16      6.80      6.25         2
07:40:01 AM         9      1041      5.12      6.04      6.16         3
07:50:01 AM        15      1019      4.54      4.78      5.42         3
08:00:01 AM        12      1047      5.46      5.33      5.45         5
08:10:01 AM        13      1047      5.76      5.51      5.47         1
08:20:01 AM         9      1036      4.70      5.63      5.61         2
08:30:01 AM        15      1027      3.80      4.52      5.14         2
08:40:01 AM        14      1216      7.50      5.40      5.11         1
08:50:01 AM        11      1223      6.28      6.47      5.98         1
09:00:01 AM        11      1232      6.45      6.08      5.98         1
09:10:01 AM        12      1345      6.02      6.68      6.48         2
09:20:01 AM         9      1222      5.41      5.60      6.10         2
09:30:01 AM        10      1040      6.80      6.57      6.42         2
09:40:01 AM        11      1053      5.96      6.06      6.27         2
09:50:01 AM        27      1087      6.54      6.59      6.49         1
10:00:01 AM        15      1093      7.38      8.12      7.55         3
10:10:01 AM        10      1066      8.34      8.17      7.82         3

10:10:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
10:20:01 AM        12      1127      5.99      7.29      7.51         2
10:30:01 AM         8      1124      7.55      6.32      6.71         3
10:40:01 AM        18      1094      5.70      6.03      6.38         1
10:50:01 AM        10      1102      5.06      5.59      6.01         2
11:00:01 AM        10      1074      6.46      5.73      5.79         3
11:10:01 AM        13      1087      4.14      5.02      5.63         1
11:20:01 AM        16      1070      5.39      5.60      5.69         2
11:30:01 AM        20      1071      5.85      5.98      5.91         2
11:40:01 AM        12      1367      5.65      6.23      6.14         0
11:50:01 AM        10      1688      4.56      5.28      5.79         2
12:00:01 PM        15      2019      4.42      4.54      5.17         3
12:10:01 PM        10      2434      4.22      4.70      4.97         2
12:20:01 PM        12      3001      4.70      5.10      5.07         0
12:30:01 PM        12      3376      3.97      4.19      4.57         1
12:40:01 PM        10      3777      2.62      3.87      4.39         0
12:50:01 PM         9      4176      4.63      4.87      4.71         0
01:00:01 PM         9      4465      4.29      3.99      4.28         0
01:10:01 PM         6      4342      1.24      1.68      2.90         0
01:20:01 PM        10      1207      2.04      1.56      2.21         0
01:30:01 PM         9      1518      3.30      3.57      3.09         0
01:40:01 PM        11      1855      4.10      3.76      3.45         0
01:50:01 PM        12      2185      5.17      4.45      3.90         0
02:00:01 PM        11      2501      4.52      3.84      3.79         1
02:10:01 PM        10      2749      3.70      3.66      3.78         0
02:20:01 PM        13      3170      3.85      4.03      3.91         0
02:30:01 PM        12      3537      3.57      4.04      4.09         0
02:40:01 PM        11      3868      4.22      4.54      4.43         0
02:50:01 PM        11      4292      5.13      5.13      4.74         0
03:00:01 PM        17      4551      5.51      5.34      5.06         0
03:10:01 PM        12      1469      5.67      5.07      4.93         0
03:20:01 PM         9      1736      6.38      5.99      5.41         4
03:30:01 PM        15      1629      5.32      7.29      6.82         1
03:40:01 PM        15      1204      5.87      6.63      6.86         2
03:50:01 PM        11      1182      7.18      6.59      6.60         1
04:00:01 PM        14      1181      5.85      5.72      6.12         4
04:10:01 PM        17      1178      6.03      6.22      6.22         1
04:20:01 PM        14      1030      6.80      6.29      6.20         1
04:30:01 PM        13      1025      5.28      5.44      5.79         1
04:40:01 PM        10      1042      4.95      5.28      5.59         2
04:50:01 PM        11      1018      8.34      6.94      6.13         1
05:00:01 PM        13      1022      5.78      6.25      6.29         1
05:10:01 PM        11      1031      3.67      4.50      5.43         1
05:20:01 PM        12      1019      6.53      5.97      5.63         1
05:30:01 PM        10      1245      6.89      5.57      5.37         2
05:40:01 PM        15      1218      5.93      6.28      5.83         1
05:50:01 PM         9      1202      3.18      4.11      5.04         0
06:00:01 PM        12      1154      4.33      3.60      4.19         1
06:10:01 PM        10      1148      2.15      3.07      3.75         0
06:20:01 PM        12      1151      2.81      3.00      3.37         0
06:30:01 PM        11      1135      2.90      2.89      3.20         0
06:40:01 PM         8      1140      2.99      2.79      3.00         0
06:50:01 PM         9      1145      3.21      3.18      3.05         0
07:00:01 PM         8      1145      2.94      2.66      2.74         0
07:10:01 PM         9      1140      1.65      2.41      2.69         0
07:20:01 PM        11      1146      2.60      2.58      2.59         0
07:30:01 PM         8      1137      2.85      2.68      2.66         0
07:40:01 PM        11      1148      2.17      2.74      2.75         0
07:50:01 PM        14      1143      2.27      2.49      2.60         0
08:00:01 PM        13      1149      3.46      3.07      2.79         0
08:10:01 PM        10      1135      2.18      2.63      2.72         0
08:20:01 PM        11      1136      2.77      2.71      2.72         1

08:20:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
08:30:01 PM        10      1140      1.91      2.27      2.58         2
08:40:01 PM         9      1138      2.29      2.42      2.56         0
08:50:01 PM        11      1144      3.05      3.22      2.91         1
09:00:01 PM        11      1146      5.32      3.94      3.35         1
Average:           12      1434      4.86      5.15      5.20         1

sar -p
Linux 4.9.87-xxxx-std-ipv6-64 ()      05/30/2018      _x86_64_        (32 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all     12.33      0.00      1.92      0.72      0.00     85.04
12:20:01 AM     all     12.46      0.00      1.95      0.61      0.00     84.99
12:30:01 AM     all     11.09      0.00      1.80      0.57      0.00     86.53
12:40:01 AM     all     12.23      0.00      1.89      0.57      0.00     85.31
12:50:01 AM     all     13.21      0.00      2.03      0.58      0.00     84.18
01:00:01 AM     all     11.91      0.00      1.90      0.69      0.00     85.49
01:10:01 AM     all     11.30      0.00      1.70      0.55      0.00     86.45
01:20:01 AM     all      9.37      0.01      1.63      0.63      0.00     88.36
01:30:01 AM     all     12.05      0.00      2.20      0.56      0.00     85.18
01:40:01 AM     all     12.03      0.00      4.70     15.69      0.00     67.58
01:50:01 AM     all      7.71      0.00      1.18      1.10      0.00     90.00
02:00:01 AM     all      9.17      0.00      1.63      0.79      0.00     88.41
02:10:01 AM     all      7.43      5.26      1.72      1.99      0.00     83.60
02:20:01 AM     all      7.79      8.05      2.24      2.64      0.00     79.28
02:30:01 AM     all      7.88      3.07      2.06      3.78      0.00     83.21
02:40:01 AM     all      8.28      2.71      1.95      4.68      0.00     82.37
02:50:01 AM     all     12.48      1.62      4.57     12.05      0.00     69.27
03:00:01 AM     all      7.01      1.11      1.25      4.19      0.00     86.45
03:10:01 AM     all      8.00      0.92      1.83      3.93      0.00     85.32
03:20:01 AM     all      6.47      0.98      1.54      4.74      0.00     86.27
03:30:01 AM     all      6.63      3.19      1.45      3.13      0.00     85.60
03:40:01 AM     all      6.64      6.45      1.81      2.95      0.00     82.15
03:50:01 AM     all      8.89      3.58      2.35      3.43      0.00     81.74
04:00:01 AM     all      6.55      3.99      1.85      3.34      0.00     84.26
04:10:01 AM     all      8.52      2.91      2.21      4.19      0.00     82.18
04:20:01 AM     all      7.58      1.99      1.90      4.10      0.00     84.43
04:30:01 AM     all      6.66      1.05      1.27      3.88      0.00     87.14
04:40:01 AM     all      7.06      1.36      1.73      3.90      0.00     85.96
04:50:01 AM     all      7.40      0.31      1.31      5.67      0.00     85.31
05:00:01 AM     all      7.59      0.03      1.13      3.59      0.00     87.66
05:10:01 AM     all     10.70      0.08      2.03      3.80      0.00     83.39
05:20:01 AM     all      8.30      0.52      1.57      3.76      0.00     85.84
05:30:01 AM     all      9.23      0.03      1.66      4.09      0.00     84.99
05:40:01 AM     all      7.11      0.28      1.21      4.75      0.00     86.65
05:50:01 AM     all      6.33      1.29      1.25      3.62      0.00     87.52
06:00:01 AM     all      8.13      1.75      1.88      3.71      0.00     84.54
06:10:01 AM     all      5.74      1.79      1.26      3.85      0.00     87.37
06:20:01 AM     all      8.26      1.84      1.55      3.90      0.00     84.44
06:30:01 AM     all      7.75      2.03      1.85      3.74      0.00     84.63
06:40:01 AM     all      5.68      1.41      1.18      3.62      0.00     88.11
06:50:01 AM     all      6.61      1.70      1.61      3.59      0.00     86.49
07:00:01 AM     all      7.19      1.97      1.69      3.66      0.00     85.49
07:10:01 AM     all      6.75      1.52      1.55      3.68      0.00     86.49
07:20:01 AM     all      7.44      1.64      1.77      3.80      0.00     85.35
07:30:01 AM     all     11.35      1.52      3.03      4.79      0.00     79.30
07:40:01 AM     all      9.46      1.78      1.71      4.05      0.00     83.01
07:50:01 AM     all      6.87      1.46      1.34      3.99      0.00     86.33
08:00:01 AM     all      9.21      1.48      1.59      3.85      0.00     83.88
08:10:01 AM     all      9.35      1.74      1.70      4.11      0.00     83.10
08:20:01 AM     all      9.82      1.01      1.87      3.96      0.00     83.33
08:30:01 AM     all      8.04      0.01      1.26      4.35      0.00     86.34
08:40:01 AM     all      8.16      0.05      1.79      3.80      0.00     86.20
08:50:01 AM     all      9.62      3.14      2.32      3.77      0.00     81.15
09:00:01 AM     all      8.41      2.20      2.05      4.37      0.00     82.97
09:10:01 AM     all     11.35      2.39      2.65      4.67      0.00     78.95
09:20:01 AM     all      8.66      3.42      2.36      3.88      0.00     81.69
09:30:01 AM     all      9.49      4.10      2.58      3.86      0.00     79.97
09:40:01 AM     all      8.09      3.70      1.97      4.03      0.00     82.21
09:50:01 AM     all      9.96      4.46      1.98      3.37      0.00     80.24
10:00:01 AM     all     13.09      6.68      2.98      3.21      0.00     74.03
10:10:01 AM     all     11.08      5.78      2.45      3.74      0.00     76.95

10:10:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:20:01 AM     all     10.55      4.94      2.89      3.07      0.00     78.55
10:30:01 AM     all      8.08      3.31      2.01      3.16      0.00     83.45
10:40:01 AM     all      8.71      3.15      2.13      4.05      0.00     81.95
10:50:01 AM     all      8.89      1.40      1.94      3.90      0.00     83.87
11:00:01 AM     all      8.70      3.29      1.74      3.54      0.00     82.73
11:10:01 AM     all      8.96      2.89      1.74      3.44      0.00     82.97
11:20:01 AM     all     10.62      0.88      1.93      3.13      0.00     83.44
11:30:01 AM     all     10.88      2.19      2.40      2.33      0.00     82.21
11:40:01 AM     all     13.85      0.81      2.34      1.31      0.00     81.70
11:50:01 AM     all     13.16      0.15      2.06      0.81      0.00     83.83
12:00:01 PM     all     11.92      0.00      1.58      0.58      0.00     85.92
12:10:01 PM     all     11.76      0.00      1.68      0.61      0.00     85.96
12:20:01 PM     all     13.54      0.00      2.09      0.50      0.00     83.87
12:30:01 PM     all      8.27      0.00      2.63      0.45      0.00     88.64
12:40:01 PM     all      9.44      0.00      2.93      0.42      0.00     87.21
12:50:01 PM     all     10.99      0.00      3.31      0.45      0.00     85.25
01:00:01 PM     all      8.26      0.00      2.94      0.53      0.00     88.27
01:10:01 PM     all      2.49      0.00      2.20      0.16      0.00     95.15
01:20:01 PM     all      2.01      0.01      2.17      0.15      0.00     95.67
01:30:01 PM     all      8.97      0.00      2.76      0.69      0.00     87.58
01:40:01 PM     all      9.18      0.00      2.56      0.49      0.00     87.77
01:50:01 PM     all      9.83      0.00      2.67      0.54      0.00     86.97
02:00:01 PM     all      8.29      0.00      2.53      0.57      0.00     88.61
02:10:01 PM     all      9.48      0.00      1.34      0.55      0.00     88.63
02:20:01 PM     all      9.36      0.00      1.41      0.48      0.00     88.74
02:30:01 PM     all     11.04      0.00      1.61      0.43      0.00     86.92
02:40:01 PM     all     12.46      0.00      1.83      0.48      0.00     85.22
02:50:01 PM     all     10.51      0.00      3.81      0.59      0.00     85.08
03:00:01 PM     all     11.98      0.00      4.10      0.54      0.00     83.38
03:10:01 PM     all     10.25      0.00      3.99      0.69      0.00     85.07
03:20:01 PM     all     12.44      0.01      4.39      0.82      0.00     82.35
03:30:01 PM     all     10.62      1.95      3.91      4.67      0.00     78.85
03:40:01 PM     all     10.53      2.46      3.28      4.01      0.00     79.72
03:50:01 PM     all     10.43      0.93      2.37      4.61      0.00     81.66
04:00:01 PM     all      8.33      0.88      2.10      4.14      0.00     84.55
04:10:01 PM     all     10.75      2.28      1.98      3.38      0.00     81.61
04:20:01 PM     all      9.59      2.63      1.86      3.57      0.00     82.35
04:30:01 PM     all      8.44      1.75      1.65      3.82      0.00     84.35
04:40:01 PM     all      9.38      1.38      1.74      3.68      0.00     83.83
04:50:01 PM     all      9.57      3.48      2.02      3.54      0.00     81.38
05:00:01 PM     all      8.95      5.17      1.96      3.33      0.00     80.58
05:10:01 PM     all      7.99      0.97      1.49      3.50      0.00     86.06
05:20:01 PM     all      7.47      4.44      2.24      2.94      0.00     82.91
05:30:01 PM     all      8.25      1.19      2.09      4.00      0.00     84.47
05:40:01 PM     all      9.35      2.30      2.20      3.97      0.00     82.18
05:50:01 PM     all      7.91      0.93      1.84      1.74      0.00     87.57
06:00:01 PM     all      7.91      0.00      1.41      0.42      0.00     90.27
06:10:01 PM     all      8.54      0.00      1.53      0.38      0.00     89.55
06:20:01 PM     all      6.27      0.00      1.21      0.38      0.00     92.15
06:30:01 PM     all      8.09      0.00      1.34      0.36      0.00     90.22
06:40:01 PM     all      7.53      0.00      1.32      0.35      0.00     90.79
06:50:01 PM     all      7.31      0.00      1.31      0.35      0.00     91.04
07:00:01 PM     all      6.62      0.00      1.23      0.34      0.00     91.81
07:10:01 PM     all      8.24      0.01      1.39      0.42      0.00     89.94
07:20:01 PM     all      7.17      0.00      1.29      0.26      0.00     91.28
07:30:01 PM     all      6.86      0.00      1.46      0.29      0.00     91.39
07:40:01 PM     all      7.33      0.00      1.25      0.37      0.00     91.05
07:50:01 PM     all      5.70      0.00      1.23      0.31      0.00     92.75
08:00:01 PM     all      6.67      0.00      1.36      0.26      0.00     91.71
08:10:01 PM     all      7.51      0.00      1.35      0.26      0.00     90.87
08:20:01 PM     all      6.51      0.00      1.27      0.28      0.00     91.93

08:20:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:30:01 PM     all      6.74      0.00      1.27      0.30      0.00     91.69
08:40:01 PM     all      6.72      0.00      1.22      0.27      0.00     91.79
08:50:01 PM     all      8.97      0.00      1.49      0.30      0.00     89.24
09:00:01 PM     all      9.99      0.00      1.70      0.27      0.00     88.04
Average:        all      8.92      1.33      1.99      2.55      0.00     85.22

grep -i "Out of Memory" /var/log/messages
May 30 01:34:26 kernel: Out of memory: Kill process 5314 (mysqld) score 179 or sacrifice child
May 30 01:35:14 kernel: Out of memory: Kill process 11083 (mysqld) score 38 or sacrifice child
May 30 01:35:37 kernel: Out of memory: Kill process 11690 (mysqld) score 19 or sacrifice child
May 30 01:36:32 kernel: Out of memory: Kill process 5941 (named) score 8 or sacrifice child
May 30 01:36:39 kernel: Out of memory: Kill process 11878 (mysqld) score 11 or sacrifice child
May 30 02:45:12 kernel: Out of memory: Kill process 12259 (mysqld) score 46 or sacrifice child
May 30 02:45:40 kernel: Out of memory: Kill process 13100 (named) score 6 or sacrifice child
May 30 02:45:56 kernel: Out of memory: Kill process 29233 (mysqld) score 9 or sacrifice child
I'm tracing the OOM issue but don't think it's related as the timestamps don't match.
 
Last edited by a moderator:

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
7,904
625
263
Houston
cPanel Access Level
DataCenter Provider
Hi @jsparks

You are correct that the timestamps don't match and it's MySQL being killed so they may not be directly related but was I am gathering from this output is you do have what appears to be consistently medium/high load on the server as well as some IO wait spikes. Are you on a VPS or standard environment?

I'd also like to know what's in the logs between/before/after these warnings:

[Wed May 30 13:18:42.782001 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5377 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782033 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5561 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782042 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 8576 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782051 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 9776 still did not exit, sending a SIGTERM
[Wed May 30 13:18:42.782059 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 11882 still did not exit, sending a SIGTERM
Thanks!
 

jsparks

Member
Sep 16, 2015
7
0
1
Halifax
cPanel Access Level
Root Administrator
Hi Lauren,

This is a dedicated machine which is running very low load (typically). What you are seeing in the sar output is likely a result of the cpbackup that started May 30 early in the AM, which I believe is what caused SQL to run OOM. I don't think this issue that I'm seeing with Apache is directly correlated with cpbackup, as it did happen a few times outside of backup windows.

Output before warning:

[y 30 13:13:45.018854 2018Wed May 30 13:11:18.422118 2018] [proxy_fcgi:error] [pid 4420:tid 140261573338880] (70007)The timeout specified has expired: [client X:59826] AH01075: Error dispatching request to : (polling), referer: Google
[Wed Ma 30 13:13:45.018854 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00297: SIGUSR1 received. Doing graceful restart
[Wed May 30 13:13:48.114379 2018] [ssl:warn] [pid 30722:tid 140261823477888] AH01909: proxy-subdomains-ssl-default-vhost.localhost:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 30 13:13:48.118733 2018] [ssl:warn] [pid 30722:tid 140261823477888] AH01909: X:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 30 13:13:48.157406 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00292: Apache/2.4.33 (cPanel) OpenSSL/1.0.2o mod_bwlimited/1.4 mod_fcgid/2.3.9 configured -- resuming normal operations
[Wed May 30 13:13:48.157437 2018] [core:notice] [pid 30722:tid 140261823477888] AH00094: Command line: '/usr/sbin/httpd'

Output after:

Truncated, >100 child process errors.
[Wed May 30 13:18:48.822710 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3185 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.823154 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3452 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.833602 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3543 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.833863 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3845 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.834489 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4203 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.834952 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4420 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835105 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4629 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835363 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4841 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835598 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4923 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835768 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 5141 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.835954 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 8263 still did not exit, sending a SIGKILL
[Wed May 30 13:18:48.836204 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 8191 still did not exit, sending a SIGKILL

This is where I manually restarted Apache via WHM CP.
[Wed May 30 13:19:38.468224 2018] [ssl:warn] [pid 9786:tid 139695678261376] AH01909: proxy-subdomains-ssl-default-vhost.localhost:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 30 13:19:38.472819 2018] [ssl:warn] [pid 9786:tid 139695678261376] AH01909: X.com:443:0 server certificate does NOT include an ID which matches the server name
[Wed May 30 13:19:38.497125 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity for Apache/2.9.2 (ModSecurity: Open Source Web Application Firewall) configured.
[Wed May 30 13:19:38.497133 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: APR compiled version="1.6.3"; loaded version="1.6.3"
[Wed May 30 13:19:38.497136 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Wed May 30 13:19:38.497138 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: LUA compiled version="Lua 5.1"
[Wed May 30 13:19:38.497140 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: LIBXML compiled version="2.9.7"
[Wed May 30 13:19:38.497142 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.