Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Apache does not recover from SIGUSR1

Discussion in 'EasyApache' started by jsparks, May 30, 2018.

  1. jsparks

    jsparks Member

    Joined:
    Sep 16, 2015
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Halifax
    cPanel Access Level:
    Root Administrator
    Hi folks,

    I've been using php-fpm / apach2.4 / engintron (nginx reverse proxy) for quite some time now and noticed every few days apache dies and does not recover until I manually intervene and restart the process.

    The server has has no resource limitation issues that I am aware of, and from what I can tell, everything is configured as it should.

    I see in the apache error.log the following:

    [Wed May 30 13:13:45.018854 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00297: SIGUSR1 received. Doing graceful restart
    [Wed May 30 13:13:48.157406 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00292: Apache/2.4.33 (cPanel) OpenSSL/1.0.2o mod_bwlimited/1.4 mod_fcgid/2.3.9 configured -- resuming normal operations
    [Wed May 30 13:13:48.157437 2018] [core:notice] [pid 30722:tid 140261823477888] AH00094: Command line: '/usr/sbin/httpd'

    followed by many:

    [Wed May 30 13:18:42.782001 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5377 still did not exit, sending a SIGTERM
    [Wed May 30 13:18:42.782033 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 5561 still did not exit, sending a SIGTERM
    [Wed May 30 13:18:42.782042 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 8576 still did not exit, sending a SIGTERM
    [Wed May 30 13:18:42.782051 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 9776 still did not exit, sending a SIGTERM
    [Wed May 30 13:18:42.782059 2018] [core:warn] [pid 30722:tid 140261823477888] AH00045: child process 11882 still did not exit, sending a SIGTERM

    I checked the php-fpm logs in both /opt/cpanel/ea-php70/root/usr/var/log/php-fpm/ & /opt/cpanel/ea-php56/root/usr/var/log/php-fpm/ and seen no errors or warnings.. What else could be controlling this process?

    Thanks in advance.
     
    #1 jsparks, May 30, 2018
  2. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,059
    Likes Received:
    213
    Trophy Points:
    173
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @jsparks

    Does the issue persist when using only Apache? Can you run the following on the server and post the output:
    Code:
    cat /etc/redhat-release
    Code:
    cat /usr/local/cpanel/version
    Code:
    sar -q
    Code:
    sar -p
    Code:
    grep -i "Out of Memory" /var/log/messages
    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #2 cPanelLauren, May 30, 2018
  3. jsparks

    jsparks Member

    Joined:
    Sep 16, 2015
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Halifax
    cPanel Access Level:
    Root Administrator
    I never actually configured this server to run apache only, it's always been apache/php-fpm/nginx, but I can try re-configuring it if you think it's worth testing.

    Please see output:

    cat /etc/redhat-release
    CentOS Linux release 7.5.1804 (Core)

    cat /usr/local/cpanel/version
    11.70.0.46
    Code:
    sar -q
Linux 4.9.87-xxxx-std-ipv6-64 ()      05/30/2018      _x86_64_        (32 CPU)

12:00:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
12:10:01 AM        13      1249      5.79      4.77      4.89         0
12:20:01 AM        12      1235      3.99      4.62      4.79         1
12:30:01 AM        10      1240      3.85      4.41      4.60         1
12:40:01 AM        14      1222      3.94      4.23      4.48         1
12:50:01 AM         9      1258      4.85      5.03      4.74         1
01:00:01 AM        16      1266      5.06      4.79      4.65         0
01:10:01 AM         7      1237      4.62      4.49      4.53         0
01:20:01 AM        10      1216      3.70      3.61      4.04         1
01:30:01 AM        11      1248      4.50      4.79      4.50         1
01:40:01 AM        15      1176      8.32     23.20     15.08         1
01:50:01 AM        12      1073      3.34      6.15      9.57         0
02:00:01 AM        11      1082      3.25      3.73      6.74         2
02:10:01 AM        18      1080      6.62      5.69      6.22         3
02:20:01 AM        18      1053      7.10      6.92      6.65         1
02:30:01 AM        17      1073      5.99      5.81      6.05         1
02:40:01 AM        20      1094      5.92      6.17      6.13         5
02:50:01 AM        12      1140      6.06     15.61     12.17         6
03:00:01 AM         9      1033      5.05      5.92      8.40         1
03:10:01 AM        16      1053      6.12      5.41      6.81         1
03:20:01 AM        13      1070      3.35      4.50      5.81         1
03:30:01 AM        11      1064      7.99      6.18      5.89         3
03:40:01 AM        12      1035      6.25      5.85      5.86         1
03:50:01 AM        10      1059      6.23      6.41      6.07         1
04:00:01 AM        14      1043      5.43      5.42      5.68         2
04:10:01 AM        16      1046      4.73      5.57      5.76         2
04:20:01 AM        13      1025      5.93      5.55      5.59         2
04:30:01 AM        15      1034      4.28      4.41      4.94         5
04:40:01 AM         8      1028      6.00      5.35      5.05         2
04:50:01 AM        11      1179      5.90      5.70      5.27         1
05:00:01 AM        10      1174      5.05      4.45      4.66         2
05:10:01 AM        12      1174      4.26      4.93      5.13         0
05:20:01 AM        12      1181      3.86      4.49      4.91         1
05:30:01 AM         9      1182      4.71      5.30      5.23         2
05:40:01 AM        10      1190      4.29      5.02      5.12         1
05:50:01 AM        13      1020      3.73      4.02      4.59         3
06:00:01 AM        14      1060      5.41      5.76      5.22         3
06:10:01 AM        16      1021      3.62      4.36      4.73         2
06:20:01 AM        15      1048      6.63      5.45      5.15         2
06:30:01 AM        12      1051      5.24      5.37      5.17         2
06:40:01 AM         8      1067      3.59      3.93      4.51         3
06:50:01 AM        12      1044      4.97      4.47      4.47         4
07:00:01 AM        10      1043      5.57      5.68      5.13         2
07:10:01 AM         7      1020      4.36      4.80      5.03         1
07:20:01 AM        17      1024      5.27      5.22      5.18         2
07:30:01 AM        14      1060      6.16      6.80      6.25         2
07:40:01 AM         9      1041      5.12      6.04      6.16         3
07:50:01 AM        15      1019      4.54      4.78      5.42         3
08:00:01 AM        12      1047      5.46      5.33      5.45         5
08:10:01 AM        13      1047      5.76      5.51      5.47         1
08:20:01 AM         9      1036      4.70      5.63      5.61         2
08:30:01 AM        15      1027      3.80      4.52      5.14         2
08:40:01 AM        14      1216      7.50      5.40      5.11         1
08:50:01 AM        11      1223      6.28      6.47      5.98         1
09:00:01 AM        11      1232      6.45      6.08      5.98         1
09:10:01 AM        12      1345      6.02      6.68      6.48         2
09:20:01 AM         9      1222      5.41      5.60      6.10         2
09:30:01 AM        10      1040      6.80      6.57      6.42         2
09:40:01 AM        11      1053      5.96      6.06      6.27         2
09:50:01 AM        27      1087      6.54      6.59      6.49         1
10:00:01 AM        15      1093      7.38      8.12      7.55         3
10:10:01 AM        10      1066      8.34      8.17      7.82         3

10:10:01 AM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
10:20:01 AM        12      1127      5.99      7.29      7.51         2
10:30:01 AM         8      1124      7.55      6.32      6.71         3
10:40:01 AM        18      1094      5.70      6.03      6.38         1
10:50:01 AM        10      1102      5.06      5.59      6.01         2
11:00:01 AM        10      1074      6.46      5.73      5.79         3
11:10:01 AM        13      1087      4.14      5.02      5.63         1
11:20:01 AM        16      1070      5.39      5.60      5.69         2
11:30:01 AM        20      1071      5.85      5.98      5.91         2
11:40:01 AM        12      1367      5.65      6.23      6.14         0
11:50:01 AM        10      1688      4.56      5.28      5.79         2
12:00:01 PM        15      2019      4.42      4.54      5.17         3
12:10:01 PM        10      2434      4.22      4.70      4.97         2
12:20:01 PM        12      3001      4.70      5.10      5.07         0
12:30:01 PM        12      3376      3.97      4.19      4.57         1
12:40:01 PM        10      3777      2.62      3.87      4.39         0
12:50:01 PM         9      4176      4.63      4.87      4.71         0
01:00:01 PM         9      4465      4.29      3.99      4.28         0
01:10:01 PM         6      4342      1.24      1.68      2.90         0
01:20:01 PM        10      1207      2.04      1.56      2.21         0
01:30:01 PM         9      1518      3.30      3.57      3.09         0
01:40:01 PM        11      1855      4.10      3.76      3.45         0
01:50:01 PM        12      2185      5.17      4.45      3.90         0
02:00:01 PM        11      2501      4.52      3.84      3.79         1
02:10:01 PM        10      2749      3.70      3.66      3.78         0
02:20:01 PM        13      3170      3.85      4.03      3.91         0
02:30:01 PM        12      3537      3.57      4.04      4.09         0
02:40:01 PM        11      3868      4.22      4.54      4.43         0
02:50:01 PM        11      4292      5.13      5.13      4.74         0
03:00:01 PM        17      4551      5.51      5.34      5.06         0
03:10:01 PM        12      1469      5.67      5.07      4.93         0
03:20:01 PM         9      1736      6.38      5.99      5.41         4
03:30:01 PM        15      1629      5.32      7.29      6.82         1
03:40:01 PM        15      1204      5.87      6.63      6.86         2
03:50:01 PM        11      1182      7.18      6.59      6.60         1
04:00:01 PM        14      1181      5.85      5.72      6.12         4
04:10:01 PM        17      1178      6.03      6.22      6.22         1
04:20:01 PM        14      1030      6.80      6.29      6.20         1
04:30:01 PM        13      1025      5.28      5.44      5.79         1
04:40:01 PM        10      1042      4.95      5.28      5.59         2
04:50:01 PM        11      1018      8.34      6.94      6.13         1
05:00:01 PM        13      1022      5.78      6.25      6.29         1
05:10:01 PM        11      1031      3.67      4.50      5.43         1
05:20:01 PM        12      1019      6.53      5.97      5.63         1
05:30:01 PM        10      1245      6.89      5.57      5.37         2
05:40:01 PM        15      1218      5.93      6.28      5.83         1
05:50:01 PM         9      1202      3.18      4.11      5.04         0
06:00:01 PM        12      1154      4.33      3.60      4.19         1
06:10:01 PM        10      1148      2.15      3.07      3.75         0
06:20:01 PM        12      1151      2.81      3.00      3.37         0
06:30:01 PM        11      1135      2.90      2.89      3.20         0
06:40:01 PM         8      1140      2.99      2.79      3.00         0
06:50:01 PM         9      1145      3.21      3.18      3.05         0
07:00:01 PM         8      1145      2.94      2.66      2.74         0
07:10:01 PM         9      1140      1.65      2.41      2.69         0
07:20:01 PM        11      1146      2.60      2.58      2.59         0
07:30:01 PM         8      1137      2.85      2.68      2.66         0
07:40:01 PM        11      1148      2.17      2.74      2.75         0
07:50:01 PM        14      1143      2.27      2.49      2.60         0
08:00:01 PM        13      1149      3.46      3.07      2.79         0
08:10:01 PM        10      1135      2.18      2.63      2.72         0
08:20:01 PM        11      1136      2.77      2.71      2.72         1

08:20:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
08:30:01 PM        10      1140      1.91      2.27      2.58         2
08:40:01 PM         9      1138      2.29      2.42      2.56         0
08:50:01 PM        11      1144      3.05      3.22      2.91         1
09:00:01 PM        11      1146      5.32      3.94      3.35         1
Average:           12      1434      4.86      5.15      5.20         1

sar -p
Linux 4.9.87-xxxx-std-ipv6-64 ()      05/30/2018      _x86_64_        (32 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all     12.33      0.00      1.92      0.72      0.00     85.04
12:20:01 AM     all     12.46      0.00      1.95      0.61      0.00     84.99
12:30:01 AM     all     11.09      0.00      1.80      0.57      0.00     86.53
12:40:01 AM     all     12.23      0.00      1.89      0.57      0.00     85.31
12:50:01 AM     all     13.21      0.00      2.03      0.58      0.00     84.18
01:00:01 AM     all     11.91      0.00      1.90      0.69      0.00     85.49
01:10:01 AM     all     11.30      0.00      1.70      0.55      0.00     86.45
01:20:01 AM     all      9.37      0.01      1.63      0.63      0.00     88.36
01:30:01 AM     all     12.05      0.00      2.20      0.56      0.00     85.18
01:40:01 AM     all     12.03      0.00      4.70     15.69      0.00     67.58
01:50:01 AM     all      7.71      0.00      1.18      1.10      0.00     90.00
02:00:01 AM     all      9.17      0.00      1.63      0.79      0.00     88.41
02:10:01 AM     all      7.43      5.26      1.72      1.99      0.00     83.60
02:20:01 AM     all      7.79      8.05      2.24      2.64      0.00     79.28
02:30:01 AM     all      7.88      3.07      2.06      3.78      0.00     83.21
02:40:01 AM     all      8.28      2.71      1.95      4.68      0.00     82.37
02:50:01 AM     all     12.48      1.62      4.57     12.05      0.00     69.27
03:00:01 AM     all      7.01      1.11      1.25      4.19      0.00     86.45
03:10:01 AM     all      8.00      0.92      1.83      3.93      0.00     85.32
03:20:01 AM     all      6.47      0.98      1.54      4.74      0.00     86.27
03:30:01 AM     all      6.63      3.19      1.45      3.13      0.00     85.60
03:40:01 AM     all      6.64      6.45      1.81      2.95      0.00     82.15
03:50:01 AM     all      8.89      3.58      2.35      3.43      0.00     81.74
04:00:01 AM     all      6.55      3.99      1.85      3.34      0.00     84.26
04:10:01 AM     all      8.52      2.91      2.21      4.19      0.00     82.18
04:20:01 AM     all      7.58      1.99      1.90      4.10      0.00     84.43
04:30:01 AM     all      6.66      1.05      1.27      3.88      0.00     87.14
04:40:01 AM     all      7.06      1.36      1.73      3.90      0.00     85.96
04:50:01 AM     all      7.40      0.31      1.31      5.67      0.00     85.31
05:00:01 AM     all      7.59      0.03      1.13      3.59      0.00     87.66
05:10:01 AM     all     10.70      0.08      2.03      3.80      0.00     83.39
05:20:01 AM     all      8.30      0.52      1.57      3.76      0.00     85.84
05:30:01 AM     all      9.23      0.03      1.66      4.09      0.00     84.99
05:40:01 AM     all      7.11      0.28      1.21      4.75      0.00     86.65
05:50:01 AM     all      6.33      1.29      1.25      3.62      0.00     87.52
06:00:01 AM     all      8.13      1.75      1.88      3.71      0.00     84.54
06:10:01 AM     all      5.74      1.79      1.26      3.85      0.00     87.37
06:20:01 AM     all      8.26      1.84      1.55      3.90      0.00     84.44
06:30:01 AM     all      7.75      2.03      1.85      3.74      0.00     84.63
06:40:01 AM     all      5.68      1.41      1.18      3.62      0.00     88.11
06:50:01 AM     all      6.61      1.70      1.61      3.59      0.00     86.49
07:00:01 AM     all      7.19      1.97      1.69      3.66      0.00     85.49
07:10:01 AM     all      6.75      1.52      1.55      3.68      0.00     86.49
07:20:01 AM     all      7.44      1.64      1.77      3.80      0.00     85.35
07:30:01 AM     all     11.35      1.52      3.03      4.79      0.00     79.30
07:40:01 AM     all      9.46      1.78      1.71      4.05      0.00     83.01
07:50:01 AM     all      6.87      1.46      1.34      3.99      0.00     86.33
08:00:01 AM     all      9.21      1.48      1.59      3.85      0.00     83.88
08:10:01 AM     all      9.35      1.74      1.70      4.11      0.00     83.10
08:20:01 AM     all      9.82      1.01      1.87      3.96      0.00     83.33
08:30:01 AM     all      8.04      0.01      1.26      4.35      0.00     86.34
08:40:01 AM     all      8.16      0.05      1.79      3.80      0.00     86.20
08:50:01 AM     all      9.62      3.14      2.32      3.77      0.00     81.15
09:00:01 AM     all      8.41      2.20      2.05      4.37      0.00     82.97
09:10:01 AM     all     11.35      2.39      2.65      4.67      0.00     78.95
09:20:01 AM     all      8.66      3.42      2.36      3.88      0.00     81.69
09:30:01 AM     all      9.49      4.10      2.58      3.86      0.00     79.97
09:40:01 AM     all      8.09      3.70      1.97      4.03      0.00     82.21
09:50:01 AM     all      9.96      4.46      1.98      3.37      0.00     80.24
10:00:01 AM     all     13.09      6.68      2.98      3.21      0.00     74.03
10:10:01 AM     all     11.08      5.78      2.45      3.74      0.00     76.95

10:10:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:20:01 AM     all     10.55      4.94      2.89      3.07      0.00     78.55
10:30:01 AM     all      8.08      3.31      2.01      3.16      0.00     83.45
10:40:01 AM     all      8.71      3.15      2.13      4.05      0.00     81.95
10:50:01 AM     all      8.89      1.40      1.94      3.90      0.00     83.87
11:00:01 AM     all      8.70      3.29      1.74      3.54      0.00     82.73
11:10:01 AM     all      8.96      2.89      1.74      3.44      0.00     82.97
11:20:01 AM     all     10.62      0.88      1.93      3.13      0.00     83.44
11:30:01 AM     all     10.88      2.19      2.40      2.33      0.00     82.21
11:40:01 AM     all     13.85      0.81      2.34      1.31      0.00     81.70
11:50:01 AM     all     13.16      0.15      2.06      0.81      0.00     83.83
12:00:01 PM     all     11.92      0.00      1.58      0.58      0.00     85.92
12:10:01 PM     all     11.76      0.00      1.68      0.61      0.00     85.96
12:20:01 PM     all     13.54      0.00      2.09      0.50      0.00     83.87
12:30:01 PM     all      8.27      0.00      2.63      0.45      0.00     88.64
12:40:01 PM     all      9.44      0.00      2.93      0.42      0.00     87.21
12:50:01 PM     all     10.99      0.00      3.31      0.45      0.00     85.25
01:00:01 PM     all      8.26      0.00      2.94      0.53      0.00     88.27
01:10:01 PM     all      2.49      0.00      2.20      0.16      0.00     95.15
01:20:01 PM     all      2.01      0.01      2.17      0.15      0.00     95.67
01:30:01 PM     all      8.97      0.00      2.76      0.69      0.00     87.58
01:40:01 PM     all      9.18      0.00      2.56      0.49      0.00     87.77
01:50:01 PM     all      9.83      0.00      2.67      0.54      0.00     86.97
02:00:01 PM     all      8.29      0.00      2.53      0.57      0.00     88.61
02:10:01 PM     all      9.48      0.00      1.34      0.55      0.00     88.63
02:20:01 PM     all      9.36      0.00      1.41      0.48      0.00     88.74
02:30:01 PM     all     11.04      0.00      1.61      0.43      0.00     86.92
02:40:01 PM     all     12.46      0.00      1.83      0.48      0.00     85.22
02:50:01 PM     all     10.51      0.00      3.81      0.59      0.00     85.08
03:00:01 PM     all     11.98      0.00      4.10      0.54      0.00     83.38
03:10:01 PM     all     10.25      0.00      3.99      0.69      0.00     85.07
03:20:01 PM     all     12.44      0.01      4.39      0.82      0.00     82.35
03:30:01 PM     all     10.62      1.95      3.91      4.67      0.00     78.85
03:40:01 PM     all     10.53      2.46      3.28      4.01      0.00     79.72
03:50:01 PM     all     10.43      0.93      2.37      4.61      0.00     81.66
04:00:01 PM     all      8.33      0.88      2.10      4.14      0.00     84.55
04:10:01 PM     all     10.75      2.28      1.98      3.38      0.00     81.61
04:20:01 PM     all      9.59      2.63      1.86      3.57      0.00     82.35
04:30:01 PM     all      8.44      1.75      1.65      3.82      0.00     84.35
04:40:01 PM     all      9.38      1.38      1.74      3.68      0.00     83.83
04:50:01 PM     all      9.57      3.48      2.02      3.54      0.00     81.38
05:00:01 PM     all      8.95      5.17      1.96      3.33      0.00     80.58
05:10:01 PM     all      7.99      0.97      1.49      3.50      0.00     86.06
05:20:01 PM     all      7.47      4.44      2.24      2.94      0.00     82.91
05:30:01 PM     all      8.25      1.19      2.09      4.00      0.00     84.47
05:40:01 PM     all      9.35      2.30      2.20      3.97      0.00     82.18
05:50:01 PM     all      7.91      0.93      1.84      1.74      0.00     87.57
06:00:01 PM     all      7.91      0.00      1.41      0.42      0.00     90.27
06:10:01 PM     all      8.54      0.00      1.53      0.38      0.00     89.55
06:20:01 PM     all      6.27      0.00      1.21      0.38      0.00     92.15
06:30:01 PM     all      8.09      0.00      1.34      0.36      0.00     90.22
06:40:01 PM     all      7.53      0.00      1.32      0.35      0.00     90.79
06:50:01 PM     all      7.31      0.00      1.31      0.35      0.00     91.04
07:00:01 PM     all      6.62      0.00      1.23      0.34      0.00     91.81
07:10:01 PM     all      8.24      0.01      1.39      0.42      0.00     89.94
07:20:01 PM     all      7.17      0.00      1.29      0.26      0.00     91.28
07:30:01 PM     all      6.86      0.00      1.46      0.29      0.00     91.39
07:40:01 PM     all      7.33      0.00      1.25      0.37      0.00     91.05
07:50:01 PM     all      5.70      0.00      1.23      0.31      0.00     92.75
08:00:01 PM     all      6.67      0.00      1.36      0.26      0.00     91.71
08:10:01 PM     all      7.51      0.00      1.35      0.26      0.00     90.87
08:20:01 PM     all      6.51      0.00      1.27      0.28      0.00     91.93

08:20:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
08:30:01 PM     all      6.74      0.00      1.27      0.30      0.00     91.69
08:40:01 PM     all      6.72      0.00      1.22      0.27      0.00     91.79
08:50:01 PM     all      8.97      0.00      1.49      0.30      0.00     89.24
09:00:01 PM     all      9.99      0.00      1.70      0.27      0.00     88.04
Average:        all      8.92      1.33      1.99      2.55      0.00     85.22

grep -i "Out of Memory" /var/log/messages
May 30 01:34:26 kernel: Out of memory: Kill process 5314 (mysqld) score 179 or sacrifice child
May 30 01:35:14 kernel: Out of memory: Kill process 11083 (mysqld) score 38 or sacrifice child
May 30 01:35:37 kernel: Out of memory: Kill process 11690 (mysqld) score 19 or sacrifice child
May 30 01:36:32 kernel: Out of memory: Kill process 5941 (named) score 8 or sacrifice child
May 30 01:36:39 kernel: Out of memory: Kill process 11878 (mysqld) score 11 or sacrifice child
May 30 02:45:12 kernel: Out of memory: Kill process 12259 (mysqld) score 46 or sacrifice child
May 30 02:45:40 kernel: Out of memory: Kill process 13100 (named) score 6 or sacrifice child
May 30 02:45:56 kernel: Out of memory: Kill process 29233 (mysqld) score 9 or sacrifice child
    I'm tracing the OOM issue but don't think it's related as the timestamps don't match.
     
    #3 jsparks, May 30, 2018
    Last edited by a moderator: May 31, 2018
  4. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,059
    Likes Received:
    213
    Trophy Points:
    173
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @jsparks

    You are correct that the timestamps don't match and it's MySQL being killed so they may not be directly related but was I am gathering from this output is you do have what appears to be consistently medium/high load on the server as well as some IO wait spikes. Are you on a VPS or standard environment?

    I'd also like to know what's in the logs between/before/after these warnings:

    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #4 cPanelLauren, May 31, 2018
  5. jsparks

    jsparks Member

    Joined:
    Sep 16, 2015
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Halifax
    cPanel Access Level:
    Root Administrator
    Hi Lauren,

    This is a dedicated machine which is running very low load (typically). What you are seeing in the sar output is likely a result of the cpbackup that started May 30 early in the AM, which I believe is what caused SQL to run OOM. I don't think this issue that I'm seeing with Apache is directly correlated with cpbackup, as it did happen a few times outside of backup windows.

    Output before warning:

    [y 30 13:13:45.018854 2018Wed May 30 13:11:18.422118 2018] [proxy_fcgi:error] [pid 4420:tid 140261573338880] (70007)The timeout specified has expired: [client X:59826] AH01075: Error dispatching request to : (polling), referer: Google
    [Wed Ma 30 13:13:45.018854 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00297: SIGUSR1 received. Doing graceful restart
    [Wed May 30 13:13:48.114379 2018] [ssl:warn] [pid 30722:tid 140261823477888] AH01909: proxy-subdomains-ssl-default-vhost.localhost:443:0 server certificate does NOT include an ID which matches the server name
    [Wed May 30 13:13:48.118733 2018] [ssl:warn] [pid 30722:tid 140261823477888] AH01909: X:443:0 server certificate does NOT include an ID which matches the server name
    [Wed May 30 13:13:48.157406 2018] [mpm_worker:notice] [pid 30722:tid 140261823477888] AH00292: Apache/2.4.33 (cPanel) OpenSSL/1.0.2o mod_bwlimited/1.4 mod_fcgid/2.3.9 configured -- resuming normal operations
    [Wed May 30 13:13:48.157437 2018] [core:notice] [pid 30722:tid 140261823477888] AH00094: Command line: '/usr/sbin/httpd'

    Output after:

    Truncated, >100 child process errors.
    [Wed May 30 13:18:48.822710 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3185 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.823154 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3452 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.833602 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3543 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.833863 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 3845 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.834489 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4203 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.834952 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4420 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.835105 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4629 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.835363 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4841 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.835598 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 4923 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.835768 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 5141 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.835954 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 8263 still did not exit, sending a SIGKILL
    [Wed May 30 13:18:48.836204 2018] [core:error] [pid 30722:tid 140261823477888] AH00046: child process 8191 still did not exit, sending a SIGKILL

    This is where I manually restarted Apache via WHM CP.
    [Wed May 30 13:19:38.468224 2018] [ssl:warn] [pid 9786:tid 139695678261376] AH01909: proxy-subdomains-ssl-default-vhost.localhost:443:0 server certificate does NOT include an ID which matches the server name
    [Wed May 30 13:19:38.472819 2018] [ssl:warn] [pid 9786:tid 139695678261376] AH01909: X.com:443:0 server certificate does NOT include an ID which matches the server name
    [Wed May 30 13:19:38.497125 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity for Apache/2.9.2 (ModSecurity: Open Source Web Application Firewall) configured.
    [Wed May 30 13:19:38.497133 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: APR compiled version="1.6.3"; loaded version="1.6.3"
    [Wed May 30 13:19:38.497136 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
    [Wed May 30 13:19:38.497138 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: LUA compiled version="Lua 5.1"
    [Wed May 30 13:19:38.497140 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: LIBXML compiled version="2.9.7"
    [Wed May 30 13:19:38.497142 2018] [:notice] [pid 9786:tid 139695678261376] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
     
    #5 jsparks, May 31, 2018
  6. cPanelLauren

    cPanelLauren Forums Analyst II
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    3,059
    Likes Received:
    213
    Trophy Points:
    173
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @jsparks

    Could you please open a ticket using the link in my signature? I feel like it might be better to troubleshoot this issue with access to the system. Once the ticket is open please reply to this thread with the ticket ID so we can update the thread with the outcome.

    Thanks!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #6 cPanelLauren, May 31, 2018
(You must log in or sign up to post here.)
Loading...
Similar Threads - Apache does recover
  1. atkhosting

    Raw Apache Log Download Does not include SSL log

    atkhosting, Oct 16, 2017, in forum: General Discussion
    Replies:
    3
    Views:
    523
    cPWilliamL
    Oct 16, 2017
  2. gu3miles

    SOLVED EA4 Migration Fails, So Does Apache 2.4 Build

    gu3miles, Aug 28, 2017, in forum: EasyApache
    Replies:
    2
    Views:
    521
    gu3miles
    Aug 28, 2017
  3. cPStacy

    EasyApache 4 Update - October 10, 2018

    cPStacy, Oct 10, 2018 at 3:08 PM, in forum: EasyApache
    Replies:
    0
    Views:
    84
    cPStacy
    Oct 10, 2018 at 3:08 PM
  4. cPStacy

    EasyApache 3.36.9 Released

    cPStacy, Oct 10, 2018 at 2:44 PM, in forum: EasyApache
    Replies:
    0
    Views:
    31
    cPStacy
    Oct 10, 2018 at 2:44 PM
  5. gosko

    Apache PHP-FPM 56 service is down

    gosko, Oct 4, 2018, in forum: EasyApache
    Replies:
    1
    Views:
    82
    cPanelLauren
    Oct 8, 2018

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Accept Learn More...
    Dismiss Notice