Apache crashes when the server load reaches 6%.

C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Hello,

I run a website with a large number of posts (about 500,000) and whenever Google bots come to index the posts, the server load will exceed 6% and apache will crash. Surely, apache will self- start and a few minutes later will crash again.

I run a VPS with 5 processors and 5GB of RAM along with PHP-FPM 7.1 and MySQL. Memcached server and client are also installed. I am not aware if there any other PHP cache module is installed.

Below is the output of the top command:

Code: 
top - 12:07:16 up 2 days, 15:12,  1 user,  load average: 6.63, 6.39, 5.91
Tasks: 199 total,   2 running, 196 sleeping,   0 stopped,   1 zombie
%Cpu(s): 48.5 us, 47.2 sy,  0.0 ni,  0.6 id,  0.0 wa,  0.0 hi,  0.3 si,  3.3 st
KiB Mem :  4931796 total,   120748 free,  2632020 used,  2179028 buff/cache
KiB Swap:  4980732 total,  4715004 free,   265728 used.  1705944 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  1181 mysql     20   0 2544528 242456   4980 S 480.1  4.9   4487:20 mysqld
100796 adb98765  20   0  546476  44700  16540 S  10.9  0.9   0:02.74 php-fpm
  1356 memcach+  20   0  566320  65232    944 S   4.0  1.3  48:01.76 memcached
 53625 root      20   0  183584  27560   2156 S   0.3  0.6   2:17.95 lfd - sleeping
     1 root      20   0  193908   5216   3000 S   0.0  0.1   4:27.34 systemd
     2 root      20   0       0      0      0 S   0.0  0.0   0:00.32 kthreadd
     3 root      20   0       0      0      0 S   0.0  0.0   0:00.55 ksoftirqd/0
     5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
     7 root      rt   0       0      0      0 S   0.0  0.0   0:00.47 migration/0
     8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
     9 root      20   0       0      0      0 R   0.0  0.0   2:33.64 rcu_sched
    10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
    11 root      rt   0       0      0      0 S   0.0  0.0   0:01.85 watchdog/0
    12 root      rt   0       0      0      0 S   0.0  0.0   0:01.72 watchdog/1
    13 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 migration/1
    14 root      20   0       0      0      0 S   0.0  0.0   0:00.59 ksoftirqd/1
    16 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
    17 root      rt   0       0      0      0 S   0.0  0.0   0:01.67 watchdog/2
    18 root      rt   0       0      0      0 S   0.0  0.0   0:00.42 migration/2
    19 root      20   0       0      0      0 S   0.0  0.0   0:00.54 ksoftirqd/2
    21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H
    22 root      rt   0       0      0      0 S   0.0  0.0   0:01.70 watchdog/3
    23 root      rt   0       0      0      0 S   0.0  0.0   0:00.54 migration/3
    24 root      20   0       0      0      0 S   0.0  0.0   0:00.54 ksoftirqd/3
    26 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/3:0H
    27 root      rt   0       0      0      0 S   0.0  0.0   0:01.72 watchdog/4
    28 root      rt   0       0      0      0 S   0.0  0.0   0:00.44 migration/4
    29 root      20   0       0      0      0 S   0.0  0.0   0:01.03 ksoftirqd/4
Below is the apache status log:

Code: 
Server Version: Apache/2.4.33 (cPanel) OpenSSL/1.0.2o mod_bwlimited/1.4
Server MPM: event
Server Built: Apr 23 2018 21:10:29

Current Time: Wednesday, 23-May-2018 12:25:25 AEST
Restart Time: Wednesday, 23-May-2018 05:58:31 AEST
Parent Server Config. Generation: 9
Parent Server MPM Generation: 8
Server uptime: 6 hours 26 minutes 53 seconds
Server load: 5.92 6.16 5.99
Total accesses: 8622 - Total Traffic: 164.9 MB
CPU Usage: u171.78 s16.27 cu0 cs0 - .81% CPU load
.371 requests/sec - 7.3 kB/second - 19.6 kB/request
63 requests currently being processed, 87 idle workers

Slot    PID    Stopping    Connections     Threads    Async connections
total    accepting    busy    idle    writing    keep-alive    closing
0    90890    yes (old gen)    22    no    0    0    0    0    0
1    90891    yes (old gen)    23    no    0    0    0    0    0
2    90892    yes (old gen)    19    no    0    0    0    0    0
3    90893    yes (old gen)    23    no    0    0    0    0    0
4    90894    yes (old gen)    27    no    0    0    0    0    0
5    91260    yes (old gen)    25    no    0    0    0    0    0
6    92447    yes (old gen)    25    no    0    0    0    0    0
7    92448    yes (old gen)    25    no    0    0    0    0    0
8    92449    yes (old gen)    25    no    0    0    0    0    0
9    92450    yes (old gen)    24    no    0    0    0    0    0
10    92451    yes (old gen)    24    no    0    0    0    0    0
11    92605    yes (old gen)    24    no    0    0    0    0    0
12    96656    yes (old gen)    26    no    0    0    0    0    0
13    96657    yes (old gen)    25    no    0    0    0    0    0
14    96658    yes (old gen)    24    no    0    0    0    0    0
15    96659    yes (old gen)    24    no    0    0    0    0    0
16    96660    yes (old gen)    25    no    0    0    0    0    0
17    96820    yes (old gen)    25    no    0    0    0    0    0
18    84498    yes (old gen)    24    no    0    0    0    0    0
19    101429    no    11    yes    10    15    0    1    0
20    84500    yes (old gen)    24    no    0    0    0    0    0
21    84501    yes (old gen)    24    no    0    0    0    0    0
22    84502    yes (old gen)    25    no    0    0    0    0    0
23    84653    yes (old gen)    25    no    0    0    0    0    0
24    101430    no    15    yes    14    11    0    0    0
25    101431    no    15    yes    14    11    0    0    0
26    101432    no    10    yes    10    15    0    0    0
27    101433    no    12    yes    11    14    0    0    1
28    101584    no    5    yes    4    21    0    0    0
Sum    29    23    625         63    87    0    1    1

GGGGGGGG.GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGG.GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG
GW___W_W____WW_W_W_WW_W___GGGGGGGGWW___WWW_WWWW__WWW_W___W_W_WW_
_W__WW_WW__WWWW_W__W_WW_____WW_WW_WW__W_W____W__W_WW_WW_W_____W_
W_WW__WW_____________W____W____

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
I would like to receive some feedback from the community in order to optimise apache, mysql and php-fpm.

Many thanks.
 
C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Hello folks,

Just learnt that the apache crashing issue may be related to "broken pipe, error writing to error". Please see the log below:

Code: 
[Wed May 23 12:15:06.795489 2018] [log_config:warn] [pid 84499:tid 139816469452544] (32)Broken pipe: [client 103.1.205.107:37260] AH00646: Error writing to |/usr/local/cpanel/bin/splitlogs --main
if this is the cause then how do I fix this issue?

Thanks for help.
 
C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Hello folks. This is an update on the issue that apache is crashing every 10 to 20 minutes when the server load reaches about 6%. Have been searching the net to resolve the error about "broken pipe" issue and have seen some recommendations to add the directives "EnableMMAP off" and "EnableSendfile off" in the top section of httpd.conf file, which I did but the issue remains.

Any suggestions?

Thanks.
 
cPanelLauren

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
Hi @Clixer

That error indicates that there's something going on when writing to the splitlogs. since /etc/apache2/logs is symlinked to /var/log/apache2/logs can you tell me if the symlink is present:

Code: 
stat /etc/apache2/logs
You might also want to review the documentation here: The splitlogs Binary - Version 68 Documentation - cPanel Documentation

Can you also check the apache error logs for the following for me:
Code: 
grep Max /etc/apache2/log/error_log
 
C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Hi @cPanelLauren. The outputs are below:

Code: 
[[email protected] ~]# stat /etc/apache2/logs
File: ‘/etc/apache2/logs’ -> ‘../../var/log/apache2’
Size: 21              Blocks: 0          IO Block: 4096   symbolic link
Device: fd00h/64768d    Inode: 4124129     Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:httpd_config_t:s0
Access: 2018-05-24 05:57:58.879356210 +1000
Modify: 2018-04-25 05:57:56.668379769 +1000
Change: 2018-04-25 05:57:56.668379769 +1000
Birth: -
And for the error log:

Code: 
[[email protected] ~]# grep Max /etc/apache2/log/error_log
grep: /etc/apache2/log/error_log: No such file or directory
I am not sure why there is no error log. I am not sure what I am doing so I am relying upon your help.

There is no directory called log in the /etc/apache2/ folder:

Code: 
[[email protected] ~]# cd /etc/apache2/
[[email protected] apache2]# ls -l
total 8
drwxr-xr-x. 2 root root   36 Apr 24 07:12 bin
drwxr-xr-x. 2 root root  103 May 24 06:03 conf
drwxr-xr-x. 7 root root  269 May 23 05:58 conf.d
drwxr-xr-x. 2 root root 4096 May 23 05:58 conf.modules.d
lrwxrwxrwx. 1 root root   21 Apr 25 05:57 logs -> ../../var/log/apache2
lrwxrwxrwx. 1 root root   31 Apr 25 05:57 modules -> ../../usr/lib64/apache2/modules
lrwxrwxrwx. 1 root root   21 Apr 25 05:57 run -> ../../var/run/apache2
Here is the output for grep Max /etc/apache2/logs/error_log:

Code: 
[Tue May 22 21:15:45.645180 2018] [pagespeed:error] [pid 118175:tid 140090198083328] [mod_pagespeed 1.11.33.4-0 @118175] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.tempYhGbbd:0: opening temp file: No such file or directory
[Tue May 22 21:15:45.645492 2018] [pagespeed:error] [pid 118175:tid 140090223261440] [mod_pagespeed 1.11.33.4-0 @118175] Could not create directories for file /var/mod_pagespeed/cache/rname/ji_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temp
[Tue May 22 21:15:45.645519 2018] [pagespeed:error] [pid 118175:tid 140090223261440] [mod_pagespeed 1.11.33.4-0 @118175] /var/mod_pagespeed/cache/rname/ji_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temphVdbfb:0: opening temp file: No such file or directory
[Tue May 22 21:15:45.710382 2018] [pagespeed:error] [pid 118175:tid 140090198083328] [mod_pagespeed 1.11.33.4-0 @118175] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temp
[Tue May 22 21:15:45.710413 2018] [pagespeed:error] [pid 118175:tid 140090198083328] [mod_pagespeed 1.11.33.4-0 @118175] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.tempA54InC:0: opening temp file: No such file or directory
[Tue May 22 21:15:45.710731 2018] [pagespeed:error] [pid 118175:tid 140090223261440] [mod_pagespeed 1.11.33.4-0 @118175] Could not create directories for file /var/mod_pagespeed/cache/rname/ce_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temp
[Tue May 22 21:15:45.710762 2018] [pagespeed:error] [pid 118175:tid 140090223261440] [mod_pagespeed 1.11.33.4-0 @118175] /var/mod_pagespeed/cache/rname/ce_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.tempdQOsDA:0: opening temp file: No such file or directory
[Wed May 23 00:03:04.749731 2018] [mpm_event:error] [pid 1621:tid 140090795137152] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Wed May 23 06:26:55.349947 2018] [mpm_event:error] [pid 75732:tid 139816991602816] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Wed May 23 13:03:48.254622 2018] [pagespeed:error] [pid 104012:tid 139816519808768] [mod_pagespeed 1.11.33.4-0 @104012] /var/mod_pagespeed/cache/prop_page/https,3A/,2Fwww.example.com/_iSyw0W7OG6,40Desktop,40beacon_cohort,.tempFMaxch:0: opening temp file: No such file or directory
[Wed May 23 20:01:22.366287 2018] [mpm_event:error] [pid 1603:tid 140137861802112] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Wed May 23 21:36:19.277172 2018] [pagespeed:error] [pid 10604:tid 140123828049664] [mod_pagespeed 1.11.33.4-0 @10604] /var/mod_pagespeed/cache/prop_page/http,3A/,2Fwww.example.com/jobs/part-time/manningham/2018/05/04/occupational-therapist-grade-2-adelaide-sa/_iSyw0W7OG6,40Desktop,40beacon_cohort,.templMaxyR:0: opening temp file: No such file or directory
[Wed May 23 22:12:07.132263 2018] [mpm_event:error] [pid 10595:tid 140124362528896] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Wed May 23 22:31:53.013149 2018] [mpm_event:error] [pid 14399:tid 140002601973888] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
 
Last edited by a moderator:
C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Hi @cPanelLauren,

I increased the value of Max Request Workers from 150 to 250, this helped in removing the error from apache log. The remaining error is about pagespeed not able to create directory. I am still looking into this. The server load is still too high, ranges from 3% to 8%.

Below is the output of the apache log:

Code: 
[[email protected] ~]# grep Max /etc/apache2/logs/error_log
[pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,.temp
[Thu May 24 15:01:55.543972 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,.temp7ZBhA3:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.549991 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/rname/cf_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,40A.,40.A_,.temp
[Thu May 24 15:01:55.550029 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/rname/cf_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,40A.,40.A_,.tempa3gVio:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.614948 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temp
[Thu May 24 15:01:55.614979 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temp9uxmWp:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.694630 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,.temp
[Thu May 24 15:01:55.694667 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,.tempEjSAOQ:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.699071 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/rname/ci_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,40,40_,.temp
[Thu May 24 15:01:55.699109 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/rname/ci_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,40,40_,.tempDgx4Xb:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.743664 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temp
[Thu May 24 15:01:55.743699 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.tempBrCJXY:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.746040 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/rname/jm_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temp
[Thu May 24 15:01:55.746074 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/rname/jm_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temp2ZyFfk:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.793846 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,.temp
[Thu May 24 15:01:55.793900 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,.tempPdxXi7:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.799873 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/rname/ce_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,40,40_,.temp
[Thu May 24 15:01:55.799905 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/rname/ce_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,40,40_,.tempRiLAKs:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.802091 2018] [pagespeed:error] [pid 79332:tid 139670871766784] [mod_pagespeed 1.11.33.4-0 @79332] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,.temp
[Thu May 24 15:01:55.802133 2018] [pagespeed:error] [pid 79332:tid 139670871766784] [mod_pagespeed 1.11.33.4-0 @79332] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.examle.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.css,3Fver=4.9.6,.tempClx6Es:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.861964 2018] [pagespeed:error] [pid 79491:tid 139670767781632] [mod_pagespeed 1.11.33.4-0 @79491] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temp
[Thu May 24 15:01:55.861992 2018] [pagespeed:error] [pid 79491:tid 139670767781632] [mod_pagespeed 1.11.33.4-0 @79491] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temptLTX5r:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.871681 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temp
[Thu May 24 15:01:55.871710 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temp4KJlwk:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.873086 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/rname/ji_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temp
[Thu May 24 15:01:55.873113 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/rname/ji_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temp4nIVP1:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.913908 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temp
[Thu May 24 15:01:55.913935 2018] [pagespeed:error] [pid 78352:tid 139670566356736] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/v3/example.com/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,.temprpyLLS:0: opening temp file: No such file or directory
[Thu May 24 15:01:55.914245 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] Could not create directories for file /var/mod_pagespeed/cache/rname/ce_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temp
[Thu May 24 15:01:55.914296 2018] [pagespeed:error] [pid 78352:tid 139670583142144] [mod_pagespeed 1.11.33.4-0 @78352] /var/mod_pagespeed/cache/rname/ce_iSyw0W7OG6bbUTqWEw0u/https,3A/,2Fwww.example.com/wp-content/plugins/w3-total-cache/Cdn_Plugin_WidgetMaxCdn_View.js,3Fver=1.0,40,40_,.temp5ZTYxe:0: opening temp file: No such file or directory
Any ideas how I could solve the pagespeed issue? Thanks for help.
 
Last edited by a moderator:
C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Hi @cPanelLauren,

The pagespeed issue has now disappeared. I have provided the folder ownership to httpd at /var/mod_pagespeed/cache/ so httpd could write to this folder.

The Max Request Worker issue is back, currently set to 250. I will keep working on it but appreciate some advise from you. Thanks.
 
C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Hi @cPanelLauren,

I have been trying to optimise apache through "Home » Service Configuration » Apache Configuration » Global Configuration" but later learnt that this is for MPM Prefork only. How do I optimise apache for mpm-event?

Thanks for help.
 
cPanelLauren

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
Hi @Clixer

Clixer said:
I have been trying to optimise apache through "Home » Service Configuration » Apache Configuration » Global Configuration" but later learnt that this is for MPM Prefork only. How do I optimise apache for mpm-event?
Click to expand...
Actually, that's incorrect and I'm opening a case with our documentation team on this. It looks like the note has been added to too many directives. Modifying these is definitely ok with prefork, event and worker.


Thanks!
 
C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Thanks @cPanelLauren.

The server is generating heaps of errors about "scoreboard is full, not at MaxRequestWorkers. Increase ServerLimit", these errors are ongoing and the apahce is crashing. Output is below:

Code: 
[Fri May 25 09:29:01.913814 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:02.920947 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:03.929326 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:04.935802 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:05.940789 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:06.948809 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:07.954094 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:08.960791 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:09.967799 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:10.970795 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:11.976205 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:12.978801 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:13.998792 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:14.999933 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:16.001609 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
[Fri May 25 09:29:17.002797 2018] [mpm_event:error] [pid 109589:tid 140179259779200] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
The following is the current httpd configuration:

Code: 
StartServers 3
<IfModule prefork.c>
MinSpareServers 3
MaxSpareServers 16
</IfModule>
ServerLimit 16
MaxRequestWorkers 400
MaxConnectionsPerChild 10000
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 100
Timeout 300
what would I need to do to increase the server limit? Is there a formula I need to apply in calculating the new severlimit? And Can I make the changes directly to the httpd.conf file.
 
C

Clixer

Active Member
Feb 24, 2018
41
2
8
Australia
cPanel Access Level
Root Administrator
Hi @cPanelLauren,

In order to completely get rid of the MaxRequestWorkers hitting the maximum allowed, I had to increase its value as illustrated below:

Code: 
StartServers 3
<IfModule prefork.c>
MinSpareServers 3
MaxSpareServers 64
</IfModule>
ServerLimit 64
MaxRequestWorkers 1600
MaxConnectionsPerChild 10000
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 200
Timeout 300
I could not do this change directly on WHM so I did it manually and modified the httpd.conf file. I then started httpd with the command "systemctl restart httpd". The apache has been running fine since then and hasnt crashed yet. What I have noted is that in WHM, the old apache configuration is still present. Is this a bug in WHM or I should not have manually modified the httpd.conf file.

The second question is, is the configuration illustrated makes sense? Whilst apache is no longer crashing, the sever load ranges between 6% to 7%. Most of the resource are taken up by sqld. It is illustrated in the output below.

Code: 
top - 16:15:20 up 1 day, 20:34,  1 user,  load average: 7.55, 6.82, 6.30
Tasks: 208 total,   4 running, 204 sleeping,   0 stopped,   0 zombie
%Cpu(s): 61.3 us, 34.8 sy,  0.0 ni,  0.3 id,  0.1 wa,  0.0 hi,  0.3 si,  3.4 st
KiB Mem :  4931780 total,   110448 free,  2565548 used,  2255784 buff/cache
KiB Swap:  4980732 total,  4690940 free,   289792 used.  1809384 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 14932 mysql     20   0 2478408 299788   5212 S 362.4  6.1   9372:09 mysqld
 52854 dotcap0+  20   0  595704  84020   6036 D  43.2  1.7   0:03.01 php-fpm
 42049 nobody    20   0 2508020  84828  16892 S  42.2  1.7   0:04.74 httpd
 52864 dotcap0+  20   0  517164  78244   4284 S  29.0  1.6   0:04.27 php-fpm
 52284 adb98765  20   0  483740  56544  16008 R   6.6  1.1   0:20.23 php-fpm
 50943 nobody    20   0 2509044  72852  13300 S   4.0  1.5   0:01.58 httpd
 51793 nobody    20   0 2508016  51536   9516 S   3.6  1.0   0:00.72 httpd
  1357 memcach+  20   0  567368  62484    848 S   2.6  1.3 110:58.19 memcached
 13133 root      20   0  182904  30192   2016 S   1.0  0.6   1:38.91 lfd - sleeping
 51205 nobody    20   0 2509044  81148  13708 S   1.0  1.6   0:04.06 httpd
 48204 nobody    20   0 2509044  62520  12632 S   0.7  1.3   0:02.50 httpd
     9 root      20   0       0      0      0 R   0.3  0.0   2:21.66 rcu_sched
   698 root      20   0   21644    948    784 S   0.3  0.0   0:39.24 irqbalance
 48855 nobody    20   0 2508020  50708  11716 S   0.3  1.0   0:01.48 httpd
 52468 nobody    20   0 2041068  39160   7328 S   0.3  0.8   0:01.19 httpd
 52794 root      20   0  160064   2308   1548 R   0.3  0.0   0:00.52 top
     1 root      20   0  193924   5148   2904 S   0.0  0.1   4:00.62 systemd
     2 root      20   0       0      0      0 S   0.0  0.0   0:00.28 kthreadd
     3 root      20   0       0      0
Any recommendations as to how I should optimise mysql. The output of my.cnf is below:

Code: 
[mysqld]
default-storage-engine=innodb
performance-schema=0
innodb_file_per_table=1
innodb_buffer_pool_instances=1
innodb_buffer_pool_size=134217728
innodb_log_file_size=512M
max_connections=300
max_allowed_packet=268435456
open_files_limit=10000
query_cache_type=1
query_cache_limit=256K
query_cache_min_res_unit=2k
query_cache_size=80M
tmp_table_size=64M
max_heap_table_size=64M
slow-query-log=1
slow-query-log-file=/var/lib/mysql/mysql-slow.log
long_query_time=1
local-infile=0
# MySQL idle Connections
wait_timeout=60
# Skip reverse DNS lookup of clients
skip-name-resolve
Thanks for help.
 
cPanelLauren

cPanelLauren

Product Owner II
Staff member
Nov 14, 2017
13,266
1,300
363
Houston
I could not do this change directly on WHM so I did it manually and modified the httpd.conf file.
Click to expand...
Why couldn't you make the changes through WHM? Edits made directly to the Apache configuration will be overwritten, to modify the MaxRequestWorkers option you'll increase MaxClients which is available through the Global Configuration options in WHM.

what would I need to do to increase the server limit? Is there a formula I need to apply in calculating the new severlimit?
Click to expand...
I would refer to Apache's Documentation on ServerLimit before making modifications mpm_common - Apache HTTP Server Version 2.4

For the MySQL optimization you might want to look at major/MySQLTuner-perl

The following thread might be a good starting point as well:
mysqlmymonlite.sh server stats gathering tool for cPanel Server
 
You must log in or register to reply here.
Thread starter Similar threads Forum Replies Date
B Optimising memory used by Apache httpd process Workarounds and Optimization 1
A Can't start Apache again after removing Litespeed Workarounds and Optimization 1
sahostking Thoughts on Apache Issue Workarounds and Optimization 1
C Alternative Apache port for individual site? Workarounds and Optimization 2
6 Memcached + Apache 2.2 crashes site every 5-10 minutes Workarounds and Optimization 7
Similar threads
Optimising memory used by Apache httpd process
Can't start Apache again after removing Litespeed
Thoughts on Apache Issue
Alternative Apache port for individual site?
Memcached + Apache 2.2 crashes site every 5-10 minutes
Top Bottom