aha_creative

Registered
Dec 1, 2022
4
0
1
United States
cPanel Access Level
Reseller Owner
Currently on Apache 2.4.57 and I've noticed a pattern that will leads to Apache needing to be restarted. When examining the error logs, there will be a large spike in mod security blocked requests from multiple IPs followed by a lot of scoreboard errors over the next couple of hours
[mpm_event:error] [pid 23084:tid 47651189847104] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
with
[http2:warn] [pid 24661:tid 47651596809984] [client 52.167.144.172:51392] h2_stream(24661-1193-1,CLEANUP): started=1, scheduled=1, ready=0, out_buffer=0
interspersed. Finally there will be multiple
[core:warn] [pid 23084:tid 47651189847104] AH00045: child process 27439 still did not exit, sending a SIGTERM
.

Current mpm event settings:
Code:
    AsyncRequestWorkerFactor 2
    ServerLimit 20
    MaxRequestWorkers 2000
    StartServers 20
    MaxConnectionsPerChild 10000
    ThreadsPerChild 100
    # MaxRequestWorkers/4
    MinSpareThreads 500
    # MinSpareThreads * 3
    MaxSpareThreads 1500
    ThreadLimit 128
It's my understanding that MaxRequestWorkers does not "count" gracefully shutting down threads but it still counts towards ThreadsPerChild. I've tried increasing the server limit to 30, but we still experience the same issue.

First question, is it better to increase the server limit to deal with the issue or should I adjust AsyncRequestWorkerFactor?

Second question is about cPanel's support documentation related to the issue.
It states that it's a know issue in Apache <= 2.4.7 and since cPanel uses 2.4.54 it's still susceptible.

Does cPanel use a different versioning scheme?

According to Apache's release tags, Tags · apache/httpd, 2.4.7 was released on Nov 19, 2013 and 2.4.54 was released on Jun 8, 2022.
 
Last edited by a moderator:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,505
2,605
363
cPanel Access Level
Root Administrator
Hey there! I believe that support article may be inaccurate as Apache has listed that bug as being resolved since 2017:


Since that is the case, it would seem something else is happening with your machine.

If you check the Apache scoreboard with the "apachectl fullstatus" command, do you see that any certain type of process is happening too much, or that an area is full?
 

aha_creative

Registered
Dec 1, 2022
4
0
1
United States
cPanel Access Level
Reseller Owner
Hey there! I believe that support article may be inaccurate as Apache has listed that bug as being resolved since 2017:


Since that is the case, it would seem something else is happening with your machine.

If you check the Apache scoreboard with the "apachectl fullstatus" command, do you see that any certain type of process is happening too much, or that an area is full?

Not seeing anything out of the ordinary, but I would imagine I would need to catch the server during the issue and not after cPanel has forced a restart.

Just to confirm, I have this block in our pre-virtual-host include. Is that the correct place to add mpm event configuration and is the if block correct?
Code:
<IfModule mod_mpm_event>
   AsyncRequestWorkerFactor 2
    ServerLimit 20
    MaxRequestWorkers 2000
    StartServers 20
    MaxConnectionsPerChild 10000
    ThreadsPerChild 100
    # MaxRequestWorkers/4
    MinSpareThreads 500
    # MinSpareThreads * 3
    MaxSpareThreads 1500
    ThreadLimit 128
</IfModule>
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,505
2,605
363
cPanel Access Level
Root Administrator
Yes, you would need to check the scoreboard during the issue to see what is happening.

You wouldn't be able to change the mpm through an include, and I wouldn't expect those settings to be applied since they are called later in the configuration than where the include is read. You can check the current mpm by checking "apachectl status" as that will show a "Server MPM:" line in the output. If it isn't set to Event already and that is what you want, you would need to change this through the WHM >> EasyApache 4 interface.

If it is already Event, there is no reason to have that include as you would adjust those values through WHM >> Apache Configuration.
 

aha_creative

Registered
Dec 1, 2022
4
0
1
United States
cPanel Access Level
Reseller Owner
Yes, you would need to check the scoreboard during the issue to see what is happening.

You wouldn't be able to change the mpm through an include, and I wouldn't expect those settings to be applied since they are called later in the configuration than where the include is read. You can check the current mpm by checking "apachectl status" as that will show a "Server MPM:" line in the output. If it isn't set to Event already and that is what you want, you would need to change this through the WHM >> EasyApache 4 interface.

If it is already Event, there is no reason to have that include as you would adjust those values through WHM >> Apache Configuration.

Sorry, I should have been clearer. MPM event is set through Easy Apache. I have that config in the pre virtual host file since WHM doesn't have every value I would like to set; specifically ThreadsPerChild, MinSpareThreads, MaxSpareThreads, and ThreadLimit. I went ahead and set the other directives there so all the mpm configuration would be in one spot. Is there a better way to set these directives than what we're using?

Checking "apachectl status" confirms MPM is Event.
Server MPM: event
Couple of follow-up questions.

Is there a good way to confirm the final values Apache is using?

If I catch the server during one of these issues, is there anything specific I'm looking for with "apachectl fullstatus"? I'm guessing the scoreboard full of G or I?
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
16,505
2,605
363
cPanel Access Level
Root Administrator
Thanks for that clarification. Yes, if you want to add values that aren't available in the interface, that is the correct include to use.

Since you have the options in the file, it would be using those configurations. The closest option available that I know of is this:

Code:
apachectl -t -D DUMP_RUN_CFG
which will give you the following output:

Code:
[root@host /]# apachectl -t -D DUMP_RUN_CFG
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/etc/apache2/htdocs"
Main ErrorLog: "/etc/apache2/logs/error_log"
Mutex rewrite-map: dir="/run/apache2" mechanism=fcntl
Mutex ssl-stapling-refresh: using_defaults
Mutex fcgid-proctbl: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: dir="/run/apache2" mechanism=fcntl
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex fcgid-pipe: using_defaults
PidFile: "/run/apache2/httpd.pid"
Define: DUMP_RUN_CFG
Define: MODSEC_2.5
Define: MODSEC_2.9
User: name="nobody" id=65534
Group: name="nobody" id=65534
"apachectl -t -D DUMP_MODULES" will show the modules loaded on the server as well.

There's some good details about this option here that you might want to read as well: Apache Web Server Dump Current Running Configuration Settings - CodingShower

As far as the output, yes, you'd want to see what letters show up in the scoreboard, or if there are any free idle workers.
 

aha_creative

Registered
Dec 1, 2022
4
0
1
United States
cPanel Access Level
Reseller Owner
Thank you cPRex, using some of the suggested commands I was able to realize what was happening. I had the wrong statement in the IfModule block so it was using default values for Min/Max Spare Threads and Threads per child. Essentially I thought the server was limited to 2000 connections when it was actually being limited to 500. In addition it was constantly shutting down and spinning up new threads because of the low Min/Max Spare Thread directives (75 and 250). So I think it was getting into a state where all the threads were either in use or shutting down but the scorecard wasn't full. Eventually it would become unable to respond in a timely manner to requests and WHM would restart the service.

I realized this when checking "apachectl fullstatus" since the scorecard looked a little small for the number of processes that should be running.

For anyone who stumbles upon this I was using this statement for the block:
Code:
<IfModule mod_mpm_event>
   ...
</IfModule>
I should have used:
Code:
<IfModule mpm_event_module>
    ...
</IfModule>