Apache Crash: Permission denied: AH00144: couldn't grab the accept mutex

monkey64 · Nov 4, 2013

Apache occasionally crashes, output from the apache error_log blow.
My PHP Handler is DSO, I have Apache suEXEC ON and Apache Ruid2 ON. I am running Apache Apache 2.4.6. The issue started around the same time I changed from SuPHP to DSO and enabled Apache Ruid2.

Code:

[Mon Nov 04 07:59:57.729935 2013] [core:error] [pid 2186] [client 85.25.133.190:59918] AH00126: Invalid URI in request GET HTTP/1.1 HTTP/1.1
[Mon Nov 04 07:59:57.803113 2013] [cgi:error] [pid 3120] [client 85.25.133.190:59941] script not found or unable to stat: /usr/local/apache/cgi-bin/php
[Mon Nov 04 07:59:57.845310 2013] [cgi:error] [pid 3119] [client 85.25.133.190:59995] script not found or unable to stat: /usr/local/apache/cgi-bin/php5
[Mon Nov 04 07:59:57.884308 2013] [cgi:error] [pid 2349] [client 85.25.133.190:60014] script not found or unable to stat: /usr/local/apache/cgi-bin/php-cgi
[Mon Nov 04 07:59:57.929495 2013] [cgi:error] [pid 30334] [client 85.25.133.190:60041] script not found or unable to stat: /usr/local/apache/cgi-bin/php.cgi
[Mon Nov 04 07:59:57.975371 2013] [cgi:error] [pid 3059] [client 85.25.133.190:60066] script not found or unable to stat: /usr/local/apache/cgi-bin/php4
[Mon Nov 04 08:24:54.354599 2013] [:error] [pid 6496] [client 180.76.5.150:63860] script '/home/windcase/public_html/clarinet_triple_lightweight_cases.php' not found or unable to stat
[Mon Nov 04 09:05:24.153575 2013] [mpm_prefork:notice] [pid 6692] AH00171: Graceful restart requested, doing restart
[Mon Nov 04 09:05:25.063435 2013] [:notice] [pid 6692] mod_ruid2/0.9.4a enabled
PHP Deprecated:  Comments starting with '#' are deprecated in /usr/local/lib/php.ini on line 1147 in Unknown on line 0
[Mon Nov 04 09:05:25.078920 2013] [mpm_prefork:notice] [pid 6692] AH00163: Apache/2.4.6 (Unix) OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.4.17 configured -- resuming normal operations
[Mon Nov 04 09:05:25.078949 2013] [core:notice] [pid 6692] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
[Mon Nov 04 09:35:21.476844 2013] [mpm_prefork:emerg] [pid 19138] (13)Permission denied: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:21.480251 2013] [mpm_prefork:emerg] [pid 20107] (13)Permission denied: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:22.136877 2013] [core:alert] [pid 6692] AH00050: Child 19138 returned a Fatal error... Apache is exiting!
[Mon Nov 04 09:35:22.149464 2013] [mpm_prefork:emerg] [pid 19082] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:22.153216 2013] [mpm_prefork:emerg] [pid 20229] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:22.157597 2013] [mpm_prefork:emerg] [pid 18296] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:22.161407 2013] [mpm_prefork:emerg] [pid 20209] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:22.164979 2013] [mpm_prefork:emerg] [pid 20104] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:22.168086 2013] [mpm_prefork:emerg] [pid 19101] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:22.171117 2013] [mpm_prefork:emerg] [pid 19243] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 09:35:22.621741 2013] [mpm_prefork:emerg] [pid 20207] (22)Invalid argument: AH00146: couldn't release the accept mutex
[Mon Nov 04 09:39:17.059919 2013] [suexec:notice] [pid 21260] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Mon Nov 04 09:39:17.060047 2013] [:notice] [pid 21260] ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.
[Mon Nov 04 09:39:17.060064 2013] [:notice] [pid 21260] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Mon Nov 04 09:39:17.060075 2013] [:notice] [pid 21260] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
[Mon Nov 04 09:39:17.060085 2013] [:notice] [pid 21260] ModSecurity: LUA compiled version="Lua 5.1"
[Mon Nov 04 09:39:17.060094 2013] [:notice] [pid 21260] ModSecurity: LIBXML compiled version="2.9.0"
[Mon Nov 04 09:39:18.030767 2013] [:notice] [pid 21261] mod_ruid2/0.9.4a enabled

This post on Apache 2 Crashing recommended adding these lines to httpd.conf file:

Code:

AcceptMutex fcntl

After I did that, Apache refused to start, see error below:

Code:

Apache restart failed. Unable to load pid from pid file and no httpd process found in process list.

If apache restart reported success but it failed soon after, it may be caused by oddities with mod_ssl.

You should run /usr/local/cpanel/scripts/ssl_crt_status as part of your troubleshooting process. Pass it --help for more details.

Also be sure to examine apache's various log files.
Apache Restart Output:

Log:
[Mon Nov 04 09:39:17.059919 2013] [suexec:notice] [pid 21260] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Mon Nov 04 09:39:17.060047 2013] [:notice] [pid 21260] ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.
[Mon Nov 04 09:39:17.060064 2013] [:notice] [pid 21260] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Mon Nov 04 09:39:17.060075 2013] [:notice] [pid 21260] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
[Mon Nov 04 09:39:17.060085 2013] [:notice] [pid 21260] ModSecurity: LUA compiled version="Lua 5.1"
[Mon Nov 04 09:39:17.060094 2013] [:notice] [pid 21260] ModSecurity: LIBXML compiled version="2.9.0"
[Mon Nov 04 09:39:18.030767 2013] [:notice] [pid 21261] mod_ruid2/0.9.4a enabled
PHP Deprecated:  Comments starting with '#' are deprecated in /usr/local/lib/php.ini on line 1147 in Unknown on line 0
[Mon Nov 04 09:39:18.063982 2013] [mpm_prefork:notice] [pid 21261] AH00163: Apache/2.4.6 (Unix) OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.4.17 configured -- resuming normal operations
[Mon Nov 04 09:39:18.064047 2013] [core:notice] [pid 21261] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
[Mon Nov 04 09:53:33.031505 2013] [mpm_prefork:notice] [pid 21261] AH00169: caught SIGTERM, shutting down

But then I read this post which says that for Apache 2.4 we need to use:

Code:

Mutex posixsem

The above at least gets Apache started, but s a relative newbie, I was surprised that I needed to hack the httpd.conf file. Is this a recommended approach?

cPanelMichael · Nov 4, 2013

Hello

I found a couple of cases where enabling piped logging via "WHM Home » Service Configuration » Apache Configuration » Piped Log Configuration" resolved this issue. Could you try enabling piped logging, and removing any custom changes you made to initially resolve the issue?

Thank you.

monkey64 · Nov 4, 2013

No luck cPanelMichael.

It crashed a couple of hours after I removed the custom changes and enabled piped logging.
Here is another output:

Code:

[Mon Nov 04 17:33:27.827354 2013] [core:alert] [pid 8313] AH00050: Child 28075 returned a Fatal error... Apache is exiting!
[Mon Nov 04 17:33:27.839808 2013] [mpm_prefork:emerg] [pid 26086] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 17:33:27.843412 2013] [mpm_prefork:emerg] [pid 29130] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 17:33:27.846220 2013] [mpm_prefork:emerg] [pid 28074] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 17:33:27.849052 2013] [mpm_prefork:emerg] [pid 29162] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 17:33:27.849406 2013] [mpm_prefork:emerg] [pid 29160] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 17:33:27.851990 2013] [mpm_prefork:emerg] [pid 29041] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 17:33:27.852454 2013] [mpm_prefork:emerg] [pid 29161] (43)Identifier removed: AH00144: couldn't grab the accept mutex
[Mon Nov 04 17:33:33.661534 2013] [mpm_prefork:emerg] [pid 27882] (22)Invalid argument: AH00144: couldn't grab the accept mutex
[Mon Nov 04 17:33:47.644485 2013] [mpm_prefork:emerg] [pid 27419] (22)Invalid argument: AH00146: couldn't release the accept mutex
[Mon Nov 04 17:35:56.097004 2013] [suexec:notice] [pid 30073] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)

cPanelMichael · Nov 4, 2013

In response to your other question in your initial post, yes, the following value is sometimes recommended:

Code:

AcceptMutex posixsem

If you want to avoid directly editing and distilling the Apache configuration file, you can add this line directly to the following file:

Code:

/usr/local/apache/conf/mod_ruid2.conf

You will find this in our documentation:

Apache Mod_Ruid2

Does the issue continue when adding this value?

Thank you.

monkey64 · Nov 5, 2013

The issue seems to be resoved, many thanks Michael.

monkey64 · Nov 6, 2013

Actaully, I spoke too soon. The problem is not fixed and Apache crashed again...
I checked that my change had been distilled, and sure enough found the line:

Code:

Mutex posixsem

in httpd.conf.
DSO seems to be unstable so I have temporarily switched back to Suexec.
There is obviously an issue when I run DSO. What do I do now?

cPanelMichael · Nov 6, 2013

You are welcome to open a support ticket if you want us to take a closer look:

Submit A Ticket

Post the ticket number here and we can update this thread with the outcome.

Thank you.

monkey64 · Dec 9, 2013

I followed the instruction on the Cpanel Apache Mod_Ruid2 page:

"To resolve this issue, you may add the following line to /usr/local/apache/conf/mod_ruid2.conf:"
AcceptMutex posixsem

So here is my mod_ruid2.conf file:

Code:

<IfModule mod_ruid2.c>
    RMode config
    RDefaultUidGid nobody nobody
    RUidGid nobody nobody
    AcceptMutex posixsem
 </IfModule>

And here is the output once I restarted Apache:

Code:

Apache restart failed. Unable to load pid from pid file and no httpd process found in process list.

If apache restart reported success but it failed soon after, it may be caused by oddities with mod_ssl.

You should run /usr/local/cpanel/scripts/ssl_crt_status as part of your troubleshooting process. Pass it --help for more details.

Also be sure to examine apache's various log files.
Apache Restart Output:

Log:
[Mon Dec 09 08:56:34.004436 2013] [:notice] [pid 32286] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
[Mon Dec 09 08:56:34.004446 2013] [:notice] [pid 32286] ModSecurity: LUA compiled version="Lua 5.1"
[Mon Dec 09 08:56:34.004455 2013] [:notice] [pid 32286] ModSecurity: LIBXML compiled version="2.9.0"
[Mon Dec 09 08:56:35.008221 2013] [:notice] [pid 32287] mod_ruid2/0.9.8 enabled
[Mon Dec 09 08:56:35.034803 2013] [mpm_prefork:notice] [pid 32287] AH00163: Apache/2.4.7 (Unix) OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.4.22 configured -- resuming normal operations
[Mon Dec 09 08:56:35.034889 2013] [core:notice] [pid 32287] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
[Mon Dec 09 08:57:31.255902 2013] [mpm_prefork:notice] [pid 32287] AH00171: Graceful restart requested, doing restart
[Mon Dec 09 08:57:32.006902 2013] [:notice] [pid 32287] mod_ruid2/0.9.8 enabled
[Mon Dec 09 08:57:32.038743 2013] [mpm_prefork:notice] [pid 32287] AH00163: Apache/2.4.7 (Unix) OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.4.22 configured -- resuming normal operations
[Mon Dec 09 08:57:32.038772 2013] [core:notice] [pid 32287] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
[Mon Dec 09 08:58:44.174091 2013] [mpm_prefork:notice] [pid 32287] AH00169: caught SIGTERM, shutting down

I should point out that removing "AcceptMutex posixsem" allows Apache to restart correctly.
There seems to be a mod_ssl error. I checked the Service Certificates in Home » Service Configuration » Manage Service SSL Certificates and they are all valid and expire next year.

Any ideas?

cPanelMichael · Dec 9, 2013

Could you open a support ticket using the link in my signature so we can take a closer look? You can post the ticket number here so we can update this thread with the outcome.

Thank you.

monkey64 · Dec 11, 2013

cPanelMichael said:
In response to your other question in your initial post, yes, the following value is sometimes recommended:

Code:

AcceptMutex posixsem

That didn't work for me. I found that adding:

Code:

Mutex posixsem

to mod_ruid2.conf at last solved the problem.
I tried adding the line to http.conf and distilling, but the only way I got it to work was to edit mod_ruid2.conf directly.

BTW: I'm running Apache 2.4.7.

KurtN. · Dec 16, 2013

I wasn't able to duplicate your issue with respect to changing mutex's using the new Mutex directive in Apache 2.4.

Here are the steps I used to do so successfully:

Look for the following lines in httpd.conf

Code:

<IfModule prefork.c>
    Mutex default mpm-accept

</IfModule>

Change it to

Code:

<IfModule prefork.c>
    Mutex posixsem mpm-accept

</IfModule>

Run the distiller

Code:

# /usr/local/cpanel/bin/apache_conf_distiller --update --store-data
Distilled successfully

Ensure it distilled correctly

Code:

# /usr/local/cpanel/bin/build_apache_conf 
Built /usr/local/apache/conf/httpd.conf OK

paoloie · May 26, 2014

I have been getting a lot of server failures lately too. The error log shows that "couldn't grab the accept mutex" line every time if crashes.
The server Automagically restarts every time but I want to get rid of the error.
So I tried adding

Code:

AcceptMutex posixsem

to

Code:

/usr/local/apache/conf/mod_ruid2.conf

But the server would not restart after that, so I had to take that line back out.
Then it restarted.
I have now enables Piped Apache Logs and will wait to see if it falls over again.

In the meantime, is there something else I could try if it does?

Thanks
Paoloie

Thread starter	Similar threads	Forum	Replies	Date
M	Directory /var/log/apache2/modsec_audit/<user>/YYYYMMDD... not being cleaned up	File Management	3	Apr 24, 2022
	limit UPLOAD files to 1KB from APACHE, NOT from scripts	File Management	41	Feb 21, 2022
H	Apache Missing	File Management	5	Jul 5, 2021
B	Apache write permission varies by server with same permissions	File Management	1	Nov 27, 2020
	[CPANEL-19869] Apache mod_evasive permissions	File Management	14	Sep 24, 2018

Search

Search

Apache Crash: Permission denied: AH00144: couldn't grab the accept mutex

monkey64

Well-Known Member

cPanelMichael

Administrator

monkey64

Well-Known Member

cPanelMichael

Administrator

monkey64

Well-Known Member

monkey64

Well-Known Member

cPanelMichael

Administrator

monkey64

Well-Known Member

cPanelMichael

Administrator

monkey64

Well-Known Member

KurtN.

Well-Known Member

paoloie

Registered