The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Apache Crash: Permission denied: AH00144: couldn't grab the accept mutex

Discussion in 'EasyApache' started by monkey64, Nov 4, 2013.

  1. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Apache occasionally crashes, output from the apache error_log blow.
    My PHP Handler is DSO, I have Apache suEXEC ON and Apache Ruid2 ON. I am running Apache Apache 2.4.6. The issue started around the same time I changed from SuPHP to DSO and enabled Apache Ruid2.

    Code:
    [Mon Nov 04 07:59:57.729935 2013] [core:error] [pid 2186] [client 85.25.133.190:59918] AH00126: Invalid URI in request GET HTTP/1.1 HTTP/1.1
    [Mon Nov 04 07:59:57.803113 2013] [cgi:error] [pid 3120] [client 85.25.133.190:59941] script not found or unable to stat: /usr/local/apache/cgi-bin/php
    [Mon Nov 04 07:59:57.845310 2013] [cgi:error] [pid 3119] [client 85.25.133.190:59995] script not found or unable to stat: /usr/local/apache/cgi-bin/php5
    [Mon Nov 04 07:59:57.884308 2013] [cgi:error] [pid 2349] [client 85.25.133.190:60014] script not found or unable to stat: /usr/local/apache/cgi-bin/php-cgi
    [Mon Nov 04 07:59:57.929495 2013] [cgi:error] [pid 30334] [client 85.25.133.190:60041] script not found or unable to stat: /usr/local/apache/cgi-bin/php.cgi
    [Mon Nov 04 07:59:57.975371 2013] [cgi:error] [pid 3059] [client 85.25.133.190:60066] script not found or unable to stat: /usr/local/apache/cgi-bin/php4
    [Mon Nov 04 08:24:54.354599 2013] [:error] [pid 6496] [client 180.76.5.150:63860] script '/home/windcase/public_html/clarinet_triple_lightweight_cases.php' not found or unable to stat
    [Mon Nov 04 09:05:24.153575 2013] [mpm_prefork:notice] [pid 6692] AH00171: Graceful restart requested, doing restart
    [Mon Nov 04 09:05:25.063435 2013] [:notice] [pid 6692] mod_ruid2/0.9.4a enabled
    PHP Deprecated:  Comments starting with '#' are deprecated in /usr/local/lib/php.ini on line 1147 in Unknown on line 0
    [Mon Nov 04 09:05:25.078920 2013] [mpm_prefork:notice] [pid 6692] AH00163: Apache/2.4.6 (Unix) OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.4.17 configured -- resuming normal operations
    [Mon Nov 04 09:05:25.078949 2013] [core:notice] [pid 6692] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
    [Mon Nov 04 09:35:21.476844 2013] [mpm_prefork:emerg] [pid 19138] (13)Permission denied: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:21.480251 2013] [mpm_prefork:emerg] [pid 20107] (13)Permission denied: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:22.136877 2013] [core:alert] [pid 6692] AH00050: Child 19138 returned a Fatal error... Apache is exiting!
    [Mon Nov 04 09:35:22.149464 2013] [mpm_prefork:emerg] [pid 19082] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:22.153216 2013] [mpm_prefork:emerg] [pid 20229] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:22.157597 2013] [mpm_prefork:emerg] [pid 18296] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:22.161407 2013] [mpm_prefork:emerg] [pid 20209] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:22.164979 2013] [mpm_prefork:emerg] [pid 20104] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:22.168086 2013] [mpm_prefork:emerg] [pid 19101] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:22.171117 2013] [mpm_prefork:emerg] [pid 19243] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 09:35:22.621741 2013] [mpm_prefork:emerg] [pid 20207] (22)Invalid argument: AH00146: couldn't release the accept mutex
    [Mon Nov 04 09:39:17.059919 2013] [suexec:notice] [pid 21260] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
    [Mon Nov 04 09:39:17.060047 2013] [:notice] [pid 21260] ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.
    [Mon Nov 04 09:39:17.060064 2013] [:notice] [pid 21260] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
    [Mon Nov 04 09:39:17.060075 2013] [:notice] [pid 21260] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
    [Mon Nov 04 09:39:17.060085 2013] [:notice] [pid 21260] ModSecurity: LUA compiled version="Lua 5.1"
    [Mon Nov 04 09:39:17.060094 2013] [:notice] [pid 21260] ModSecurity: LIBXML compiled version="2.9.0"
    [Mon Nov 04 09:39:18.030767 2013] [:notice] [pid 21261] mod_ruid2/0.9.4a enabled
    
    This post on Apache 2 Crashing recommended adding these lines to httpd.conf file:

    Code:
    AcceptMutex fcntl
    
    After I did that, Apache refused to start, see error below:

    Code:
    Apache restart failed. Unable to load pid from pid file and no httpd process found in process list.
    
    If apache restart reported success but it failed soon after, it may be caused by oddities with mod_ssl.
    
    You should run /usr/local/cpanel/scripts/ssl_crt_status as part of your troubleshooting process. Pass it --help for more details.
    
    Also be sure to examine apache's various log files.
    Apache Restart Output:
    
    Log:
    [Mon Nov 04 09:39:17.059919 2013] [suexec:notice] [pid 21260] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
    [Mon Nov 04 09:39:17.060047 2013] [:notice] [pid 21260] ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.
    [Mon Nov 04 09:39:17.060064 2013] [:notice] [pid 21260] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
    [Mon Nov 04 09:39:17.060075 2013] [:notice] [pid 21260] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
    [Mon Nov 04 09:39:17.060085 2013] [:notice] [pid 21260] ModSecurity: LUA compiled version="Lua 5.1"
    [Mon Nov 04 09:39:17.060094 2013] [:notice] [pid 21260] ModSecurity: LIBXML compiled version="2.9.0"
    [Mon Nov 04 09:39:18.030767 2013] [:notice] [pid 21261] mod_ruid2/0.9.4a enabled
    PHP Deprecated:  Comments starting with '#' are deprecated in /usr/local/lib/php.ini on line 1147 in Unknown on line 0
    [Mon Nov 04 09:39:18.063982 2013] [mpm_prefork:notice] [pid 21261] AH00163: Apache/2.4.6 (Unix) OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.4.17 configured -- resuming normal operations
    [Mon Nov 04 09:39:18.064047 2013] [core:notice] [pid 21261] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
    [Mon Nov 04 09:53:33.031505 2013] [mpm_prefork:notice] [pid 21261] AH00169: caught SIGTERM, shutting down
    
    But then I read this post which says that for Apache 2.4 we need to use:

    Code:
    Mutex posixsem
    
    The above at least gets Apache started, but s a relative newbie, I was surprised that I needed to hack the httpd.conf file. Is this a recommended approach?
     
    #1 monkey64, Nov 4, 2013
    Last edited: Nov 4, 2013
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    648
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    I found a couple of cases where enabling piped logging via "WHM Home » Service Configuration » Apache Configuration » Piped Log Configuration" resolved this issue. Could you try enabling piped logging, and removing any custom changes you made to initially resolve the issue?

    Thank you.
     
  3. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    No luck cPanelMichael.

    It crashed a couple of hours after I removed the custom changes and enabled piped logging.
    Here is another output:

    Code:
    [Mon Nov 04 17:33:27.827354 2013] [core:alert] [pid 8313] AH00050: Child 28075 returned a Fatal error... Apache is exiting!
    [Mon Nov 04 17:33:27.839808 2013] [mpm_prefork:emerg] [pid 26086] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 17:33:27.843412 2013] [mpm_prefork:emerg] [pid 29130] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 17:33:27.846220 2013] [mpm_prefork:emerg] [pid 28074] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 17:33:27.849052 2013] [mpm_prefork:emerg] [pid 29162] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 17:33:27.849406 2013] [mpm_prefork:emerg] [pid 29160] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 17:33:27.851990 2013] [mpm_prefork:emerg] [pid 29041] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 17:33:27.852454 2013] [mpm_prefork:emerg] [pid 29161] (43)Identifier removed: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 17:33:33.661534 2013] [mpm_prefork:emerg] [pid 27882] (22)Invalid argument: AH00144: couldn't grab the accept mutex
    [Mon Nov 04 17:33:47.644485 2013] [mpm_prefork:emerg] [pid 27419] (22)Invalid argument: AH00146: couldn't release the accept mutex
    [Mon Nov 04 17:35:56.097004 2013] [suexec:notice] [pid 30073] AH01232: suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
    
     
  4. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    648
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    In response to your other question in your initial post, yes, the following value is sometimes recommended:

    Code:
    AcceptMutex posixsem
    If you want to avoid directly editing and distilling the Apache configuration file, you can add this line directly to the following file:

    Code:
    /usr/local/apache/conf/mod_ruid2.conf
    You will find this in our documentation:

    Apache Mod_Ruid2

    Does the issue continue when adding this value?

    Thank you.
     
  5. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    The issue seems to be resoved, many thanks Michael.
     
  6. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Actaully, I spoke too soon. The problem is not fixed and Apache crashed again...
    I checked that my change had been distilled, and sure enough found the line:

    Code:
    Mutex posixsem 
    
    in httpd.conf.
    DSO seems to be unstable so I have temporarily switched back to Suexec.
    There is obviously an issue when I run DSO. What do I do now?
     
    #6 monkey64, Nov 6, 2013
    Last edited: Nov 6, 2013
  7. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    648
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You are welcome to open a support ticket if you want us to take a closer look:

    Submit A Ticket

    Post the ticket number here and we can update this thread with the outcome.

    Thank you.
     
  8. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    I followed the instruction on the Cpanel Apache Mod_Ruid2 page:

    "To resolve this issue, you may add the following line to /usr/local/apache/conf/mod_ruid2.conf:"
    AcceptMutex posixsem

    So here is my mod_ruid2.conf file:

    Code:
    <IfModule mod_ruid2.c>
        RMode config
        RDefaultUidGid nobody nobody
        RUidGid nobody nobody
        AcceptMutex posixsem
     </IfModule>
    
    And here is the output once I restarted Apache:

    Code:
    Apache restart failed. Unable to load pid from pid file and no httpd process found in process list.
    
    If apache restart reported success but it failed soon after, it may be caused by oddities with mod_ssl.
    
    You should run /usr/local/cpanel/scripts/ssl_crt_status as part of your troubleshooting process. Pass it --help for more details.
    
    Also be sure to examine apache's various log files.
    Apache Restart Output:
    
    Log:
    [Mon Dec 09 08:56:34.004436 2013] [:notice] [pid 32286] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
    [Mon Dec 09 08:56:34.004446 2013] [:notice] [pid 32286] ModSecurity: LUA compiled version="Lua 5.1"
    [Mon Dec 09 08:56:34.004455 2013] [:notice] [pid 32286] ModSecurity: LIBXML compiled version="2.9.0"
    [Mon Dec 09 08:56:35.008221 2013] [:notice] [pid 32287] mod_ruid2/0.9.8 enabled
    [Mon Dec 09 08:56:35.034803 2013] [mpm_prefork:notice] [pid 32287] AH00163: Apache/2.4.7 (Unix) OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.4.22 configured -- resuming normal operations
    [Mon Dec 09 08:56:35.034889 2013] [core:notice] [pid 32287] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
    [Mon Dec 09 08:57:31.255902 2013] [mpm_prefork:notice] [pid 32287] AH00171: Graceful restart requested, doing restart
    [Mon Dec 09 08:57:32.006902 2013] [:notice] [pid 32287] mod_ruid2/0.9.8 enabled
    [Mon Dec 09 08:57:32.038743 2013] [mpm_prefork:notice] [pid 32287] AH00163: Apache/2.4.7 (Unix) OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 PHP/5.4.22 configured -- resuming normal operations
    [Mon Dec 09 08:57:32.038772 2013] [core:notice] [pid 32287] AH00094: Command line: '/usr/local/apache/bin/httpd -D SSL'
    [Mon Dec 09 08:58:44.174091 2013] [mpm_prefork:notice] [pid 32287] AH00169: caught SIGTERM, shutting down
    
    I should point out that removing "AcceptMutex posixsem" allows Apache to restart correctly.
    There seems to be a mod_ssl error. I checked the Service Certificates in Home » Service Configuration » Manage Service SSL Certificates and they are all valid and expire next year.

    Any ideas?
     
    #8 monkey64, Dec 9, 2013
    Last edited: Dec 9, 2013
  9. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    648
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Could you open a support ticket using the link in my signature so we can take a closer look? You can post the ticket number here so we can update this thread with the outcome.

    Thank you.
     
  10. monkey64

    monkey64 Well-Known Member

    Joined:
    Nov 6, 2011
    Messages:
    86
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    That didn't work for me. I found that adding:

    Code:
    Mutex posixsem
    
    to mod_ruid2.conf at last solved the problem.
    I tried adding the line to http.conf and distilling, but the only way I got it to work was to edit mod_ruid2.conf directly.

    BTW: I'm running Apache 2.4.7.
     
  11. cPanelKurtN

    cPanelKurtN Well-Known Member
    Staff Member

    Joined:
    Jan 29, 2013
    Messages:
    95
    Likes Received:
    1
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    I wasn't able to duplicate your issue with respect to changing mutex's using the new Mutex directive in Apache 2.4.

    Here are the steps I used to do so successfully:
    • Look for the following lines in httpd.conf
      Code:
      <IfModule prefork.c>
          Mutex default mpm-accept
      
      </IfModule>
    • Change it to
      Code:
      <IfModule prefork.c>
          Mutex posixsem mpm-accept
      
      </IfModule>
    • Run the distiller
      Code:
      # /usr/local/cpanel/bin/apache_conf_distiller --update --store-data
      Distilled successfully
    • Ensure it distilled correctly
      Code:
      # /usr/local/cpanel/bin/build_apache_conf 
      Built /usr/local/apache/conf/httpd.conf OK
     
  12. paoloie

    paoloie Registered

    Joined:
    May 15, 2014
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I have been getting a lot of server failures lately too. The error log shows that "couldn't grab the accept mutex" line every time if crashes.
    The server Automagically restarts every time but I want to get rid of the error.
    So I tried adding
    Code:
    AcceptMutex posixsem
    to

    Code:
    /usr/local/apache/conf/mod_ruid2.conf
    But the server would not restart after that, so I had to take that line back out.
    Then it restarted.
    I have now enables Piped Apache Logs and will wait to see if it falls over again.

    In the meantime, is there something else I could try if it does?

    Thanks
    Paoloie
     
Loading...

Share This Page