Apache graceful restarts and FastCGI - mod_fcgid: can't lock process table in pid

[ahostli]

Hi,

First of all sorry for my english

I was wondering if any of You, using Apache 2.2.X, mod_fcgid and SuExec for handling php requests, experienced any problem after most recent EasyApache update?

After upgrading, Ive noticed, that number of sleeping fcgid processes increased dramaticaly, "eating" RAM and eventualy OS starts killing other services (mysql, bind...) to free up memory. Only after doing hard apache reset all fcgid sleeping processes are killed.
All mod_fcgid configuration was left the same.


In apache error_log:

[Wed Mar 17 09:23:45 2010] [notice] Graceful restart requested, doing restart
[Wed Mar 17 09:23:45 2010] [emerg] mod_fcgid: server is restarted, pid 10345 must exit
[Wed Mar 17 09:23:45 2010] [emerg] (22)Invalid argument: mod_fcgid: can't lock process table in PM, pid 10345
[Wed Mar 17 09:23:45 2010] [emerg] [client 88.216.6.174] (22)Invalid argument: mod_fcgid: can't lock process table in pid 104$
[Wed Mar 17 09:23:45 2010] [emerg] [client 88.216.6.174] (22)Invalid argument: mod_fcgid: can't lock process table in pid 104$
[Wed Mar 17 09:23:45 2010] [emerg] [client 88.216.6.174] (22)Invalid argument: mod_fcgid: can't lock process table in pid 113$

[Wed Mar 17 14:02:57 2010] [notice] Graceful restart requested, doing restart
[Wed Mar 17 14:02:57 2010] [emerg] [client 82.135.207.33] (43)Identifier removed: mod_fcgid: can't get pipe mutex, referer: h$
[Wed Mar 17 14:02:57 2010] [emerg] [client 78.61.82.52] (43)Identifier removed: mod_fcgid: can't get pipe mutex, referer: htt$
[Wed Mar 17 14:02:57 2010] [emerg] [client 91.121.87.87] (22)Invalid argument: mod_fcgid: can't lock process table in pid 170$
[Wed Mar 17 14:02:57 2010] [emerg] [client 91.121.88.99] (22)Invalid argument: mod_fcgid: can't lock process table in pid 160$
[Wed Mar 17 14:02:57 2010] [emerg] mod_fcgid: server is restarted, pid 27466 must exit
[Wed Mar 17 14:02:57 2010] [emerg] (22)Invalid argument: mod_fcgid: can't lock process table in PM, pid 27466

I will try to debug the problem, but perhaps someone already experiencied the same problem and was able to find the solution to it?

 

[linuxtuts]

I have the same problem ...

[Sat Mar 20 02:08:00 2010] [emerg] [client 88.11.171.140] (22)Invalid argument: mod_fcgid: can't lock process table in pid 10635, referer: http://www.mysite.net/
[Sat Mar 20 02:08:01 2010] [emerg] [client 88.11.171.140] (22)Invalid argument: mod_fcgid: can't lock process table in pid 10633, referer: http://www.mysite.net/
[Sat Mar 20 02:08:01 2010] [emerg] [client 88.11.171.140] (22)Invalid argument: mod_fcgid: can't lock process table in pid 13310, referer: http://www.mysite.net/
[Sat Mar 20 02:08:01 2010] [emerg] [client 88.11.171.140] (22)Invalid argument: mod_fcgid: can't lock process table in pid 13324, referer: http://www.mysite.net/

Any thought is to how to fix this?

Thanks in advanced.

 

[ahostli]

It seems, that any fcgid processes that were active/sleeping before executing graceful apache httpd restart, never gets killed after such restart. They get killed only after hard apache restart. This problem causes fcgid processes to „pile up“after each graceful restart, therefore consuming memory and eventually making the whole system unusable.

 

[linuxtuts]

Hum, I never even checked the memory level until you just mentioned it, and you are right.
Out of 2GB of RAM I had 232MB left. ( Thank God it was enough to reboot )

But I also use a "System Integrity Monitor" script from

http://www.rfxn.com/projects/system-integrity-monitor/

which does a GREAT job along with cPanel's built in functions so I never have to worry about the sever services not running or the system overloading.

Looks like I'm gonna have to find that old script I used when I had PLESK to let me know when I was outta memory, and automatically reboot when needed. ( I hate that, for-real )

No one more familiar with mod_fcgid and SuExec having these problems?

Thanks,

 

[wizzy420]

Same problem here, been going on for months. Had to write a script to test for it and manually kill off PHP processes.

PITA

 

[Mo9a7i]

any update on this one guys? if there's any patch or work around,
i've been facing this for a month and my clients are being annoyed.
I'm using a combo of apache+nginx+fcgi and this error occurs whenever we see a 502 nginx bad gateway message.


wizzy, can you share your code.
Maybe I can customize it to help me a bit if there's no patch yet.

 

[cPanelTristan]

Anyone still experiencing this issue, please open a ticket. We had a patch for FCGI issues related to the graceful restarts based on a case with development (36547), so if that patch hasn't resolved any issues with the graceful restarts, we need tickets and systems to check into the issue and see it happening. Please try not to correct the issue on the system when opening the ticket so we can see the system in the state of high memory usage.

You can open a ticket using the link in my signature or in WHM > Support Center > Contact cPanel

Thanks!

 

[LeGastronome]

Apache mod_fcgid and SuExec problems, plz help

Hello,

Since fews days i have this type of warn and error in the apache's error log :

Sometimes server show a 500 internal error page. It seems to come from mod_fcgid

[Sat Oct 30 00:38:12 2010] [warn] [client 86.201.143.192] (104)Connection reset by peer: mod_fcgid: can't get data from http client, referer: http:

or

[Fri Oct 29 13:05:41 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Fri Oct 29 13:05:42 2010] [notice] Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.5 configured -- resuming normal operations
[Fri Oct 29 13:05:48 2010] [notice] SIGUSR1 received.  Doing graceful restart
[Fri Oct 29 13:05:50 2010] [emerg] [client 66.249.65.104] (22)Invalid argument: mod_fcgid: can't lock process table in pid 1332
[Fri Oct 29 13:05:52 2010] [warn] [client 77.202.105.147] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server
[Fri Oct 29 13:05:52 2010] [error] [client 77.202.105.147] Premature end of script headers: index.php
[Fri Oct 29 13:05:52 2010] [error] [client 77.202.105.147] File does not exist: /home/**********
[Fri Oct 29 13:05:53 2010] [notice] Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.5 configured -- resuming normal operations
[Fri Oct 29 13:05:53 2010] [emerg] [client 77.202.105.147] (43)Identifier removed: mod_fcgid: can't lock process table in pid 32743
[Fri Oct 29 13:07:10 2010] [notice] caught SIGTERM, shutting down
[Fri Oct 29 13:07:56 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Fri Oct 29 13:07:57 2010] [notice] Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.5 configured -- resuming normal operations

Thanks,

 

[cPanelTristan]

Please submit a ticket using the link in my signature or using WHM > Support Center > Contact cPanel area. We have another forum post about this same issue:

Apache mod_fcgid problems after recent update of Apache 2.2.15 - cPanel Forums

We also have a prior bug case (36547) about similar issues with the Apache graceful restarts when using fcgi. We need servers to investigate the error to see what is happening to cause this "Invalid argument: mod_fcgid: can't lock process table in pid #" type of error.

 

[cPanelTristan]

It would be really great if anyone having these issues would open a ticket so we have cases to add to the bug report I've yet to have any tickets opened after the two forum posts about the issue that were posted recently.

 

[cPanelDon]

Friendly Moderator Note

I have merged two threads that appear to involve the same or very similar symptoms and error messages that are experienced when using FastCGI, via mod_fcgid, and performing a graceful restart of Apache/httpd.

 

[cPanelDon]

The following is a formal (third-party) upstream bug report for Apache-httpd that describes the same issue being discussed in this forums thread: Bug 48949 – fcgid processes never get killed after graceful restart

If you are experiencing the described symptoms using the latest available version of Apache-httpd and if you can reliably reproduce the problem, please post in or contribute to the formal Apache bug report so that it may receive more attention from their upstream developers. When submitting a support request to cPanel referencing this thread please mention case 36547 and then reply in this thread with your ticket ID number so that we can follow-up internally.

 

[UNIXy]

Hi,

From my understanding, there's a case about this bug in cPanel issue tracker. Can someone confirm whether the fix has been pushed to Edge?

Regards

 

[cPanelTristan]

There is a case that I referenced previously (36547), but since people have not opened tickets to link to the case for us to troubleshoot further, there has been no additional activity on the case beyond the upstream bug report. As mentioned in prior posts, please certainly feel free to open a ticket, then reference the ticket here. If you have already opened a ticket, please note the ticket number here.

I am uncertain why the requests made in this thread and another for this issue have not been successful in having tickets opened for us to reproduce the issue to confirm they are all the same.

 

[trhosting.net]

Hello Don,

I've submitted a ticket.
Ticket ID: 1363434

Thank you

 

[sdanf]

Is this still an issue or has it been fixed? I recently recompiled apache with mod_fcgid support using easyapache and see these same errors.

 

[cPanelTristan]

Please post the exact error message being received. Please quote the ticket number you've opened asking us to investigate. Thanks!

 

[sdanf]

Thanks, I just opened a ticket today #1599134

Here are the errors Im getting after fcgi was added. I removed client IP's from this post but included them in the support ticket.

[Wed Jun 08 15:00:23 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds
[Wed Jun 08 15:00:23 2011] [error] [client ] Premature end of script headers: index.php
[Wed Jun 08 15:00:23 2011] [warn] mod_fcgid: process 22383 graceful kill fail, sending SIGKILL
[Wed Jun 08 15:00:23 2011] [warn] mod_fcgid: process 22382 graceful kill fail, sending SIGKILL
sh: checksuexec: command not found
sh: checksuexec: command not found
[Wed Jun 08 15:00:31 2011] [warn] mod_fcgid: process 22397 graceful kill fail, sending SIGKILL
sh: checksuexec: command not found
[Wed Jun 08 15:00:35 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds, referer: http://www.azmusicconnect.com/feed/
[Wed Jun 08 15:00:36 2011] [error] [client ] Premature end of script headers: index.php, referer: http://www.azmusicconnect.com/feed/
sh: checksuexec: command not found
sh: checksuexec: command not found
[Wed Jun 08 15:00:47 2011] [warn] mod_fcgid: process 22193 graceful kill fail, sending SIGKILL
[Wed Jun 08 15:01:08 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds
[Wed Jun 08 15:01:08 2011] [error] [client ] Premature end of script headers: wp-cron.php
sh: checksuexec: command not found
[Wed Jun 08 15:04:36 2011] [error] [client ] Invalid method in request \x16\x03\x01
[Wed Jun 08 15:04:36 2011] [error] [client ] File does not exist: /usr/local/apache/htdocs/501.shtml
[Wed Jun 08 15:05:46 2011] [error] [client ] Invalid method in request \x16\x03\x01
[Wed Jun 08 15:05:46 2011] [error] [client ] File does not exist: /usr/local/apache/htdocs/501.shtml
[Wed Jun 08 15:07:10 2011] [notice] Graceful restart requested, doing restart
[Wed Jun 08 15:07:11 2011] [emerg] [client ] (22)Invalid argument: mod_fcgid: can't lock process table in pid 23679
[Wed Jun 08 15:07:12 2011] [notice] Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations
[Wed Jun 08 15:51:21 2011] [notice] caught SIGTERM, shutting down
[Wed Jun 08 15:51:22 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Wed Jun 08 15:51:22 2011] [notice] Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations

I have not noticed any problems with memory yet but I only just did the recompile this morning and came across this thread while searching for the error.

 

[boonjackmedia]

Thanks, I just opened a ticket today #1599134

Here are the errors Im getting after fcgi was added. I removed client IP's from this post but included them in the support ticket.

[Wed Jun 08 15:00:23 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds
[Wed Jun 08 15:00:23 2011] [error] [client ] Premature end of script headers: index.php
[Wed Jun 08 15:00:23 2011] [warn] mod_fcgid: process 22383 graceful kill fail, sending SIGKILL
[Wed Jun 08 15:00:23 2011] [warn] mod_fcgid: process 22382 graceful kill fail, sending SIGKILL
sh: checksuexec: command not found
sh: checksuexec: command not found
[Wed Jun 08 15:00:31 2011] [warn] mod_fcgid: process 22397 graceful kill fail, sending SIGKILL
sh: checksuexec: command not found
[Wed Jun 08 15:00:35 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds, referer: http://www.azmusicconnect.com/feed/
[Wed Jun 08 15:00:36 2011] [error] [client ] Premature end of script headers: index.php, referer: http://www.azmusicconnect.com/feed/
sh: checksuexec: command not found
sh: checksuexec: command not found
[Wed Jun 08 15:00:47 2011] [warn] mod_fcgid: process 22193 graceful kill fail, sending SIGKILL
[Wed Jun 08 15:01:08 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds
[Wed Jun 08 15:01:08 2011] [error] [client ] Premature end of script headers: wp-cron.php
sh: checksuexec: command not found
[Wed Jun 08 15:04:36 2011] [error] [client ] Invalid method in request \x16\x03\x01
[Wed Jun 08 15:04:36 2011] [error] [client ] File does not exist: /usr/local/apache/htdocs/501.shtml
[Wed Jun 08 15:05:46 2011] [error] [client ] Invalid method in request \x16\x03\x01
[Wed Jun 08 15:05:46 2011] [error] [client ] File does not exist: /usr/local/apache/htdocs/501.shtml
[Wed Jun 08 15:07:10 2011] [notice] Graceful restart requested, doing restart
[Wed Jun 08 15:07:11 2011] [emerg] [client ] (22)Invalid argument: mod_fcgid: can't lock process table in pid 23679
[Wed Jun 08 15:07:12 2011] [notice] Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations
[Wed Jun 08 15:51:21 2011] [notice] caught SIGTERM, shutting down
[Wed Jun 08 15:51:22 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Wed Jun 08 15:51:22 2011] [notice] Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations

I have not noticed any problems with memory yet but I only just did the recompile this morning and came across this thread while searching for the error.

I had a similar issue with with mod_fcgid during a Wordpress development job, basically we couldn't upload images, graphics, media via Admin on a clean install.

I noticed you've got a "wp-cron.php" listed in your error log above... check out my post on the WP support site and see if the fix solves your problem: WordPress › Support » Media Library not displaying media

The fix is to change the MaxRequestLen setting for mod_fcgid to a higher value.

In your WHM > Apache Configuration > Include Editor > Post VirtualHost Include > All

We added this (mod_fcgid version 2.35):

<IfModule mod_fcgid.c>
MaxRequestLen 1000000000
</IfModule>

For mod_fcgid version 2.36, add this:

<IfModule mod_fcgid.c>
FcgidMaxRequestLen 1000000000
</IfModule>

If that doesn't work look at your ConfigServer Firewall and/or Mod Security settings, goto their site and search for: Wordpress cron errors. You'll need to add an allow setting in your firewall.

And if that doesn't work, I noticed you said you upgraded to Fastcgi using EasyApache, and you're running Apache mod_fcgid version 2.36... as far as I know cPanel still uses mod_fcgid version 2.35. That alone would definitely cause issues, because Apache mod_fcgid 2.36 is a completely new version including a re-vamped naming convention for the settings. We've looked into upgrading to version 2.36 manually, but it wasn't feasible. Reason being, it's a major part of Apache and the CPanel/WHM system, so there would simply be way too much config work involved - too many tech issues. So we're waiting for cPanel/WHM to configure/support the new upgraded version.

 

[imagize]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

I'm still having this issue with apache/mod_fcgi, I logged a support request with ID: 1905021