Apache graceful restarts and FastCGI - mod_fcgid: can't lock process table in pid

[ahostli]

Hi,

First of all sorry for my english

I was wondering if any of You, using Apache 2.2.X, mod_fcgid and SuExec for handling php requests, experienced any problem after most recent EasyApache update?

After upgrading, Ive noticed, that number of sleeping fcgid processes increased dramaticaly, "eating" RAM and eventualy OS starts killing other services (mysql, bind...) to free up memory. Only after doing hard apache reset all fcgid sleeping processes are killed.
All mod_fcgid configuration was left the same.


In apache error_log:

[Wed Mar 17 09:23:45 2010] [notice] Graceful restart requested, doing restart
[Wed Mar 17 09:23:45 2010] [emerg] mod_fcgid: server is restarted, pid 10345 must exit
[Wed Mar 17 09:23:45 2010] [emerg] (22)Invalid argument: mod_fcgid: can't lock process table in PM, pid 10345
[Wed Mar 17 09:23:45 2010] [emerg] [client 88.216.6.174] (22)Invalid argument: mod_fcgid: can't lock process table in pid 104$
[Wed Mar 17 09:23:45 2010] [emerg] [client 88.216.6.174] (22)Invalid argument: mod_fcgid: can't lock process table in pid 104$
[Wed Mar 17 09:23:45 2010] [emerg] [client 88.216.6.174] (22)Invalid argument: mod_fcgid: can't lock process table in pid 113$

[Wed Mar 17 14:02:57 2010] [notice] Graceful restart requested, doing restart
[Wed Mar 17 14:02:57 2010] [emerg] [client 82.135.207.33] (43)Identifier removed: mod_fcgid: can't get pipe mutex, referer: h$
[Wed Mar 17 14:02:57 2010] [emerg] [client 78.61.82.52] (43)Identifier removed: mod_fcgid: can't get pipe mutex, referer: htt$
[Wed Mar 17 14:02:57 2010] [emerg] [client 91.121.87.87] (22)Invalid argument: mod_fcgid: can't lock process table in pid 170$
[Wed Mar 17 14:02:57 2010] [emerg] [client 91.121.88.99] (22)Invalid argument: mod_fcgid: can't lock process table in pid 160$
[Wed Mar 17 14:02:57 2010] [emerg] mod_fcgid: server is restarted, pid 27466 must exit
[Wed Mar 17 14:02:57 2010] [emerg] (22)Invalid argument: mod_fcgid: can't lock process table in PM, pid 27466

I will try to debug the problem, but perhaps someone already experiencied the same problem and was able to find the solution to it?

[linuxtuts]

I have the same problem ...

[Sat Mar 20 02:08:00 2010] [emerg] [client 88.11.171.140] (22)Invalid argument: mod_fcgid: can't lock process table in pid 10635, referer: http://www.mysite.net/
[Sat Mar 20 02:08:01 2010] [emerg] [client 88.11.171.140] (22)Invalid argument: mod_fcgid: can't lock process table in pid 10633, referer: http://www.mysite.net/
[Sat Mar 20 02:08:01 2010] [emerg] [client 88.11.171.140] (22)Invalid argument: mod_fcgid: can't lock process table in pid 13310, referer: http://www.mysite.net/
[Sat Mar 20 02:08:01 2010] [emerg] [client 88.11.171.140] (22)Invalid argument: mod_fcgid: can't lock process table in pid 13324, referer: http://www.mysite.net/

Any thought is to how to fix this?

Thanks in advanced.

[ahostli]

It seems, that any fcgid processes that were active/sleeping before executing graceful apache httpd restart, never gets killed after such restart. They get killed only after hard apache restart. This problem causes fcgid processes to „pile up“after each graceful restart, therefore consuming memory and eventually making the whole system unusable.

[linuxtuts]

Hum, I never even checked the memory level until you just mentioned it, and you are right.
Out of 2GB of RAM I had 232MB left. ( Thank God it was enough to reboot )

But I also use a "System Integrity Monitor" script from

http://www.rfxn.com/projects/system-integrity-monitor/

which does a GREAT job along with cPanel's built in functions so I never have to worry about the sever services not running or the system overloading.

Looks like I'm gonna have to find that old script I used when I had PLESK to let me know when I was outta memory, and automatically reboot when needed. ( I hate that, for-real )

No one more familiar with mod_fcgid and SuExec having these problems?

Thanks,

[wizzy420]

Same problem here, been going on for months. Had to write a script to test for it and manually kill off PHP processes.

PITA

[Mo9a7i]

any update on this one guys? if there's any patch or work around,
i've been facing this for a month and my clients are being annoyed.
I'm using a combo of apache+nginx+fcgi and this error occurs whenever we see a 502 nginx bad gateway message.


wizzy, can you share your code.
Maybe I can customize it to help me a bit if there's no patch yet.

[cPanelTristan]

Anyone still experiencing this issue, please open a ticket. We had a patch for FCGI issues related to the graceful restarts based on a case with development (36547), so if that patch hasn't resolved any issues with the graceful restarts, we need tickets and systems to check into the issue and see it happening. Please try not to correct the issue on the system when opening the ticket so we can see the system in the state of high memory usage.

You can open a ticket using the link in my signature or in WHM > Support Center > Contact cPanel

Thanks!

[LeGastronome]

Hello,

Since fews days i have this type of warn and error in the apache's error log :

Sometimes server show a 500 internal error page. It seems to come from mod_fcgid

[Sat Oct 30 00:38:12 2010] [warn] [client 86.201.143.192] (104)Connection reset by peer: mod_fcgid: can't get data from http client, referer: http:

or

[Fri Oct 29 13:05:41 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Fri Oct 29 13:05:42 2010] [notice] Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.5 configured -- resuming normal operations
[Fri Oct 29 13:05:48 2010] [notice] SIGUSR1 received.  Doing graceful restart
[Fri Oct 29 13:05:50 2010] [emerg] [client 66.249.65.104] (22)Invalid argument: mod_fcgid: can't lock process table in pid 1332
[Fri Oct 29 13:05:52 2010] [warn] [client 77.202.105.147] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server
[Fri Oct 29 13:05:52 2010] [error] [client 77.202.105.147] Premature end of script headers: index.php
[Fri Oct 29 13:05:52 2010] [error] [client 77.202.105.147] File does not exist: /home/**********
[Fri Oct 29 13:05:53 2010] [notice] Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.5 configured -- resuming normal operations
[Fri Oct 29 13:05:53 2010] [emerg] [client 77.202.105.147] (43)Identifier removed: mod_fcgid: can't lock process table in pid 32743
[Fri Oct 29 13:07:10 2010] [notice] caught SIGTERM, shutting down
[Fri Oct 29 13:07:56 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Fri Oct 29 13:07:57 2010] [notice] Apache/2.2.16 (Unix) mod_ssl/2.2.16 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.5 configured -- resuming normal operations

Thanks,

[cPanelTristan]

Please submit a ticket using the link in my signature or using WHM > Support Center > Contact cPanel area. We have another forum post about this same issue:

Apache mod_fcgid problems after recent update of Apache 2.2.15 - cPanel Forums

We also have a prior bug case (36547) about similar issues with the Apache graceful restarts when using fcgi. We need servers to investigate the error to see what is happening to cause this "Invalid argument: mod_fcgid: can't lock process table in pid #" type of error.

[cPanelTristan]

It would be really great if anyone having these issues would open a ticket so we have cases to add to the bug report I've yet to have any tickets opened after the two forum posts about the issue that were posted recently.

[cPanelDon]

I have merged two threads that appear to involve the same or very similar symptoms and error messages that are experienced when using FastCGI, via mod_fcgid, and performing a graceful restart of Apache/httpd.

[cPanelDon]

The following is a formal (third-party) upstream bug report for Apache-httpd that describes the same issue being discussed in this forums thread: Bug 48949 – fcgid processes never get killed after graceful restart

If you are experiencing the described symptoms using the latest available version of Apache-httpd and if you can reliably reproduce the problem, please post in or contribute to the formal Apache bug report so that it may receive more attention from their upstream developers. When submitting a support request to cPanel referencing this thread please mention case 36547 and then reply in this thread with your ticket ID number so that we can follow-up internally.

[UNIXy]

Hi,

From my understanding, there's a case about this bug in cPanel issue tracker. Can someone confirm whether the fix has been pushed to Edge?

Regards

[cPanelTristan]

There is a case that I referenced previously (36547), but since people have not opened tickets to link to the case for us to troubleshoot further, there has been no additional activity on the case beyond the upstream bug report. As mentioned in prior posts, please certainly feel free to open a ticket, then reference the ticket here. If you have already opened a ticket, please note the ticket number here.

I am uncertain why the requests made in this thread and another for this issue have not been successful in having tickets opened for us to reproduce the issue to confirm they are all the same.

[trhosting.net]

Hello Don,

I've submitted a ticket.
Ticket ID: 1363434

Thank you