Apache graceful restarts and FastCGI - mod_fcgid: can't lock process table in pid

[ikillbill]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

someone has posted a fix on linux server

https://issues.apache.org/bugzilla/show_bug.cgi?id=50309#c6

hopefully cpanel team would work out a temp fix

thanks

 

[webmastergreg]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

spotted this issue too for me:
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18714 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18713 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18712 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18701 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18696 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18692 graceful kill fail, sending SIGKILL

So much more like this, and it append erratically.

Server load is about between 0.20 - 0.70 for a day and suddenly jump to 30.0 to 40.0 without any reason.
But each time the netstat show me that a parent php process still spawn child process that are all at zero access.
And it seems like my conf as no effect at all at this moment despite of:
<IfModule mod_fcgid.c>
FcgidIdleScanInterval 1
FcgidErrorScanInterval 3
FcgidBusyScanInterval 120
FcgidBusyTimeout 240
FcgidProcessLifeTime 10
FcgidMaxRequestsPerProcess 500
FcgidMaxProcesses 45
FcgidConnectTimeout 240
FcgidIOTimeout 240
FcgidIdleTimeout 240
FcgidMaxRequestLen 15728640
</IfModule>

Really hope that this fix will be implemented soon, and hope it will solve my issue too.

 

[ikillbill]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

any progress on this please?

 

[sreekumar]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

I too faced the problem

[Mon Jan 16 10:28:21 2012] [warn] mod_fcgid: process 1448 graceful kill fail, sending SIGKILL
[Mon Jan 16 10:34:45 2012] [warn] mod_fcgid: process 2132 graceful kill fail, sending SIGKILL
[Mon Jan 16 10:34:45 2012] [warn] mod_fcgid: process 1808 graceful kill fail, sending SIGKILL
[Mon Jan 16 10:34:45 2012] [warn] mod_fcgid: process 2084 graceful kill fail, sending SIGKILL
[Mon Jan 16 10:34:45 2012] [warn] mod_fcgid: process 1376 graceful kill fail, sending SIGKILL

Giving sufficiently large values for the below two configuration settings seems to have solved it

FcgidProcessLifeTime 3600
FcgidIdleTimeout 3600

Note that the values for the time period is in seconds.

 

[ikillbill]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

will 3600 seems too high and increase server resources usages?

and it looks like fastcgi is part of PHP 5.3, hopefully there will be at least a temp fix ~

 

[johanee]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

We have the same problem with PHP processes sticking around after graceful restart.

We do not usually get any error messages though the old PHP processes always get orphaned and stick around forever.

Ticket: 2323218

 

[InterServed]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

We have the same problem with PHP processes sticking around after graceful restart.

We do not usually get any error messages though the old PHP processes always get orphaned and stick around forever.

Ticket: 2323218

I can also confirm this, please keep us updated if you get any fix from cPanel via the ticket you submitted.

Tho i also noticed that cPanel does patch mod_fcgid with a patch that should fix the problem we reported:

/home/cpeasyapache/src/mod_fcgid_patches/0002-graceful_reaper.patch

 

[cPanelTristan]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

We have a few internal cases to rework the patch that went into EasyApache 3.9.1 for the previously reported FCGI + Apache graceful restart issues. The main case 57097 concerned the issue with orphaned processes not being stopped during the EasyApache restart and it is being investigated for EasyApache 3.10

Until the issue has been corrected with the existing patch for these orphaned processes, the following suggestions are possible ways to reduce the issue:

1) Change to WHM > Apache Configuration > Piped Logging to reduce the number of Apache graceful restarts (most graceful restarts are performed for domlog rotations)
2) Create a nightly cron to hard restart Apache so the processes will be stopped

 

[KevinKings]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Submitted ticket: 2329090

Server Version: Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_fcgid/2.3.6
Server Built: Feb 19 2012 15:42:26

Seeing similar symptoms of unbounded PHP orphaned process growth until the server runs out of memory or hangs.

Here's the process count chart for the past few days. The down spikes are when the server reboots or apache/php are force killed.

 

[X-Josh]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

I am also having this problem over the past 4-5 days - a bunch of PHP processes seem to run indefinitely until they are manually killed, or the httpd service is restarted.

PHP:

PHP 5.2.17 (cli) (built: Feb 23 2012 21:51:21)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
    with the ionCube PHP Loader v4.0.12, Copyright (c) 2002-2011, by ionCube Ltd    ., and
    with Zend Optimizer v3.3.9, Copyright (c) 1998-2009, by Zend Technologies
    with Suhosin v0.9.33, Copyright (c) 2007-2012, by SektionEins GmbH

Apache:

Server version: Apache/2.2.22 (Unix)
Server built:   Feb 23 2012 21:48:01
Cpanel::Easy::Apache v3.9.2 rev9999

I've got a few errors like this showing up:

[Fri Feb 24 07:15:17 2012] [notice] Graceful restart requested, doing restart
[Fri Feb 24 07:15:20 2012] [notice] SSL FIPS mode disabled
[Fri Feb 24 07:15:20 2012] [notice] mod_bw : Memory Allocated 0 bytes (each conf takes 48 bytes)
[Fri Feb 24 07:15:20 2012] [notice] mod_bw : Version 0.92 - Initialized [0 Confs]
[Fri Feb 24 07:15:20 2012] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/1.0.0-fips mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations

There are also occasional warnings about suhosin not supporting PHP 5.2 and below anymore, but I don't think this is the issue, as I've also used PHP 5.3.10 and the same problems occur (same configuration, other than version).

I was using the same configuration on the previous install, which I setup in January - that worked perfectly. I set this up again last Monday, and since then, there have been these repeated Apache errors. I did switch to SuPHP for some time, but I did not do a hard restart on the httpd service (from what I read, it seems that EasyApache doesn't kill existing processes), so I am switching back now and doing a service restart to see if it will make a difference.

ldf (CSF) is reporting many errors like this:

Time:         Fri Feb 24 08:24:39 2012 +0000
Account:      discove8
Resource:     Process Time
Exceeded:     18090 > 1800 (seconds)
Executable:   /usr/bin/php
Command Line: /usr/bin/php
PID:          24928
Killed:       No

Did anyone manage to fix this problem?

Thanks.

 

[cPanelMichael]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Hello

We do have an internal case open in regards to the "can't lock process table" messages in the log after a graceful restart when using FastCGI. For reference, this case number is 57619. Please note that we encourage users to submit bug reports using the following URL:

Submit A Bug Report

The more reports we receive on the same issue, the more likely it is the issue will be addressed sooner. It also allows us to confirm your specific issue you are experiencing is related to the existing case.

Thank you.

 

[KevinKings]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Here's the reply to my support ticket I filed. See below for the current workaround I'm using.

Hello,

This is a currently known bug with fcgid.

Essentially, this is a bug with mod_fcgid (not cPanel specific code) that causes this problem. It relates to mod_fcgid and its inability to properly maintain the the fcgi process table during graceful restarts. This causes undesired behavior; one such behavior being orphaned fcgi processes that don't end as expected or within any reasonably expected time.

Per that case, this bug has been communicated upstream to the developers of mod_fcgid itself. They seem slow to produce any fixes, and our developers have been investigating the possibility of perhaps deploying a patch our own to resolve the problem with mod_fcgid. Our internal case 57619 is to address this.

I wouldn't have any sort of ETA available for you on this, but we are aware of the problem and working to investigate/resolve it so that a patch can be released to the public. Your best bet is to continue to watch the EasyApache change logs for 57619 to appear. Once a fix is released to a public version, it will be posted on that page.

Until then, you will need to work around it by occasionally issuing a full restart (instead of a graceful restart) and potentially cleaning up any leftover/orphaned fcgi processes. Alternatively, you could explore using an alternate PHP Handler like mod_suphp. mod_suphp would be the most similar to mod_fcgid in terms of its requirements of user file permissions/ownerships. It should produce the least amount of issues when moving to it from mod_fcgid.

--
Jerry Smith
Technical Analyst I
cPanel Inc.

As per the comment above I've switched to piped logs and also disabled WHM httpd service moniter which also seemed to trigger graceful restarts. I went from 2-3 OS crashes or reboots per day to none yesterday. Workarounds appear to be working.

You can see the process chart has been flat for the past ~ 24hrs.

 

[X-Josh]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

My current workaround is simply to switch to SuPHP from FCGI, and that's worked (at the expense of slightly more memory, it seems, but I have plenty spare now it isn't maxing it out).

I'll file a bug report once I get home tomorrow, and see if we can get it sorted from there.

Thanks.

 

[cypheroz]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

G'Day - new to this forum!

FYI
I run this via a cron job wrapper that captures the output into a logfile every 10 minutes.
I also made sure the log job only runs once a day at around 1:00am
See: http://docs.cpanel.net/twiki/pub/AllDocumentation/ReleaseNotes/cPanellogd.pdf
[a]useracct is the user account 'auseracct' under which the fcgid is running.

#!/bin/bash
#
# Sometimes PHP fast CGI tasks get left hanging around, this cleans them up
#
set -x
echo "List all FCGID processes..."
ps -efl | egrep "[a]useracct.*php"
echo "List any specific zombie FCGID processes..."
ps -efl | egrep "[a]useracct [ ]+?[0-9]*?[ ]+?[1][ ] .*php"
echo "Kill the zombie FCGID processes..."
kill -9 $(ps -efl | egrep "[a]useracct [ ]+?[0-9]*?[ ]+?[1][ ] .*php" | awk '{print $4}')
set +x
# end.

 

[KraKaZabRa]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Hi,

Process kill script is not a perfect solution.

I recomend to patch and replace current mod_fcgid with new one.

Bug: https://issues.apache.org/bugzilla/show_bug.cgi?id=50309

Fix : https://issues.apache.org/bugzilla/attachment.cgi?id=27982&action=diff

Feel free to download and change source code of fcgi module. After - install it to your apache ( replace existing ).
You will need to reinstall it each time after apache rebuild via EasyApache.

My patched module is attached ( mod_fcgid-2.3.6 )
For me this bug is fully fixed. View attachment mod_fcgid-2.3.6.zip



Good luck ! Have a nice day ))

 

[InterServed]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Hi,

Process kill script is not a perfect solution.

I recomend to patch and replace current mod_fcgid with new one.

Bug: https://issues.apache.org/bugzilla/show_bug.cgi?id=50309

Fix : https://issues.apache.org/bugzilla/attachment.cgi?id=27982&action=diff

Feel free to download and change source code of fcgi module. After - install it to your apache ( replace existing ).
You will need to reinstall it each time after apache rebuild via EasyApache.

My patched module is attached ( mod_fcgid-2.3.6 )
For me this bug is fully fixed. View attachment 9341



Good luck ! Have a nice day ))

Bumping this topic up. Expecting cPanel to validate the patch and push it as update.

 

[Del Drago]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

I'm really hoping that this is dealt with soon. I spent the past week dealing with this nightmare, and just ran across this thread yesterday. In the meantime, I've switched to piped logs and will be creating a cron to do automated hard restarts.

 

[Del Drago]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

2) Create a nightly cron to hard restart Apache so the processes will be stopped

Is there a specific way to ensure a "hard restart"?

I created a cron job to restart apache, which looks like this:

0 12 * * * /etc/init.d/httpd restart >/dell/null 2>&1

However, when run it doesn't kill the processes. They just keep going and going, as if they are immortal or something.

Any help would be appreciated. Thanks!

 

[Quemeros]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Is there a specific way to ensure a "hard restart"?

I created a cron job to restart apache, which looks like this:

0 12 * * * /etc/init.d/httpd restart >/dell/null 2>&1

However, when run it doesn't kill the processes. They just keep going and going, as if they are immortal or something.

Any help would be appreciated. Thanks!

Can you try this:

ps auxwwwf | grep '[0-9] /usr/bin/php' | awk '{ print $2 }' | xargs kill -9

Is a simiar problem, that discused here: https://forums.cpanel.net/f189/lost...-2-2-17-mod_fcgid-2-3-5-suexec-180571-p2.html

EDIT: Light appears... we need to wait to EA 3.10 http://forums.cpanel.net/f5/zombie-fcgid-processes-don-t-die-261891.html#post1093911

 

[Del Drago]

Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Can you try this:

ps auxwwwf | grep '[0-9] /usr/bin/php' | awk '{ print $2 }' | xargs kill -9

That line really works! It killed all of them.

I'm sorry, but I'm pretty new to this. How would I set up that command as a cron job?

Thanks!