Apache graceful restarts and FastCGI - mod_fcgid: can't lock process table in pid

[sdanf]

Is this still an issue or has it been fixed? I recently recompiled apache with mod_fcgid support using easyapache and see these same errors.

[cPanelTristan]

Please post the exact error message being received. Please quote the ticket number you've opened asking us to investigate. Thanks!

[sdanf]

Thanks, I just opened a ticket today #1599134

Here are the errors Im getting after fcgi was added. I removed client IP's from this post but included them in the support ticket.

[Wed Jun 08 15:00:23 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds
[Wed Jun 08 15:00:23 2011] [error] [client ] Premature end of script headers: index.php
[Wed Jun 08 15:00:23 2011] [warn] mod_fcgid: process 22383 graceful kill fail, sending SIGKILL
[Wed Jun 08 15:00:23 2011] [warn] mod_fcgid: process 22382 graceful kill fail, sending SIGKILL
sh: checksuexec: command not found
sh: checksuexec: command not found
[Wed Jun 08 15:00:31 2011] [warn] mod_fcgid: process 22397 graceful kill fail, sending SIGKILL
sh: checksuexec: command not found
[Wed Jun 08 15:00:35 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds, referer: http://www.azmusicconnect.com/feed/
[Wed Jun 08 15:00:36 2011] [error] [client ] Premature end of script headers: index.php, referer: http://www.azmusicconnect.com/feed/
sh: checksuexec: command not found
sh: checksuexec: command not found
[Wed Jun 08 15:00:47 2011] [warn] mod_fcgid: process 22193 graceful kill fail, sending SIGKILL
[Wed Jun 08 15:01:08 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds
[Wed Jun 08 15:01:08 2011] [error] [client ] Premature end of script headers: wp-cron.php
sh: checksuexec: command not found
[Wed Jun 08 15:04:36 2011] [error] [client ] Invalid method in request \x16\x03\x01
[Wed Jun 08 15:04:36 2011] [error] [client ] File does not exist: /usr/local/apache/htdocs/501.shtml
[Wed Jun 08 15:05:46 2011] [error] [client ] Invalid method in request \x16\x03\x01
[Wed Jun 08 15:05:46 2011] [error] [client ] File does not exist: /usr/local/apache/htdocs/501.shtml
[Wed Jun 08 15:07:10 2011] [notice] Graceful restart requested, doing restart
[Wed Jun 08 15:07:11 2011] [emerg] [client ] (22)Invalid argument: mod_fcgid: can't lock process table in pid 23679
[Wed Jun 08 15:07:12 2011] [notice] Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations
[Wed Jun 08 15:51:21 2011] [notice] caught SIGTERM, shutting down
[Wed Jun 08 15:51:22 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Wed Jun 08 15:51:22 2011] [notice] Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations

I have not noticed any problems with memory yet but I only just did the recompile this morning and came across this thread while searching for the error.

[boonjackmedia]

Thanks, I just opened a ticket today #1599134

Here are the errors Im getting after fcgi was added. I removed client IP's from this post but included them in the support ticket.

[Wed Jun 08 15:00:23 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds
[Wed Jun 08 15:00:23 2011] [error] [client ] Premature end of script headers: index.php
[Wed Jun 08 15:00:23 2011] [warn] mod_fcgid: process 22383 graceful kill fail, sending SIGKILL
[Wed Jun 08 15:00:23 2011] [warn] mod_fcgid: process 22382 graceful kill fail, sending SIGKILL
sh: checksuexec: command not found
sh: checksuexec: command not found
[Wed Jun 08 15:00:31 2011] [warn] mod_fcgid: process 22397 graceful kill fail, sending SIGKILL
sh: checksuexec: command not found
[Wed Jun 08 15:00:35 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds, referer: http://www.azmusicconnect.com/feed/
[Wed Jun 08 15:00:36 2011] [error] [client ] Premature end of script headers: index.php, referer: http://www.azmusicconnect.com/feed/
sh: checksuexec: command not found
sh: checksuexec: command not found
[Wed Jun 08 15:00:47 2011] [warn] mod_fcgid: process 22193 graceful kill fail, sending SIGKILL
[Wed Jun 08 15:01:08 2011] [warn] [client ] mod_fcgid: read data timeout in 40 seconds
[Wed Jun 08 15:01:08 2011] [error] [client ] Premature end of script headers: wp-cron.php
sh: checksuexec: command not found
[Wed Jun 08 15:04:36 2011] [error] [client ] Invalid method in request \x16\x03\x01
[Wed Jun 08 15:04:36 2011] [error] [client ] File does not exist: /usr/local/apache/htdocs/501.shtml
[Wed Jun 08 15:05:46 2011] [error] [client ] Invalid method in request \x16\x03\x01
[Wed Jun 08 15:05:46 2011] [error] [client ] File does not exist: /usr/local/apache/htdocs/501.shtml
[Wed Jun 08 15:07:10 2011] [notice] Graceful restart requested, doing restart
[Wed Jun 08 15:07:11 2011] [emerg] [client ] (22)Invalid argument: mod_fcgid: can't lock process table in pid 23679
[Wed Jun 08 15:07:12 2011] [notice] Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations
[Wed Jun 08 15:51:21 2011] [notice] caught SIGTERM, shutting down
[Wed Jun 08 15:51:22 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Wed Jun 08 15:51:22 2011] [notice] Apache/2.2.19 (Unix) mod_ssl/2.2.19 OpenSSL/0.9.8e-fips-rhel5 mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations

I have not noticed any problems with memory yet but I only just did the recompile this morning and came across this thread while searching for the error.

I had a similar issue with with mod_fcgid during a Wordpress development job, basically we couldn't upload images, graphics, media via Admin on a clean install.

I noticed you've got a "wp-cron.php" listed in your error log above... check out my post on the WP support site and see if the fix solves your problem: WordPress › Support » Media Library not displaying media

The fix is to change the MaxRequestLen setting for mod_fcgid to a higher value.

In your WHM > Apache Configuration > Include Editor > Post VirtualHost Include > All

We added this (mod_fcgid version 2.35):

<IfModule mod_fcgid.c>
MaxRequestLen 1000000000
</IfModule>

For mod_fcgid version 2.36, add this:

<IfModule mod_fcgid.c>
FcgidMaxRequestLen 1000000000
</IfModule>

If that doesn't work look at your ConfigServer Firewall and/or Mod Security settings, goto their site and search for: Wordpress cron errors. You'll need to add an allow setting in your firewall.

And if that doesn't work, I noticed you said you upgraded to Fastcgi using EasyApache, and you're running Apache mod_fcgid version 2.36... as far as I know cPanel still uses mod_fcgid version 2.35. That alone would definitely cause issues, because Apache mod_fcgid 2.36 is a completely new version including a re-vamped naming convention for the settings. We've looked into upgrading to version 2.36 manually, but it wasn't feasible. Reason being, it's a major part of Apache and the CPanel/WHM system, so there would simply be way too much config work involved - too many tech issues. So we're waiting for cPanel/WHM to configure/support the new upgraded version.

[imagize]

I'm still having this issue with apache/mod_fcgi, I logged a support request with ID: 1905021

[ikillbill]

someone has posted a fix on linux server

https://issues.apache.org/bugzilla/s...gi?id=50309#c6

hopefully cpanel team would work out a temp fix

thanks

[webmastergreg]

spotted this issue too for me:
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18714 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18713 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18712 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18701 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18696 graceful kill fail, sending SIGKILL
[Fri Dec 02 21:55:42 2011] [warn] mod_fcgid: process 18692 graceful kill fail, sending SIGKILL

So much more like this, and it append erratically.

Server load is about between 0.20 - 0.70 for a day and suddenly jump to 30.0 to 40.0 without any reason.
But each time the netstat show me that a parent php process still spawn child process that are all at zero access.
And it seems like my conf as no effect at all at this moment despite of:
<IfModule mod_fcgid.c>
FcgidIdleScanInterval 1
FcgidErrorScanInterval 3
FcgidBusyScanInterval 120
FcgidBusyTimeout 240
FcgidProcessLifeTime 10
FcgidMaxRequestsPerProcess 500
FcgidMaxProcesses 45
FcgidConnectTimeout 240
FcgidIOTimeout 240
FcgidIdleTimeout 240
FcgidMaxRequestLen 15728640
</IfModule>

Really hope that this fix will be implemented soon, and hope it will solve my issue too.

[ikillbill]

any progress on this please?

[sreekumar]

I too faced the problem

[Mon Jan 16 10:28:21 2012] [warn] mod_fcgid: process 1448 graceful kill fail, sending SIGKILL
[Mon Jan 16 10:34:45 2012] [warn] mod_fcgid: process 2132 graceful kill fail, sending SIGKILL
[Mon Jan 16 10:34:45 2012] [warn] mod_fcgid: process 1808 graceful kill fail, sending SIGKILL
[Mon Jan 16 10:34:45 2012] [warn] mod_fcgid: process 2084 graceful kill fail, sending SIGKILL
[Mon Jan 16 10:34:45 2012] [warn] mod_fcgid: process 1376 graceful kill fail, sending SIGKILL

Giving sufficiently large values for the below two configuration settings seems to have solved it

FcgidProcessLifeTime 3600
FcgidIdleTimeout 3600

Note that the values for the time period is in seconds.

[ikillbill]

will 3600 seems too high and increase server resources usages?

and it looks like fastcgi is part of PHP 5.3, hopefully there will be at least a temp fix ~

[johanee]

We have the same problem with PHP processes sticking around after graceful restart.

We do not usually get any error messages though the old PHP processes always get orphaned and stick around forever.

Ticket: 2323218

[InterServed]

We have the same problem with PHP processes sticking around after graceful restart.

We do not usually get any error messages though the old PHP processes always get orphaned and stick around forever.

Ticket: 2323218

I can also confirm this, please keep us updated if you get any fix from cPanel via the ticket you submitted.

Tho i also noticed that cPanel does patch mod_fcgid with a patch that should fix the problem we reported:

/home/cpeasyapache/src/mod_fcgid_patches/0002-graceful_reaper.patch

[cPanelTristan]

We have a few internal cases to rework the patch that went into EasyApache 3.9.1 for the previously reported FCGI + Apache graceful restart issues. The main case 57097 concerned the issue with orphaned processes not being stopped during the EasyApache restart and it is being investigated for EasyApache 3.10

Until the issue has been corrected with the existing patch for these orphaned processes, the following suggestions are possible ways to reduce the issue:

1) Change to WHM > Apache Configuration > Piped Logging to reduce the number of Apache graceful restarts (most graceful restarts are performed for domlog rotations)
2) Create a nightly cron to hard restart Apache so the processes will be stopped

[KevinKings]

Submitted ticket: 2329090

Server Version: Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/0.9.8e-fips-rhel5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_fcgid/2.3.6
Server Built: Feb 19 2012 15:42:26

Seeing similar symptoms of unbounded PHP orphaned process growth until the server runs out of memory or hangs.

Here's the process count chart for the past few days. The down spikes are when the server reboots or apache/php are force killed.

[X-Josh]

I am also having this problem over the past 4-5 days - a bunch of PHP processes seem to run indefinitely until they are manually killed, or the httpd service is restarted.

PHP:

PHP 5.2.17 (cli) (built: Feb 23 2012 21:51:21)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies
    with the ionCube PHP Loader v4.0.12, Copyright (c) 2002-2011, by ionCube Ltd    ., and
    with Zend Optimizer v3.3.9, Copyright (c) 1998-2009, by Zend Technologies
    with Suhosin v0.9.33, Copyright (c) 2007-2012, by SektionEins GmbH

Apache:

Server version: Apache/2.2.22 (Unix)
Server built:   Feb 23 2012 21:48:01
Cpanel::Easy::Apache v3.9.2 rev9999

I've got a few errors like this showing up:

[Fri Feb 24 07:15:17 2012] [notice] Graceful restart requested, doing restart
[Fri Feb 24 07:15:20 2012] [notice] SSL FIPS mode disabled
[Fri Feb 24 07:15:20 2012] [notice] mod_bw : Memory Allocated 0 bytes (each conf takes 48 bytes)
[Fri Feb 24 07:15:20 2012] [notice] mod_bw : Version 0.92 - Initialized [0 Confs]
[Fri Feb 24 07:15:20 2012] [notice] Apache/2.2.22 (Unix) mod_ssl/2.2.22 OpenSSL/1.0.0-fips mod_bwlimited/1.4 mod_fcgid/2.3.6 configured -- resuming normal operations

There are also occasional warnings about suhosin not supporting PHP 5.2 and below anymore, but I don't think this is the issue, as I've also used PHP 5.3.10 and the same problems occur (same configuration, other than version).

I was using the same configuration on the previous install, which I setup in January - that worked perfectly. I set this up again last Monday, and since then, there have been these repeated Apache errors. I did switch to SuPHP for some time, but I did not do a hard restart on the httpd service (from what I read, it seems that EasyApache doesn't kill existing processes), so I am switching back now and doing a service restart to see if it will make a difference.

ldf (CSF) is reporting many errors like this:

Time:         Fri Feb 24 08:24:39 2012 +0000
Account:      discove8
Resource:     Process Time
Exceeded:     18090 > 1800 (seconds)
Executable:   /usr/bin/php
Command Line: /usr/bin/php
PID:          24928
Killed:       No

Did anyone manage to fix this problem?

Thanks.