Apache graceful restarts and FastCGI - mod_fcgid: can't lock process table in pid

Quemeros

Member
Mar 4, 2012
16
0
51
cPanel Access Level
Root Administrator
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

That line really works! It killed all of them.

I'm sorry, but I'm pretty new to this. How would I set up that command as a cron job?

Thanks!
Code:
crontab -e
to access to edit crontab. Do as root ( su - ).
Then, add this line:
Code:
*/5 * * * * ps auxwwwf | grep '[0-9] /usr/bin/php' | awk '{ print $2 }' | xargs kill -9 >/dev/null 2>&1
Where "*/5 * * * *" means every 5 minutes, and ">/dev/null 2>&1" send respond information (that command do if don't find any zombie php process) to null; and you don't recive any mail.

Bye :)
 

wheelerc

Member
Mar 21, 2012
7
0
51
cPanel Access Level
Root Administrator
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

brainyron - That works perfectly for me, thanks.

I'd just switched from SuPHP to mod_fcgid and couldn't believe the memory usage (a fairly busy server with around 140 accounts and 8GB RAM went from 15% to 80% memory usage and 5% to 50% swap usage over a few hours).

After a lot of searching I found this thread, so thanks to all who have contributed.

Server configuration is as follows:

CENTOS 5.8 x86_64
Apache 2.2.2
PHP 5.3.10
mod_fcgi 2.3.6
APC 3.1.9

I also found that cPanel (I assume) had put the FcgidMaxRequestLen in pre_main_global.conf and it han no effect until I moved it to post_virtualhost_global.conf
 

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
40
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

I'd just switched from SuPHP to mod_fcgid and couldn't believe the memory usage (a fairly busy server with around 140 accounts and 8GB RAM went from 15% to 80% memory usage and 5% to 50% swap usage over a few hours).
The best setup for fastest processing and lower memory usage would be to use DSO with mod_ruid2 rather than mod_fcgid for PHP. mod_ruid2 will not only run processes for PHP as the user, it will run Apache processes as the user for html and other pages. Everything under Apache that runs on the account will run as the user. DSO doesn't have the issues and configuration requirements that FCGI does.

Additionally, if you add RDocumentChRoot for each account, you can lock users into their /home/username or /home/username/public_html from reading files outside of their account all requests.
 

rackaid

Well-Known Member
Jan 18, 2003
89
28
168
Jacksonville, FL
cPanel Access Level
DataCenter Provider
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

This issue is becoming critical for many cPanel customers. Can cPanel recommend a workaround until they have a solution. This has been reported as fixed in upstream versions of Apache.

I have seen the following recommendations:

  1. Switched to piped logs
  2. Hard restart apache via cron

However, this does not address the problem of orphaned PHP processes already in existence.

Suggestion:
In our testing, we find on Apache 2.0 w/prefork MPM, the problem only exists with graceful restarts. Full restarts do not cause this.

Since cPanel supplies the restart scripts for Apache, can you supply one that will do a forceful restart always?

****UPDATE****
We just rebuilt Apache with the same profile as 2 days ago. But this time we cleared out the easyapache src tree to force a new download. This rebuild does not appear to suffer from the FastCGI issues.

With the prior build a graceful restart would not terminate fast-cgi processes. Now it does.


We need to let the system run for a bit more but this may have resolved it.

If we switch out the prior mod_fastcgi.so with the current one, the problem returns.

So you may want to try clearing easyapache src directory prior to any attempts to fix this issue.


****UPDATE 2****
Appears cPanel may have pushed a new EA build during our work. Try to rebuild your profile but make sure you have the latest EA by clearing out the src directory.
 
Last edited:

ikillbill

Well-Known Member
Feb 18, 2008
119
0
66
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

just applied EasyApache 3.11.2 , apache restart still does not kill orhpan PHP process.....
when this will be formally fixed please?
 

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
40
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Hello,

Please reply to any open ticket or open up a ticket if any patch appears not to fix the issue. The only way to ensure development is aware if a specific patch doesn't function fully for resolving an issue would be to ensure the bug report ticket is the one used for reporting purposes.

Thanks!
 

ikillbill

Well-Known Member
Feb 18, 2008
119
0
66
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

bug report

#2464286

still NOT fixed :(

thanks
 

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
40
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Please try to update using the latest EasyApache to see the results. There were fixes placed into the most recent builds and some users have indicated that the issue appears resolved. If it is not resolved with the latest EasyApache, please submit a ticket because there is no way to investigate why the issue would still be occurring in your environment via the forums.
 

Quemeros

Member
Mar 4, 2012
16
0
51
cPanel Access Level
Root Administrator
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Old server got the problem... new server, with a fresh install, newest EasyApache, same result's... i don't stand where is the problem, but i will go to open another ticket.
 

complexi

Registered
Mar 3, 2012
2
0
51
cPanel Access Level
Website Owner
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

@Quemeros

Just an FYI:

This suggestion by brainyron currently works for us:

*/5 * * * * pkill -9 -P 1 php

Not ideal as we don't want to depend on cron jobs to clean up orphaned php process.

Like your last post read, we'd rather have cPanel fix the bug.

complexi
 

Quemeros

Member
Mar 4, 2012
16
0
51
cPanel Access Level
Root Administrator
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

@Complexi Thanks! I was use that exact command, is the best for the moment.

cPanel parcialy fix the problem if you have Log Rotation Log's active. I confirmed, but you CAN'T restart apache, because the process generated.
 

Avanti

Registered
Nov 27, 2007
4
0
51
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

We have used the below suggestion as a workaround and can confirm that our servers have been stable since this work around and speed seems comparable to FCGI to us.

We are so happy with this set-up that this is our new standard set-up. Configuration is also a lot simpler than FCGI. We didn't need to add any configuration settings into the Apache configuration editor.

For us this has been an all-round winner.

The best setup for fastest processing and lower memory usage would be to use DSO with mod_ruid2 rather than mod_fcgid for PHP. mod_ruid2 will not only run processes for PHP as the user, it will run Apache processes as the user for html and other pages. Everything under Apache that runs on the account will run as the user. DSO doesn't have the issues and configuration requirements that FCGI does.
 

ikillbill

Well-Known Member
Feb 18, 2008
119
0
66
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

just applied latest easyapache again, still not working
new ticket is issued again
#2511323
 
Last edited:

mofolo

Member
Jan 26, 2012
18
0
51
cPanel Access Level
Website Owner
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

Hi,

Process kill script is not a perfect solution.

I recomend to patch and replace current mod_fcgid with new one.

Bug: https://issues.apache.org/bugzilla/show_bug.cgi?id=50309

Fix : https://issues.apache.org/bugzilla/attachment.cgi?id=27982&action=diff

Feel free to download and change source code of fcgi module. After - install it to your apache ( replace existing ).
You will need to reinstall it each time after apache rebuild via EasyApache.

My patched module is attached ( mod_fcgid-2.3.6 )
For me this bug is fully fixed. View attachment 9341



Good luck ! Have a nice day ))

Confirmed to be a working fix.

This is what you have to do:

1. Download mod_fcgid from the link above.

2. Extract the .tar and use something like SSH/FTP to transfer the directory in your Apache root directory
For me running CentOS 5 it was /etc/httpd/

3. Use SSH and direct it to the folder you have just uploaded
For me it was.
Code:
# cd /etc/httpd/mod_fcgid-2.3.6/
4. Run the configuration file

Code:
# ./configure.apxs
5. Make and install

Code:
# make
Code:
# make install
6. Restart Apache
Code:
# /etc/init.d/httpd restart

This problem hasn't bothered me since (it's been 24 hours and i was having this problem every 2-3 hours).

I hope this helps someone else. I was pulling out my hair cause of this.
 

cPanelTristan

Quality Assurance Analyst
Staff member
Oct 2, 2010
7,607
40
248
somewhere over the rainbow
cPanel Access Level
Root Administrator
Re: Apache graceful restarts and FastCGI - mod_fcgid: can't lock process ta

You can try mofolo's solution, but any module you build using apxs that adds to /usr/local/apache/modules/ folder needs to be backed up to /root folder and then re-added at the end of any EasyApache you run. If you don't have a /usr/local/cpanel/scripts/posteasyapache script where you copy the module back into /usr/local/apache/modules/ folder, your EasyApache recompile will load the other fcgi module that we'll replace it with.