PHP processes being stalled a few times a day

[Bdzzld]

Hi,

I've got three dual opteron servers, which are all experiencing the same problem. All three servers are running CentOS 4.3 x86_64 with the latest *stable* release of cPanel/WHM. PHP/Apache has been compiled with :

* GD support
* IMAP support
* PHPsuexec support
* Zend Optimizer

A few times a day the PHP module just stops responding (noticeble by PHP processes being stalled). During that time normal html pages are displayed without problems, thus the assumption that it has something to do with the PHP module. The "lock" always lasts for a few seconds and then everything is being served as usual. During such time there's no heavy load at all. What I also noticed is that you're unable to log onto the server via SSH during the "lock". User_id and password are being accepted, but the information is not being processed during the "lock".

I've already created a ticket for it @ cPanel, but after three weeks of testing, cPanel claims it's a problem with the hardware and not related to cPanel not functioning correctly... All three servers are different brands and are almost brand-new, so I doubt that's the case here...

Does anyone have similar problems? And most important... was able to solve it?

Thanking you in advance.

[chirpy]

Odd. Could be a resource leak or even running out of MySQL connections. Things to try:

In httpd.conf set:

KeepAlives 3
MaxRequestsPerChild 50

Then restart httpd. The latter helps with resource leaks.

In /usr/lib/php.ini:

mysql.allow_persistent = Off

Then restart httpd.

In /etc/my.cnf

set-variable = max_connections=750

Then restart MySQL. This shouldn't really be necessary with the httpd.conf and php.ini changes, though.

Final thought, make sure /tmp isn't nearly full incase the problem is with creating session files within it.

[Bdzzld]

Hi Chirpy,

Thanks for trying to help me out...

Odd. Could be a resource leak or even running out of MySQL connections. Things to try:

In httpd.conf set:

KeepAlives 3
MaxRequestsPerChild 50

Then restart httpd. The latter helps with resource leaks.

The following is currently set up :

Timeout 50
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
MinSpareServers 20
MaxSpareServers 35
StartServers 50
MaxClients 512
MaxRequestsPerChild 10000

I could not find "KeepAlives". Is that a typo?
The other options you mentioned were already present/altered...

Thanks.

[chirpy]

Sorry, KeepAlives should be KeepAliveTimeout. As mentioned above, I'd suggest that you set MaxRequestsPerChild down to 50

[Bdzzld]

Hi Chirpy,

Thanks for the suggestions. Unfortunately they did solve the issue...

PHP processes are still stalled on this server a few times a day...
This is also noticable by executing :

root@xxxxx [~]# php -v
PHP 4.4.2 (cli) (built: Mar 26 2006 06:52:46)
Copyright (c) 1997-2006 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
    with Zend Extension Manager v1.0.9, Copyright (c) 2003-2006, by Zend Technologies
    with Zend Optimizer v2.6.2, Copyright (c) 1998-2006, by Zend Technologies

Normally this will show the installed version immediately, but at the time the PHP processes are being stalled, it takes up to a few seconds before that information is displayed.

Server load does not seem to have anything to do with the PHP processes being stalled; it also happens when server load is @ only 0.38.

*) Is it possible one client on the server could be the source of this issue (although nothing out of the ordinary is shown)?
*) Do you have any other suggestions I can try?

Thanks once again.

[Bdzzld]

[Update]
Last night was the worsed of all cases. All three servers in the same DNS cluster did not respond to any PHP request for quite some time at the same(!) time. Restarting httpd on one of these servers did not solve the problem.

I really would like some help on this problem, becuase this is really costing me my sleep (and my customers)...

Thanks to any one who can help...

[brianoz]

Any chance this is some sort of DNS lookup (or reverse DNS lookup) problem?

[MMarko]

KeepAlive On - turn this off!

[Bdzzld]

Any chance this is some sort of DNS lookup (or reverse DNS lookup) problem?

I don't think so... plain HTML sites are serviced without any problems during the problem...

KeepAlive On - turn this off!

That should only be the case if the problem could be determined to "Server load" levels, but that's certainly not the case here... because PHP scripts were not (able to be) serviced "server load" dropped down to .22 levels at most...

[Bdzzld]

[Update 2]

During another stall of PHP scripts on one of the servers I noticed the following in the /etc/httpd/logs/access_log :

killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl

Any idea what this line does and where it comes from (I surely did not execute it)?

Thanks...

[brianoz]

Sure you haven't been compromised somehow? Did you give us the entire, exact line in the log? The effect of that line would be to lock out PHP and Apache for a few seconds, definitely!

[Bdzzld]

Sure you haven't been compromised somehow?

Nope, I don't think so... Chkrootkit, Rkhunter, /tmp and /dev/shm do not show anything out of the ordinary...

Did you give us the entire, exact line in the log?

This is the full line in the access_log :

127.0.0.1 - - [06/May/2006:15:35:37 +0200] "GET / HTTP/2.0;killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl" 400 299

The effect of that line would be to lock out PHP and Apache for a few seconds, definitely!

According to the ppl managing the server this command was issued by the chkservd process run by cPanel. It checks whether the httpd service is running or not every ten minutes and restarts it when it is down...

Regards.

[brianoz]

Could well be; but if so, chkservd is broken, at least on your system. The killall stuff shouldn't be appearing in the access log! They're shell commands, only meant to be executed in the shell, and only then if httpd is down. Why are they appearing in the log at all?

I have chkservd running on my server and I don't get lines like that in my access log.

[Bdzzld]

Could well be; but if so, chkservd is broken, at least on your system. The killall stuff shouldn't be appearing in the access log! They're shell commands, only meant to be executed in the shell, and only then if httpd is down. Why are they appearing in the log at all?

I have chkservd running on my server and I don't get lines like that in my access log.

The commands does not seem to be issued by checkservd.
The commands in the file /etc/chkserv.d/httpd are not identical to the ones found in the log file...

[Bdzzld]

[update]

access_log :

127.0.0.1 - - [06/May/2006:15:35:37 +0200] "GET / HTTP/2.0;killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl" 400 299

/etc/chkserv.d/https :

service[https]=443,GET / HTTP/2.0;killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl

These are identical!! Does this mean https goes "down" that often and is restarted by chkserv.d each time?? I do not receive any messages from SIM or PRM...

Thanks.