The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

PHP processes being stalled a few times a day

Discussion in 'General Discussion' started by Bdzzld, Apr 14, 2006.

  1. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    Hi,

    I've got three dual opteron servers, which are all experiencing the same problem. All three servers are running CentOS 4.3 x86_64 with the latest *stable* release of cPanel/WHM. PHP/Apache has been compiled with :

    * GD support
    * IMAP support
    * PHPsuexec support
    * Zend Optimizer

    A few times a day the PHP module just stops responding (noticeble by PHP processes being stalled). During that time normal html pages are displayed without problems, thus the assumption that it has something to do with the PHP module. The "lock" always lasts for a few seconds and then everything is being served as usual. During such time there's no heavy load at all. What I also noticed is that you're unable to log onto the server via SSH during the "lock". User_id and password are being accepted, but the information is not being processed during the "lock".

    I've already created a ticket for it @ cPanel, but after three weeks of testing, cPanel claims it's a problem with the hardware and not related to cPanel not functioning correctly... All three servers are different brands and are almost brand-new, so I doubt that's the case here...

    Does anyone have similar problems? And most important... was able to solve it?

    Thanking you in advance.
     
  2. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Odd. Could be a resource leak or even running out of MySQL connections. Things to try:

    In httpd.conf set:

    KeepAlives 3
    MaxRequestsPerChild 50

    Then restart httpd. The latter helps with resource leaks.

    In /usr/lib/php.ini:

    mysql.allow_persistent = Off

    Then restart httpd.

    In /etc/my.cnf

    set-variable = max_connections=750

    Then restart MySQL. This shouldn't really be necessary with the httpd.conf and php.ini changes, though.

    Final thought, make sure /tmp isn't nearly full incase the problem is with creating session files within it.
     
  3. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    Hi Chirpy,

    Thanks for trying to help me out...

    The following is currently set up :

    Code:
    Timeout 50
    KeepAlive On
    MaxKeepAliveRequests 100
    KeepAliveTimeout 5
    MinSpareServers 20
    MaxSpareServers 35
    StartServers 50
    MaxClients 512
    MaxRequestsPerChild 10000
    
    I could not find "KeepAlives". Is that a typo?
    The other options you mentioned were already present/altered...

    Thanks.
     
  4. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Sorry, KeepAlives should be KeepAliveTimeout. As mentioned above, I'd suggest that you set MaxRequestsPerChild down to 50
     
  5. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    Hi Chirpy,

    Thanks for the suggestions. Unfortunately they did solve the issue...

    PHP processes are still stalled on this server a few times a day...
    This is also noticable by executing :
    Code:
    root@xxxxx [~]# php -v
    PHP 4.4.2 (cli) (built: Mar 26 2006 06:52:46)
    Copyright (c) 1997-2006 The PHP Group
    Zend Engine v1.3.0, Copyright (c) 1998-2004 Zend Technologies
        with Zend Extension Manager v1.0.9, Copyright (c) 2003-2006, by Zend Technologies
        with Zend Optimizer v2.6.2, Copyright (c) 1998-2006, by Zend Technologies
    
    Normally this will show the installed version immediately, but at the time the PHP processes are being stalled, it takes up to a few seconds before that information is displayed.

    Server load does not seem to have anything to do with the PHP processes being stalled; it also happens when server load is @ only 0.38.

    *) Is it possible one client on the server could be the source of this issue (although nothing out of the ordinary is shown)?
    *) Do you have any other suggestions I can try?

    Thanks once again.
     
  6. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    [Update]
    Last night was the worsed of all cases. All three servers in the same DNS cluster did not respond to any PHP request for quite some time at the same(!) time. Restarting httpd on one of these servers did not solve the problem.

    I really would like some help on this problem, becuase this is really costing me my sleep (and my customers)... :(

    Thanks to any one who can help...
     
  7. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Any chance this is some sort of DNS lookup (or reverse DNS lookup) problem?
     
  8. MMarko

    MMarko Well-Known Member

    Joined:
    Apr 18, 2005
    Messages:
    316
    Likes Received:
    0
    Trophy Points:
    16
    KeepAlive On - turn this off!
     
  9. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    I don't think so... plain HTML sites are serviced without any problems during the problem...

    That should only be the case if the problem could be determined to "Server load" levels, but that's certainly not the case here... because PHP scripts were not (able to be) serviced "server load" dropped down to .22 levels at most...
     
  10. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    [Update 2]

    During another stall of PHP scripts on one of the servers I noticed the following in the /etc/httpd/logs/access_log :

    Code:
    killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl
    
    Any idea what this line does and where it comes from (I surely did not execute it)?

    Thanks...
     
  11. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Sure you haven't been compromised somehow? Did you give us the entire, exact line in the log? The effect of that line would be to lock out PHP and Apache for a few seconds, definitely!
     
  12. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    Nope, I don't think so... Chkrootkit, Rkhunter, /tmp and /dev/shm do not show anything out of the ordinary...

    This is the full line in the access_log :
    Code:
    127.0.0.1 - - [06/May/2006:15:35:37 +0200] "GET / HTTP/2.0;killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl" 400 299
    
    According to the ppl managing the server this command was issued by the chkservd process run by cPanel. It checks whether the httpd service is running or not every ten minutes and restarts it when it is down...

    Regards.
     
  13. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Could well be; but if so, chkservd is broken, at least on your system. The killall stuff shouldn't be appearing in the access log! They're shell commands, only meant to be executed in the shell, and only then if httpd is down. Why are they appearing in the log at all?

    I have chkservd running on my server and I don't get lines like that in my access log.
     
  14. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    The commands does not seem to be issued by checkservd.
    The commands in the file /etc/chkserv.d/httpd are not identical to the ones found in the log file... :confused:
     
  15. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    [update]

    access_log :
    Code:
    127.0.0.1 - - [06/May/2006:15:35:37 +0200] "GET / HTTP/2.0;killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl" 400 299
    
    /etc/chkserv.d/https :
    Code:
    service[https]=443,GET / HTTP/2.0;killall -TERM httpd;sleep 2;killall -9 httpd;/etc/rc.d/init.d/httpd stop;/etc/rc.d/init.d/httpd startssl;/usr/local/apache/bin/apachectl startssl
    
    These are identical!! Does this mean https goes "down" that often and is restarted by chkserv.d each time?? I do not receive any messages from SIM or PRM...

    Thanks.
     
  16. Bdzzld

    Bdzzld Well-Known Member

    Joined:
    Apr 3, 2004
    Messages:
    356
    Likes Received:
    1
    Trophy Points:
    18
    [UPDATE]
    Yesterday morning I upgraded cPanel/WHM from the latest *stable* version WHM 10.8.0 cPanel 10.8.1-S114 to the latest *release* version WHM 10.8.0 cPanel 10.8.2-R83 in an desperate(!) attempt to see if that would make some difference.

    After several weeks(!) of problems, several tickets to cPanel support - which did not seem to be able to replicate the error or did not think it was a cPanel problem - this seems to have solved the issue...

    During the whole day I've not experienced any "stall" moment. The account which I suspected to be responsible for the "stall" moments was putting Apache in a wait ("W") status again, but this time this was not affecting the other PHP web sites on the server, only the account itself (which it should have in the 1st place imho)...

    Now everything seems to be working as it should be, I'd like to know which change or bugfix between the *stable* and *release* version may be responsible for solving this issue... I don't want it to return again in the future after an upgrade to another version if you know what I mean... Can someone share this info with me?

    Thanks.
     
  17. freedman

    freedman Well-Known Member

    Joined:
    Feb 13, 2005
    Messages:
    312
    Likes Received:
    1
    Trophy Points:
    18
    looks like the cuplrit was an error in the chkservd script.

    look at your new one and you'll see.

    it was checking for httpd on port 443 instead of 80, didn't see one, and restarted httpd.
    you got web pages, because that's one of the first things apache can do when it's coming up--deliver html pages.. the php pages had to wait til it loaded the php module, I presume.
     
Loading...

Share This Page