The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Lost or Abandoned PHP Processes - Apache 2.2.17 + mod_fcgid 2.3.5 with suexec

Discussion in 'Workarounds and Optimization' started by trizzo, Dec 20, 2010.

  1. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    I'm running Apache 2.2.17 + mod_fcgid 2.3.5 with suexec on multiple cPanel VPS installs. And they all seem to have an issue with lost or abandoned PHP processes. The following is an example.

    Code:
    [root@srv01:/root]# ps auxwwf | egrep 'php|httpd'
    root     30056  0.0  0.1   4860   680 pts/0    S+   22:54   0:00          \_ egrep php|httpd
    root     11726  0.0  0.9  11204  4848 ?        Ss   Dec17   4:16 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   23871  0.0  0.4  10568  2228 ?        S    22:15   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    root     23883  0.0  0.2  10568  1552 ?        S    22:15   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   23884  0.0  0.3  11204  2092 ?        S    22:15   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    503 27698  0.2  3.6  33300 19004 ?        S    22:52   0:00  |   \_ /usr/bin/php
    nobody   23885  0.0  0.9 288484  4928 ?        Sl   22:15   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   23886  0.0  0.9 288520  4856 ?        Sl   22:15   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   23917  0.0  0.9 288504  4924 ?        Sl   22:15   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   32118  0.0  0.9 288652  5116 ?        Sl   22:18   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    503  5488  0.1  3.8  33832 19960 ?        S    21:03   0:08 /usr/bin/php
    
    In the above output there are two /usr/bin/php processes. One is forked under the httpd process and with an strace seems to be functioning fine. Another is not forked under anything, it looks like it was just spawned by itself and an strace seems like the process is stalled.

    The example above is from a VPS that has only one site and is low traffic. The 503 is the id of the user, I've removed the username for security.

    I haven't found any information about this situation, just wondering what might be happening. I haven't tried downgrading pr upgrading mod_fgid as I don't know the best way to do it.
     
  2. InterServed

    InterServed Well-Known Member

    Joined:
    Jul 10, 2007
    Messages:
    255
    Likes Received:
    2
    Trophy Points:
    18
    cPanel Access Level:
    DataCenter Provider
    Hopefully cPanel will update easyapache to use mod_fcgid 2.3.6:
    Code:
    2010-11-06
    
        The Apache Software Foundation and the Apache HTTP Server Project are pleased to announce the release of version 2.3.6 of mod_fcgid, a FastCGI implementation for Apache HTTP Server versions 2.0, 2.2, and future 2.4. This version of mod_fcgid is a bug fix release.
    
        A fix is included for CVE-2010-3872, a potential vulnerability which can affect sites with untrusted FastCGI applications.
    
        Additionally, default configuration settings for request body handling have been changed to prevent large system resource use. Administrators of all versions of mod_fcgid are strongly cautioned to ensure that FcgidMaxRequestLen is configured appropriately. 
    Download - The Apache HTTP Server Project
     
  3. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    I could download and compile 2.3.6 and replace the .so and not really have many issues. Until I run easyapache again.

    *EDIT* Which I've done and will report back. Hopefully this resolves the issue.
     
    #3 trizzo, Dec 21, 2010
    Last edited: Dec 21, 2010
  4. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    After upgrading to mod_fcgid 2.3.6 the problem is still occurring.
     
  5. InterServed

    InterServed Well-Known Member

    Joined:
    Jul 10, 2007
    Messages:
    255
    Likes Received:
    2
    Trophy Points:
    18
    cPanel Access Level:
    DataCenter Provider
    edit/add-to: /usr/local/apache/conf/includes/post_virtualhost_global.conf
    Code:
    <IfModule mod_fcgid.c>
    FcgidMaxProcesses 150
    FcgidMaxProcessesPerClass 100
    FcgidIOTimeout 300
    FcgidMinProcessesPerClass 1
    FcgidIdleTimeout 300
    FcgidIdleScanInterval 120
    FcgidBusyTimeout 300
    FcgidBusyScanInterval 120
    FcgidErrorScanInterval 10
    FcgidZombieScanInterval 3
    FcgidProcessLifeTime 3600
    </IfModule>
    Tweak the code as desired and then restart httpd service.
     
    #5 InterServed, Dec 22, 2010
    Last edited: Dec 22, 2010
  6. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    No difference. The issue still occurs on two machines.
     
  7. cPanelJared

    cPanelJared Technical Analyst
    Staff Member

    Joined:
    Feb 25, 2010
    Messages:
    1,842
    Likes Received:
    18
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    Friendly moderator note

    I moved this thread to the cPanel and WHM Optimization section.
     
  8. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    Something isn't correctly configured or there is a bug somewhere causing this to occur. I've had to stop using mod_fcgid because of this.
     
  9. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    I've found something really interesting.

    Here is the current apache2 process that forks all the php processes.

    Code:
    root       737  0.0  1.5 11524 8152 ?        Ss   Dec27   0:07 /usr/local/apache/bin/httpd -k start -DSSL
    nobody   26638  0.0  0.4 10888 2520 ?        S    11:14   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    root     26639  0.0  1.0  8484 5636 ?        S    11:14   0:00  \_ /usr/bin/perl /usr/local/cpanel/bin/leechprotect
    nobody   26640  0.0  0.4 11524 2604 ?        S    11:14   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    site1 26669  0.0  2.8 30196 15088 ?       S    11:14   0:06  |   \_ /usr/bin/php
    site2 26783  0.0  2.8 27972 15068 ?       S    11:14   0:01  |   \_ /usr/bin/php
    sute3 26807  0.0  2.7 30120 14356 ?       S    11:15   0:01  |   \_ /usr/bin/php
    site4 26825  2.0  3.1 28448 16348 ?       S    11:16   2:18  |   \_ /usr/bin/php
    site5 26846  0.0  2.8 30288 15020 ?       S    11:17   0:00  |   \_ /usr/bin/php
    site6 26907  0.0  2.0 27404 10688 ?       S    11:19   0:00  |   \_ /usr/bin/php
    site7 26934  0.0  2.0 27404 10728 ?       S    11:20   0:00  |   \_ /usr/bin/php
    site8 27484  0.0  2.8 27472 14688 ?       S    11:31   0:02  |   \_ /usr/bin/php
    site9 27532  0.0  2.8 30216 14924 ?       S    11:32   0:00  |   \_ /usr/bin/php
    site10 27818  0.0  1.6 27272 8900 ?        S    11:38   0:00  |   \_ /usr/bin/php
    site2 29153  1.5  3.0 28244 15956 ?       T    12:21   0:43  |   \_ /usr/bin/php
    site11 29217  0.0  2.7 27784 14668 ?       S    12:23   0:00  |   \_ /usr/bin/php
    site9 29510  0.0  2.2 27456 11628 ?       S    12:32   0:00  |   \_ /usr/bin/php
    site12 30644  0.4  2.1 27532 11088 ?       S    13:05   0:01  |   \_ /usr/bin/php
    nobody   30659  0.1  1.1 112936 6144 ?       Sl   13:05   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    nobody   30722  0.0  1.0 112424 5624 ?       Sl   13:06   0:00  \_ /usr/local/apache/bin/httpd -k start -DSSL
    Here is the abandoned php process (26825) that was left over after httpd (26640) died.

    Code:
    site1 26825  1.9  3.1 28448 16348 ?       S    11:16   2:18 /usr/bin/php
    
    Here is the strace of all the php pid's getting killed and stopped

    Code:
    26640 --- SIGCHLD (Child exited) @ 0 (0) ---
    26640 select(0, NULL, NULL, NULL, {0, 94048}) = 0 (Timeout)
    26640 waitpid(30644, [{WIFEXITED(s) && WEXITSTATUS(s) == 147}], WNOHANG|WSTOPPED) = 30644
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.17") = 0
    26640 waitpid(26669, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 26669
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.1") = 0
    26640 waitpid(26783, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 26783
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.2") = 0
    26640 waitpid(26807, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 26807
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.3") = 0
    26640 waitpid(29153, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 29153
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.13") = 0
    26640 waitpid(26846, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 26846
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.5") = 0
    26640 waitpid(26907, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 26907
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.6") = 0
    26640 waitpid(26934, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 26934
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.7") = 0
    26640 waitpid(27484, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 27484
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.8") = 0
    26640 waitpid(27532, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 27532
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.9") = 0
    26640 waitpid(27818, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 27818
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.11") = 0
    26640 waitpid(29217, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 29217
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.14") = 0
    26640 waitpid(29510, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG|WSTOPPED) = 29510
    26640 unlink("/usr/local/apache/logs/fcgidsock/26640.15") = 0
    26640 exit_group(0)                     = ?
    As you can see it doesn't kill (26825) at all, which is why its left over. So what condition is 26825 in?


    Code:
    Process 26825 attached - interrupt to quit
    accept(0, 0xbfc64600, [112])            = ? ERESTARTSYS (To be restarted)
    --- SIGCONT (Continued) @ 0 (0) ---
    accept(0, 0xbfc64600, [112])            = ? ERESTARTSYS (To be restarted)
    --- SIGCONT (Continued) @ 0 (0) ---
    accept(0, 0xbfc64600, [112])            = ? ERESTARTSYS (To be restarted)
    --- SIGSTOP (Stopped (signal)) @ 0 (0) ---
    --- SIGCONT (Continued) @ 0 (0) ---
    accept(0, 0xbfc64600, [112])            = ? ERESTARTSYS (To be restarted)
    --- SIGSTOP (Stopped (signal)) @ 0 (0) ---
    --- SIGSTOP (Stopped (signal)) @ 0 (0) ---
    accept(0, 0xbfc64600, [112])            = ? ERESTARTSYS (To be restarted)
    --- SIGCONT (Continued) @ 0 (0) ---
    accept(0, 0xbfc64600, [112])            = ? ERESTARTSYS (To be restarted)
    --- SIGSTOP (Stopped (signal)) @ 0 (0) ---
    --- SIGSTOP (Stopped (signal)) @ 0 (0) ---
    accept(0, 0xbfc64600, [112])            = ? ERESTARTSYS (To be restarted)
    
    I guess this is a bug?
     
  10. xanubi

    xanubi Well-Known Member

    Joined:
    Jun 28, 2006
    Messages:
    86
    Likes Received:
    1
    Trophy Points:
    8
    I had this problem, but it was because of the Configserver Firewall ( ConfigServer Security & Firewall ), i had configured the value of PT_USERTIME. What i did was turning the value of PT_USERTIME to 0 (zero). I killed all the php processes, restarted apache ( with /scripts/restartsrv_http ). And no more Lost or abandoned processes.

    If you don't have this configured, something else is killing your processes and doing the lost or abandoned php's that you see. Try to see if you have something monitoring apache and processes, and if that monitoring has the ability to kill, that's the problem for sure.

    EDIT: my server is like yours, Apache 2.2.17 MPM Worker + mod_fcgid 2.3.5 with suexec
     
    #10 xanubi, Jan 30, 2011
    Last edited: Jan 30, 2011
  11. xanubi

    xanubi Well-Known Member

    Joined:
    Jun 28, 2006
    Messages:
    86
    Likes Received:
    1
    Trophy Points:
    8
    Ok, new information on this one, my solution didn't resolve the problem!!
    It's doing the same like you. I'll fill a bug report
     
  12. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Hello xanubi,

    Please post the ticket number when you've filed the report for us to look into it further. This way we can have an easy reference to find it in our system.

    Thanks!
     
  13. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    I was working with someone in #apache on freenode trying to narrow it down. I eventually just moved to worker + suphp for the time being. Now my big box Dual Quad + 12GB of memory is doing the same, previous it wasn't acting like this. Just small VPS boxes were. Something triggered it, and someone recently asked for the path to php-cli. Dunno if thats a coincidence.
     
  14. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    Re: Lost or Abandoned PHP Processes - Apache 2.2.17 + mod_fcgid 2.3.5 with

    Reviving an old thread, this issue came back after enabling the PDO and MySQL PDO extensions and re-compiling apache/php.

    So Strange!
     
  15. srpurdy

    srpurdy Well-Known Member

    Joined:
    Jun 1, 2011
    Messages:
    101
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Re: Lost or Abandoned PHP Processes - Apache 2.2.17 + mod_fcgid 2.3.5 with

    Yeah I've noticed this happen all of sudden on 11.32 (edge) on one box. I've re-compiled through easyapache thought maybe something was amiss with the 3.91 compile So see if that makes any difference. Are you Edge Trizzo ? wondering if it's an edge problem or not. I also have PDO and MYsql PDO enabled.
     
  16. trizzo

    trizzo Member

    Joined:
    Dec 20, 2010
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    Re: Lost or Abandoned PHP Processes - Apache 2.2.17 + mod_fcgid 2.3.5 with

    I'm only on release, and this is killing my box each time after like 8 hours. So to help I've set the kernel option to reboot on panic.

    Add the below to /etc/sysctl.conf

    Then run sysctl -s

    or

    If you want to kill all the abandoned PHP processes, just run this.

    ps auxwwwf | grep '[0-9] /usr/bin/php' | awk '{ print $2 }' | xargs kill -9
     
  17. srpurdy

    srpurdy Well-Known Member

    Joined:
    Jun 1, 2011
    Messages:
    101
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Re: Lost or Abandoned PHP Processes - Apache 2.2.17 + mod_fcgid 2.3.5 with

    Oddly for me it seems to happen at the sametime everyday. around 6:30am. I have no crons or anything at that time... confused :|
     
  18. Quemeros

    Quemeros Member

    Joined:
    Mar 4, 2012
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Re: Lost or Abandoned PHP Processes - Apache 2.2.17 + mod_fcgid 2.3.5 with

    I have the exactly same problem with the same configuration: MPM Worker, mod_fcgid, etc.
    I only can re-create that lost process (after kill them), restarting apache, and i think that problem isn't in mod_fcgid because in the Apache stats the module don't conunt that process.

    If you check peridly process in the server with command pstree, you will see that all process generate in a cuple off minutes, and may be the random think to generate again is Apache re-start or similar.

    May be some config can repair this? Because it's too problematic, and isn't a very good idea have a permantly cron-command for kill all process.

    Thanks trizzo for "killing line" :)

    EDIT: Light appears... we need to wait to EA 3.10 http://forums.cpanel.net/f5/zombie-fcgid-processes-don-t-die-261891.html#post1093911
     
    #18 Quemeros, Mar 4, 2012
    Last edited: Mar 5, 2012
  19. srpurdy

    srpurdy Well-Known Member

    Joined:
    Jun 1, 2011
    Messages:
    101
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Re: Lost or Abandoned PHP Processes - Apache 2.2.17 + mod_fcgid 2.3.5 with

    EA 3.10 didn't solve the problem here.....
     
  20. ikillbill

    ikillbill Well-Known Member

    Joined:
    Feb 18, 2008
    Messages:
    119
    Likes Received:
    0
    Trophy Points:
    16
    Re: Lost or Abandoned PHP Processes - Apache 2.2.17 + mod_fcgid 2.3.5 with

    +1 , EA 3.10 did not fix this issue...
     

Share This Page