The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

SOS apache fails every few seconds

Discussion in 'EasyApache' started by s_2_s, Feb 5, 2008.

  1. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    hello
    i have few servers where i deleted domlogs folders because that was above 10-20 GB
    suddenly after few hours i suddenly discovered that apache fails every few seconds

    i then recreated domlogs made sure that domlogs have same permission and owner

    # ll
    total 76
    drwxr-xr-x 15 root root 4096 Feb 5 22:00 ./
    drwxr-xr-x 23 root root 4096 Jan 10 20:00 ../
    drwxr-xr-x 2 root root 4096 Jan 21 00:33 bin/
    drwxr-xr-x 2 root root 4096 Jan 21 01:04 cgi-bin/
    drwxr-xr-x 11 root root 4096 Feb 5 22:35 conf/
    drwxr-xr-x 9 root root 4096 Nov 23 14:02 conf_pre_ea3/
    drwx--x--x 2 root wheel 12288 Feb 5 22:42 domlogs/
    drwxr-xr-x 70 root wheel 12288 Feb 5 21:45 domlogsx/
    drwxr-xr-x 4 root root 4096 Feb 4 17:06 htdocs/
    drwxr-xr-x 3 root root 4096 Jan 21 00:33 icons/
    drwxr-xr-x 3 root root 4096 Jan 21 00:33 include/
    drwxr-xr-x 2 root root 4096 Jan 21 00:48 libexec/
    drwxr-xr-x 2 root root 4096 Feb 5 22:48 logs/
    drwxr-xr-x 4 root root 4096 Jan 21 00:33 man/
    drwxr-xr-x 2 nobody nobody 4096 Jan 21 00:33 proxy/


    then restarted and problem repeated

    then i commented out the
    #CustomLog
    #BytesLog
    and restarted but same problem

    i rebuilded apache and php several times with different versions and it didn't help either
    i have the same problem on all the servers from which i delete domlogs

    on var/log/messages nothing
    on apache error log there is

    [Tue Feb 4 12:16:04 2008] [notice] caught SIGTERM, shutting down

    which usually apache dies after it appear
    please do help me
    waiting ..........
     
  2. troxalias

    troxalias Well-Known Member

    Joined:
    Nov 21, 2001
    Messages:
    96
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Athens - Greece
    1. Do any files get created in the domlogs directory?
    2. Does the process dies immediately ?
    3. Have you tried to strace the apache process ?
     
  3. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    thank you for your reply
    1-yes and their owner is root:root or root:user
    2- it dies in few seconds only
    3- no i didn't and dont know how either

    waiting for your update
     
  4. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    i saw this on one of the affected server before apache dies too

    [Wed Feb 6 00:05:03 2008] [error] Bad pid (7747) in scoreboard slot 23
    [Wed Feb 6 00:05:03 2008] [error] Bad pid (7809) in scoreboard slot 24
    [Wed Feb 6 00:05:03 2008] [error] Bad pid (9910) in scoreboard slot 25
    [Wed Feb 6 00:05:03 2008] [error] Bad pid (10105) in scoreboard slot 26
    [Wed Feb 6 00:05:03 2008] [error] Bad pid (7461) in scoreboard slot 22
    [Wed Feb 6 00:05:03 2008] [error] Bad pid (7747) in scoreboard slot 23
    [Wed Feb 6 00:05:03 2008] [error] Bad pid (7809) in scoreboard slot 24
    [Wed Feb 6 00:05:03 2008] [error] Bad pid (9910) in scoreboard slot 25
    [Wed Feb 6 00:05:03 2008] [error] Bad pid (10105) in scoreboard slot 26
    [Wed Feb 6 00:05:03 2008] [notice] caught SIGTERM, shutting down



    now running your command
     
  5. nyjimbo

    nyjimbo Well-Known Member

    Joined:
    Jan 25, 2003
    Messages:
    1,125
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    New York
    Are there any lines above it that give you any more ideas?. Can you turn up the debug level on apache.conf manually to LogLevel debug and see if more info comes out ?
     
  6. troxalias

    troxalias Well-Known Member

    Joined:
    Nov 21, 2001
    Messages:
    96
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Athens - Greece
    Try to run the following command:

    Code:
    strace -o /tmp/lala -f /usr/local/apache/bin/httpd -DSSL
    it will probably take some time until apache dies. then check the file /tmp/lala (especially the last 50-100 lines) for any files that could not be accessed either with the error ENOENT of EACC. I know it is not an easy way to debug this but it's the best i can suggest. If you can compress the file /tmp/lala and post it as an attachment would be perfect.
     
  7. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    also could catch 25821 bind(16, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)


    appearing everywhile but apache is not died yet
     
  8. troxalias

    troxalias Well-Known Member

    Joined:
    Nov 21, 2001
    Messages:
    96
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Athens - Greece

    Is apache already running ???? this means that a process is already bind to TCP port 443. Try
    Code:
    lsof -i tcp |grep https 
    to check the process that is bind to port 443.
     
  9. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    now apache dies again

    attached is the lala file

    at the time oif apache failure this reappaeared in error log

    [Wed Feb 6 00:10:07 2008] [error] Bad pid (28416) in scoreboard slot 16
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (22008) in scoreboard slot 18
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (28574) in scoreboard slot 19
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (22136) in scoreboard slot 21
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (22518) in scoreboard slot 22
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (23590) in scoreboard slot 23
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (23634) in scoreboard slot 25
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (23664) in scoreboard slot 26
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (28416) in scoreboard slot 16
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (22008) in scoreboard slot 18
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (28574) in scoreboard slot 19
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (22136) in scoreboard slot 21
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (22518) in scoreboard slot 22
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (23590) in scoreboard slot 23
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (23634) in scoreboard slot 25
    [Wed Feb 6 00:10:07 2008] [error] Bad pid (23664) in scoreboard slot 26
    [Wed Feb 6 00:10:07 2008] [notice] caught SIGTERM, shutting down
     

    Attached Files:

  10. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    thanks for your interest in helping me

    while httpd was down there were nothing listening to the https port

    but after i restarted it , its only https

    httpd 27962 root 16u IPv4 29004321 TCP *:https (LISTEN)
    httpd 28034 nobody 16u IPv4 29004321 TCP *:https (LISTEN)
    httpd 28035 nobody 16u IPv4 29004321 TCP *:https (LISTEN)
    httpd 28036 nobody 16u IPv4 29004321 TCP *:https (LISTEN)
    httpd 28037 nobody 16u IPv4 29004321 TCP *:https (LISTEN)
    httpd 28038 nobody 16u IPv4 29004321 TCP *:https (LISTEN)


    also apache dies only after [Wed Feb 6 00:10:07 2008] [notice] caught SIGTERM, shutting down
    however the error of listening on ssl port happens but it doesn't kill apache
     
  11. troxalias

    troxalias Well-Known Member

    Joined:
    Nov 21, 2001
    Messages:
    96
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Athens - Greece
    Can you please post the file again but run it with the command (please make sure that no apache processes are still running, if any, and post also the results of lsof i ask ed you befora:

    Code:
    strace -s 512 -o /tmp/lala -f /usr/local/apache/bin/httpd -DSSL
    I would also suggest you to disable any eaccelarator entry in /usr/local/lib/php.ini file before you try to start apache.
     
  12. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    died again

    this time i see only


    [Wed Feb 6 00:35:03 2008] [notice] caught SIGTERM, shutting down

    i commented out all php extensions
    in error log 200+ mb this time so here are the last 100 lines of it
     
  13. troxalias

    troxalias Well-Known Member

    Joined:
    Nov 21, 2001
    Messages:
    96
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Athens - Greece
    Not very helpful. If you zip the file how big is it ? Please try to run it again under strace but instead of 512 (in -s 512) try 64 to see it the trace file gets smaller. Also try to compress it with gzip -9 in order to get maximum compression.
     
  14. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    its 115 mb zipped to maximum :(
     
  15. troxalias

    troxalias Well-Known Member

    Joined:
    Nov 21, 2001
    Messages:
    96
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Athens - Greece
    That's not good. One last try to see if the file gets smaller:
    Code:
    strace -s 64 -o /tmp/lala -f /usr/local/apache/bin/httpd -F -DSSL

    If the file is huge again I have to try to guess...
    1. Disable any php modules in your configuration file.
    2. Disable any other loaded modules (one at a time).
     
  16. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    sorry
    sadly the log file is over 90 mb
    what can i do now please ?
     
  17. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    any help please ?
    many server are down
     
  18. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    i believe it may be all because of the domlogs folder that i deleted please help
     
  19. s_2_s

    s_2_s Well-Known Member

    Joined:
    Aug 9, 2004
    Messages:
    215
    Likes Received:
    0
    Trophy Points:
    16
    problem fixed by forcing apache to listen to the ip only and not on 0.0.0.0.0
    for an unknown reason apache was competing with its own processes or kinda
     
Loading...

Share This Page