The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Apache keeps crashing: No space left on device.

Discussion in 'EasyApache' started by Go3team, Apr 22, 2005.

  1. Go3team

    Go3team Member

    Joined:
    Dec 4, 2003
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    1
    This happens at random, but when it does, the automatic restart feature will not work. I have to go in, and restart via the command line. Here is the log:

    [Fri Apr 22 11:47:07 2005] [warn] child process 16194 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 16195 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 16197 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 16198 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 16200 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 16204 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 15414 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 15419 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 15420 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 20045 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 20287 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 20314 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:07 2005] [warn] child process 20317 still did not exit, sending a SIGTERM
    [Fri Apr 22 11:47:10 2005] [warn] pid file /usr/local/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
    semget: No space left on device

    I did not post the remaining lines as they are identical to the last line except for the timestamp.

    The disk space lists as such:
    Disk hda7 (/) 40 %
    Disk hda1 (/boot) 10 %
    Disk hda8 (/home) 17 %
    Disk hda5 (/tmp) 4 %
    Disk hda2 (/usr) 59 %
    Disk hda3 (/var) 11 %

    None are even near full. I am at a loss. Any help would be appreciated. Thanks.
     
  2. trparky

    trparky Well-Known Member

    Joined:
    Apr 23, 2003
    Messages:
    184
    Likes Received:
    1
    Trophy Points:
    0
    Anything from /var/messages and /usr/local/apache/logs/error_log?
     
  3. Renjith Rajan

    Renjith Rajan Member

    Joined:
    Aug 22, 2004
    Messages:
    17
    Likes Received:
    0
    Trophy Points:
    1
    Is mod_python installed in your server? sometimes it leaks the semaphores and causes the issue. try ipcs -s to check the Semaphore Arrays.
     
  4. Go3team

    Go3team Member

    Joined:
    Dec 4, 2003
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    1
    trparky:
    /var/log/messages shows nothing of consequence during the time listed below. The errors above came from /usr/local/apache/logs/error_log.

    Renjith Rajan:
    ------ Semaphore Arrays --------
    key semid owner perms nsems
    0x00000000 655360 nobody 600 1
    0x0052e2c1 12255233 postgres 600 17
    0x0052e2c2 12288002 postgres 600 17
    0x0052e2c3 12320771 postgres 600 17
    0x00000000 884740 nobody 600 1
    0x00000000 917509 nobody 600 1
    0x00000000 1015814 nobody 600 1
    0x00000000 1048583 nobody 600 1
    0x00000000 1114120 nobody 600 1
    0x00000000 1146889 nobody 600 1
    0x00000000 1179658 nobody 600 1
    0x00000000 1245195 nobody 600 1
    0x00000000 1310732 nobody 600 1
    0x00000000 1376269 nobody 600 1
    0x00000000 1409038 nobody 600 1
    0x00000000 1441807 nobody 600 1
    0x00000000 1474576 nobody 600 1
    0x00000000 1540113 nobody 600 1
    0x00000000 1638418 nobody 600 1
    0x00000000 1671187 nobody 600 1
    0x00000000 1736724 nobody 600 1
    0x00000000 1998869 nobody 600 1
    0x00000000 2097174 nobody 600 1
    0x00000000 2162711 nobody 600 1
    0x00000000 2424856 nobody 600 1
    0x00000000 2457625 nobody 600 1
    0x00000000 2490394 nobody 600 1
    0x00000000 2588699 nobody 600 1
    0x00000000 2621468 nobody 600 1
    0x00000000 2949149 nobody 600 1
    0x00000000 2981918 nobody 600 1
    0x00000000 3047455 nobody 600 1
    0x00000000 3080224 nobody 600 1
    0x00000000 3145761 nobody 600 1
    0x00000000 3866658 nobody 600 1
    0x00000000 3899427 nobody 600 1
    0x00000000 4063268 nobody 600 1
    0x00000000 4227109 nobody 600 1
    0x00000000 4259878 nobody 600 1
    0x00000000 4358183 nobody 600 1
    0x00000000 4390952 nobody 600 1
    0x00000000 4423721 nobody 600 1
    0x00000000 4620330 nobody 600 1
    0x00000000 4784171 nobody 600 1
    0x00000000 5144620 nobody 600 1
    0x00000000 5177389 nobody 600 1
    0x00000000 5341230 nobody 600 1
    0x00000000 5668911 nobody 600 1
    0x00000000 5734448 nobody 600 1
    0x00000000 5767217 nobody 600 1
    0x00000000 5799986 nobody 600 1
    0x00000000 5832755 nobody 600 1
    0x00000000 5865524 nobody 600 1
    0x00000000 5898293 nobody 600 1
    0x00000000 5931062 nobody 600 1
    0x00000000 5996599 nobody 600 1
    0x00000000 6094904 nobody 600 1
    0x00000000 6127673 nobody 600 1
    0x00000000 6225978 nobody 600 1
    0x00000000 6324283 nobody 600 1
    0x00000000 6488124 nobody 600 1
    0x00000000 6520893 nobody 600 1
    0x00000000 6651966 nobody 600 1
    0x00000000 6783039 nobody 600 1
    0x00000000 7110720 nobody 600 1
    0x00000000 7307329 nobody 600 1
    0x00000000 7340098 nobody 600 1
    0x00000000 7405635 nobody 600 1
    0x00000000 7438404 nobody 600 1
    0x00000000 7471173 nobody 600 1
    0x00000000 7700550 nobody 600 1
    0x00000000 7766087 nobody 600 1
    0x00000000 7864392 nobody 600 1
    0x00000000 7897161 nobody 600 1
    0x00000000 7962698 nobody 600 1
    0x00000000 7995467 nobody 600 1
    0x00000000 8028236 nobody 600 1
    0x00000000 8061005 nobody 600 1
    0x00000000 8093774 nobody 600 1
    0x00000000 8126543 nobody 600 1
    0x00000000 8192080 nobody 600 1
    0x00000000 8257617 nobody 600 1
    0x00000000 8290386 nobody 600 1
    0x00000000 8323155 nobody 600 1
    0x00000000 8388692 nobody 600 1
    0x00000000 8454229 nobody 600 1
    0x00000000 8519766 nobody 600 1
    0x00000000 8650839 nobody 600 1
    0x00000000 8683608 nobody 600 1
    0x00000000 8716377 nobody 600 1
    0x00000000 8814682 nobody 600 1
    0x00000000 8912987 nobody 600 1
    0x00000000 8945756 nobody 600 1
    0x00000000 8978525 nobody 600 1
    0x00000000 9044062 nobody 600 1
    0x00000000 9076831 nobody 600 1
    0x00000000 9142368 nobody 600 1
    0x00000000 9338977 nobody 600 1
    0x00000000 9633890 nobody 600 1
    0x00000000 9666659 nobody 600 1
    0x00000000 9764964 nobody 600 1
    0x00000000 9830501 nobody 600 1
    0x00000000 9896038 nobody 600 1
    0x00000000 9928807 nobody 600 1
    0x00000000 9961576 nobody 600 1
    0x00000000 10059881 nobody 600 1
    0x00000000 10289258 nobody 600 1
    0x00000000 10354795 nobody 600 1
    0x00000000 10420332 nobody 600 1
    0x00000000 10485869 nobody 600 1
    0x00000000 10518638 nobody 600 1
    0x00000000 10616943 nobody 600 1
    0x00000000 10879088 nobody 600 1
    0x00000000 11010161 nobody 600 1
    0x00000000 11042930 nobody 600 1
    0x00000000 11075699 nobody 600 1
    0x00000000 11141236 nobody 600 1
    0x00000000 11174005 nobody 600 1
    0x00000000 11239542 nobody 600 1
    0x00000000 11305079 nobody 600 1
    0x00000000 11337848 nobody 600 1
    0x00000000 11468921 nobody 600 1
    0x00000000 11534458 nobody 600 1
    0x00000000 11567227 nobody 600 1
    0x00000000 11632764 nobody 600 1
    0x00000000 11665533 nobody 600 1
    0x00000000 11731070 nobody 600 1
    0x00000000 12025983 nobody 600 1
     
  5. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Have you checked the sizes of the apache log files in /etc/httpd/logs/ and /etc/httpd/domlogs, if any are near or over 2GB apache will have problems and you should rotate them out of the way.
     
  6. Go3team

    Go3team Member

    Joined:
    Dec 4, 2003
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    1
    Files in /etc/httpd/logs/:

    access_log 200.9M
    error_log 14.9M
    httpd.pid 6
    ssl_engine_log 3.7M
    and a bunch of ssl_mutex.xxxxx files with 0 file length. Can these be deleted?

    The largest file in /etc/httpd/domlogs is just about 21mb.
     
  7. Go3team

    Go3team Member

    Joined:
    Dec 4, 2003
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    1
    Is there a way to make the automatic restart more powerful? When I restart httpd under root, it does it right away. If I try in WHM it won't restart it either.
     
  8. gemby

    gemby Well-Known Member
    PartnerNOC

    Joined:
    Feb 16, 2002
    Messages:
    182
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Pula, Croatia
    cPanel Access Level:
    DataCenter Provider
    I had same problems few weeks ago, i have no mod_python, but i think that this problem makes httpd itself due to unclean shutdowns, child crashings etc. Those days i had tons of ..reading.. requests.....

    If restarting httpd does not clean at least one semaphore place, it will say segmet: no space left on device regarles that it has a lot of space on all partitions....

    I found some "cleaning script" on net somewhere, an it solved my problems. Ok maybe not solved, but now at least my httpd-s are working now :))
    __________________________________________________________
    #!/bin/bash

    ipcs -s | grep nobody | perl -e 'while (<STDIN>) {
    @a=split(/\s+/); print `ipcrm sem $a[1]`}'

    /scripts/restartsrv httpd
    ___________________________________________________________


    I run it every day @ 2 am on my most affected box, and have it ready on all boxes, just in case.
     
    #8 gemby, May 2, 2005
    Last edited: May 2, 2005
  9. rpmws

    rpmws Well-Known Member

    Joined:
    Aug 14, 2001
    Messages:
    1,824
    Likes Received:
    5
    Trophy Points:
    38
    Location:
    back woods of NC, USA

    damn!!! I am glad I found this tonight!!! https would not run no matter what ..space looked fine everywhere. I had a DBD perl module fail at upcp tonight and https failed after that ..could get httpd to run but not https. Thanks for this post!! and the script!!!
     
  10. gemby

    gemby Well-Known Member
    PartnerNOC

    Joined:
    Feb 16, 2002
    Messages:
    182
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Pula, Croatia
    cPanel Access Level:
    DataCenter Provider
    I am glad that i could help you, i have few dozens of such SOS scripts on servers, made to help me solve such small hedaches....
     
  11. ramprage

    ramprage Well-Known Member

    Joined:
    Jul 21, 2002
    Messages:
    667
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    Great script, just worked for me as well thanks
     
  12. naveen3

    naveen3 Active Member

    Joined:
    Jan 28, 2005
    Messages:
    30
    Likes Received:
    0
    Trophy Points:
    6
    Big thanks to Gemby

    Thanks for you solution.
    it save my Hrs.

    Regards
    Naveen
     
Loading...

Share This Page