Results 1 to 5 of 5

Thread: server crash mystery

  1. #1
    Registered Member
    Join Date
    May 2003
    Posts
    10

    Default server crash mystery

    This has happened a number of times in the past week, and we can't figure out what the souce of the problem is.

    Our server goes through a complete break down, one service after the next, seemingly starting with mysql. From there, apache goes out and we can't get in w/ ssh. The server has to be physically restarted.

    The first errors we can see in the logs are from php scripts that connect to mysql:

    mysql_connect(): Can't create a new thread (errno 11). If you are not out of available memory, you can consult the manual for a possible OS-dependent bug in....
    during this time perl scripts connecting to myql give the error:

    Resource temporarily unavailable: couldn't spawn child process:
    after a few minutes of this, the whole server goes down and we have to call the hosting company to manually restart. Our server has 256 RAM, which may be part of the problem, though most of the time the ram is at less than 30%. Also, one would hope that maxing out the ram would have less dire consequences.

    Any ideas would be very helpful.

    Neil

    cPanel.net Support Ticket Number:

  2. #2
    Registered Member
    Join Date
    Aug 2002
    Posts
    1,052

    Default

    Well I think you answered your own question. It sounds like something is munching all of your RAM/SWAP and the machine is falling over. Add more RAM and I'm sure the issue will go away.

    cPanel.net Support Ticket Number:

    The rest of those who have gone before us cannot steady the unrest of those to follow.

  3. #3
    Registered Member nyjimbo's Avatar
    Join Date
    Jan 2003
    Location
    New York
    Posts
    1,125

    Default

    If you can, run a "TOP" and look for zombies. If you see alot of them then you have a bad apache or bad frontpage install.

    We had a similiar problem and noticed hundreds of zombies from author.exe and fpcount.exe, all FP stuff and had to do a kill -1 (pid) every couple hours until we had a fix for the bad frontpage extensions.

    cPanel.net Support Ticket Number:

  4. #4
    Registered Member
    Join Date
    May 2003
    Posts
    10

    Default

    Thanks so much for the help. We are looking into getting more ram or moving to a new machine.

    I am left with more questions though. Even if the server had a burst of memory demanding traffic, shouldn't it be able to handle this in ways that don't involve the whole machine shutting down needing a restart? What about sending errors to requests saying that the server is temp. unavail. etc.

    Right now there are no appearent zombees. There is no way to find out what happened with RAM/CPU at a given time of day in the logs is there? I imagine that storing that much information would be a log overkill. The apache logs are not telling us very much. Where else can one look for clues?

    Thanks again,
    Neil

    cPanel.net Support Ticket Number:

  5. #5
    Registered Member
    Join Date
    Aug 2002
    Posts
    1,052

    Default

    Sure, some processes won't be able to fork and you will see out of memory errors in dmesg and maybe some in messages depending on the process. However, if an already present process on the box grows, it will kill your box. That's the way the box is supposed to perform, if you'd like to kill processes that take x amount of ram or processor cycles, then find/buy (http://www.web-bazaar.com/cgi/process_doctor.html) or create a process watching daemon.

    You can install some scripts, etc. to monitor your servers resource usage. A popular script is http://hotsanic.sourceforge.net/

    cPanel.net Support Ticket Number:

    The rest of those who have gone before us cannot steady the unrest of those to follow.

Similar Threads

  1. Server crash
    By Hostprofil in forum General Discussion
    Replies: 7
    Last Post: 10-31-2013, 02:10 PM
  2. Server Crash?
    By lancealx in forum General Discussion
    Replies: 1
    Last Post: 08-06-2007, 06:12 AM
  3. Server Crash
    By Corrado in forum General Discussion
    Replies: 5
    Last Post: 01-13-2006, 11:07 AM
  4. New server setup steps make server crash?
    By Freezer in forum General Discussion
    Replies: 8
    Last Post: 10-25-2005, 04:30 PM
  5. need you help, server crash always...
    By Jedia in forum General Discussion
    Replies: 2
    Last Post: 03-17-2005, 05:27 AM
bargain