1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

server crash mystery

Discussion in 'General Discussion' started by neilh, Aug 3, 2003.

  1. neilh

    neilh New Member

    Joined:
    May 30, 2003
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    This has happened a number of times in the past week, and we can't figure out what the souce of the problem is.

    Our server goes through a complete break down, one service after the next, seemingly starting with mysql. From there, apache goes out and we can't get in w/ ssh. The server has to be physically restarted.

    The first errors we can see in the logs are from php scripts that connect to mysql:

    during this time perl scripts connecting to myql give the error:

    after a few minutes of this, the whole server goes down and we have to call the hosting company to manually restart. Our server has 256 RAM, which may be part of the problem, though most of the time the ram is at less than 30%. Also, one would hope that maxing out the ram would have less dire consequences.

    Any ideas would be very helpful.

    Neil

    cPanel.net Support Ticket Number:
     
  2. ciphervendor

    ciphervendor Active Member

    Joined:
    Aug 26, 2002
    Messages:
    1,052
    Likes Received:
    0
    Trophy Points:
    36
    Well I think you answered your own question. It sounds like something is munching all of your RAM/SWAP and the machine is falling over. Add more RAM and I'm sure the issue will go away.

    cPanel.net Support Ticket Number:
     
  3. nyjimbo

    nyjimbo Active Member

    Joined:
    Jan 25, 2003
    Messages:
    1,125
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    New York
    If you can, run a "TOP" and look for zombies. If you see alot of them then you have a bad apache or bad frontpage install.

    We had a similiar problem and noticed hundreds of zombies from author.exe and fpcount.exe, all FP stuff and had to do a kill -1 (pid) every couple hours until we had a fix for the bad frontpage extensions.

    cPanel.net Support Ticket Number:
     
  4. neilh

    neilh New Member

    Joined:
    May 30, 2003
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Thanks so much for the help. We are looking into getting more ram or moving to a new machine.

    I am left with more questions though. Even if the server had a burst of memory demanding traffic, shouldn't it be able to handle this in ways that don't involve the whole machine shutting down needing a restart? What about sending errors to requests saying that the server is temp. unavail. etc.

    Right now there are no appearent zombees. There is no way to find out what happened with RAM/CPU at a given time of day in the logs is there? I imagine that storing that much information would be a log overkill. The apache logs are not telling us very much. Where else can one look for clues?

    Thanks again,
    Neil

    cPanel.net Support Ticket Number:
     
  5. ciphervendor

    ciphervendor Active Member

    Joined:
    Aug 26, 2002
    Messages:
    1,052
    Likes Received:
    0
    Trophy Points:
    36
    Sure, some processes won't be able to fork and you will see out of memory errors in dmesg and maybe some in messages depending on the process. However, if an already present process on the box grows, it will kill your box. That's the way the box is supposed to perform, if you'd like to kill processes that take x amount of ram or processor cycles, then find/buy (http://www.web-bazaar.com/cgi/process_doctor.html) or create a process watching daemon.

    You can install some scripts, etc. to monitor your servers resource usage. A popular script is http://hotsanic.sourceforge.net/

    cPanel.net Support Ticket Number:
     

Share This Page