server crash mystery

[neilh]

This has happened a number of times in the past week, and we can't figure out what the souce of the problem is.

Our server goes through a complete break down, one service after the next, seemingly starting with mysql. From there, apache goes out and we can't get in w/ ssh. The server has to be physically restarted.

The first errors we can see in the logs are from php scripts that connect to mysql:

mysql_connect(): Can't create a new thread (errno 11). If you are not out of available memory, you can consult the manual for a possible OS-dependent bug in....

during this time perl scripts connecting to myql give the error:

Resource temporarily unavailable: couldn't spawn child process:

after a few minutes of this, the whole server goes down and we have to call the hosting company to manually restart. Our server has 256 RAM, which may be part of the problem, though most of the time the ram is at less than 30%. Also, one would hope that maxing out the ram would have less dire consequences.

Any ideas would be very helpful.

Neil

cPanel.net Support Ticket Number:

[ciphervendor]

Well I think you answered your own question. It sounds like something is munching all of your RAM/SWAP and the machine is falling over. Add more RAM and I'm sure the issue will go away.

cPanel.net Support Ticket Number:

[nyjimbo]

If you can, run a "TOP" and look for zombies. If you see alot of them then you have a bad apache or bad frontpage install.

We had a similiar problem and noticed hundreds of zombies from author.exe and fpcount.exe, all FP stuff and had to do a kill -1 (pid) every couple hours until we had a fix for the bad frontpage extensions.

cPanel.net Support Ticket Number:

[neilh]

Thanks so much for the help. We are looking into getting more ram or moving to a new machine.

I am left with more questions though. Even if the server had a burst of memory demanding traffic, shouldn't it be able to handle this in ways that don't involve the whole machine shutting down needing a restart? What about sending errors to requests saying that the server is temp. unavail. etc.

Right now there are no appearent zombees. There is no way to find out what happened with RAM/CPU at a given time of day in the logs is there? I imagine that storing that much information would be a log overkill. The apache logs are not telling us very much. Where else can one look for clues?

Thanks again,
Neil

cPanel.net Support Ticket Number:

[ciphervendor]

Sure, some processes won't be able to fork and you will see out of memory errors in dmesg and maybe some in messages depending on the process. However, if an already present process on the box grows, it will kill your box. That's the way the box is supposed to perform, if you'd like to kill processes that take x amount of ram or processor cycles, then find/buy (http://www.web-bazaar.com/cgi/process_doctor.html) or create a process watching daemon.

You can install some scripts, etc. to monitor your servers resource usage. A popular script is http://hotsanic.sourceforge.net/

cPanel.net Support Ticket Number: