Server keeps crashing

[GrAfiX]

Server keeps crashing need help

I have a hosted server and the server keeps crashing. The server is hosted on a farm so I can only remotely administrate it. There are no Kernel panics that I can see since the kernel log never gets created when it crashes. However in the messages log I see tons of wierd activity. There are a lot of attempted hacks going on and every time the server dies the last few entries in the messages log and there are a bunch but they look like this.

Oct 12 18:09:43 ns1 pure-ftpd: ([email protected]) [INFO] Logout.
Oct 12 18:19:08 ns1 pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Oct 12 18:19:10 ns1 pure-ftpd: ([email protected]) [INFO] Logout.
Oct 12 18:27:59 ns1 pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Oct 12 18:28:00 ns1 pure-ftpd: ([email protected]) [INFO] Logout.
Oct 12 18:38:51 ns1 pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Oct 12 18:38:52 ns1 pure-ftpd: ([email protected]) [INFO] Logout.
Oct 12 18:47:24 ns1 pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Oct 12 18:47:26 ns1 pure-ftpd: ([email protected]) [INFO] Logout.
Oct 12 18:56:43 ns1 pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1
Oct 12 18:56:44 ns1 pure-ftpd: ([email protected]) [INFO] Logout.
Oct 12 21:32:17 ns1 syslogd 1.4.1: restart.


The FTP logins happen like this for pages and pages and you can see the last entry before the restart is always a logout.

I just switched to the prue-ftpd last night to see if that stopped it but obviously not since it was doing the same thing with proftpd. What would be causing these connections from local host???

Also when running "top" nothing out of the ordinary is happening when the server locks. I will get a screen shot the next time it happens.

I'm trying to get the NOC to update to Kernel 2.6xx what ever the latest version is since up2date only grabes 2.4 something.

Is there anyone that can help?? Where else could I look for problems without being in front of the terminal to see whats happening there?? I know some Linux but really just enough to get me in trouble.

Thanks in advance for any help you can provide me with.
Mike

 

[GrAfiX]

Oh a couple more things... The NOC replaced the MB, RAM, Processor and NIC to try and eliminate the problem.

the server is RH9
Linux ns1.hostbulb.com 2.4.20-31.9 #1 Tue Apr 13 17:38:16 EDT 2004 i686 athlon i386 GNU/Linux

I do have a couple very busy sites on the box but not enough to bring it to its knee's I wouldn't think.

 

[GrAfiX]

Here's what the NOC said..

Subject:
Hello,

If you stroll through /var/log/messages you will find a number of
issues. One of which is xinetd constantly complaining about
non-accessible binaries being called in its configs, specifically for
'talk', 'telnet', and 'ntalk'.. all 3 of those in /etc/xinetd.d/ have no
path on line 8 of the file before the binary, so I assume that's the
problem (shouldn't be 'in.ntalkd', should be '/sbin/in.ntalkd' or
wherever it is). These may or may not be related to the 'crashes' you're
experiencing.

Also, there are a lot of errors related to portsentry; it is possible
those are related to this as well, but not neccessarily.

I can find no logs however that point at a hardware problem. I would
suggest starting by fixing the problems with portsentry (which I've
never seen have errors like this on a cPanel box, so I assume you
installed/modified portsentry) and your xinetd.d entries that I assume
you added as those also I don't recall as being standard. As they're not
standard, I wouldn't feel comfortable modifying them, nor is it really
supported.

I took the liberty of installing the 'sysstat' RPM on your server, which
includes a backend cronjob and a client called 'sar', which basically
runs every few minutes and pulls a snapshop of your system state (CPU,
RAM, HD usage, etc); perhaps if the problem continues looking at the
last entry in sar prior to the crash might shed some light on what's
going on.

If the issue continues, please let us know, but at present I can find no
logs specifically causing issues nor has it crashed while I was on or
any time in the past number of hours.

I wouldn't have the first clue about how to modify the stuff they said I did basically if I can't do it through whm or simple cpanel scripts then it hasn't been done. And really have not changed anything there either.

Can anyone point me in the right direction??

 

[GrAfiX]

anyone????

 

[chirpy]

The stuff about xinetd, though valid, is the same on any Linux cPanel server and would have absolutely nothing to do with your server crashing.

More information would be helpful:

What OS and version are you running and what kernel version are you running? The 2.4 kernel might be perfectly valid depending on your OS, but the full version would help:

uname -a

I would suspect that portsentry has absolutely nothing to do with it either. The above information would help as a starting point.

Another thing you could consider installing is PRM, incase you have runaway processes:
http://r-fx.org/prm.php

Another is to make sure that you have WHM > Shell fork bomb protection enabled, incase you have runaway background or CGI processes.

Lastly, it would help to have your NOC look at the actual server console port after a crash for the actual reason. Log files can be useless if they can't be written to, but the console nearly always has some reason for a crash displayed on it.

 

[sky]

actually i am also having the exact problem.

there is no processes that was using alot of resources and such. It is as if, someone went to cut the power supply off suddenly. no panic messages etc.

This only happened after I upgraded to latest 9.9.8 - s6

any idea?

 

[challii]

if you can, can you try running top, and seeing if anything weird happens when the load suddenly shoots through the roof. Our server had a problem yesterday (not sure if its still there ... PRM hasnt complained yet today!) But basically what was happening was the number of processes was suddenly shooting through the roof from ~200 processes to 600 processes!

The server simply couldnt cope with this and would just struggle until the proccesses were killed or died. I think it might be due to a dodgy bandwidth monitor, but thats only what I think.

is there any way you can list all the proccesses ? would be quite interested to see whats running.

 

[sky]

the funny thing is that there isn't process overload.

it was ok for a few days and it came back again. worst still, the server is shutoff just like that ..... i am still wondering ... why it happen after the upgrade and not before.

 

[sky]

ok some updates.....

i found out it was due to my fan. Apparently, the CPU heat up as there is no cooling and motherboard just shuts off power. Safety mechanism.

I will see since it has been 2 days since replacement of cpu fan and everything is ok now.

Sometimes it ain't always software .... but sad to say most of us jump at what is simplest ... lol ... hopefully it is just the cpu fan.... i am happy about it :P

 

[rhenderson]

?? Any Luck

Oct 12 18:09:43 ns1 pure-ftpd: ([email protected]) [INFO] Logout.
Oct 12 18:19:08 ns1 pure-ftpd: ([email protected]) [INFO] New connection from 127.0.0.1

Mike

I have been experiencing the same problem at 0000 everynight, SIM restarts the http but it always seems to be related to Pure-Ftp...
Was wondering if you ever figured anything out?

Thanks

 

[nickp666]

if you can, can you try running top, and seeing if anything weird happens when the load suddenly shoots through the roof. Our server had a problem yesterday (not sure if its still there ... PRM hasnt complained yet today!) But basically what was happening was the number of processes was suddenly shooting through the roof from ~200 processes to 600 processes!

The server simply couldnt cope with this and would just struggle until the proccesses were killed or died. I think it might be due to a dodgy bandwidth monitor, but thats only what I think.

is there any way you can list all the proccesses ? would be quite interested to see whats running.

to see all running processes, run:

ps -e | more