Kernel Paniced HELP!!

[AnthonyCaesar]

My server dissapeared from the internet I couldn't log into WHM or ping it. They couldn't even ping it at the data center its colocated at. They told me the kernel paniced and a reboot seemed to fix the problem now its up and running again.

What does it mean - kernel paniced?

From what I think I know and I admit I don't know much a panic shouldn't have caused the server to dissapear.

 

[NightStorm]

Think of it like a Windows Blue Screen Of Death. A system error that caused the system to essentially choke (or panic) and freeze.
It doesn't happen very often with Linux, but it does happen. See if you can find any triggers in your system logs that may have set it off.

 

[xerophyte]

Kernel panic happens when the server restart after the kernel upgrade , server will load the linux kernel, if the kernel does not have matching hardware support it will panic,

For example if the kernel configured with different cpu then your server cpu it will panic. thats one of the example but it can panic for many reason.

Normally happen if you upgrade your kenrel and your new kernel missing some hardware support for the server,

hope that helps

 

[AndyReed]

What does it mean - kernel paniced?

A kernel panic is a message displayed by an operating system upon detecting an internal system error from which it cannot recover. Kernel panics often provide cryptic debugging information that is useful only to the developers of the operating system.

Attempts by the operating system to read an invalid or unpermitted memory address are a common source of kernel panics. A panic may also occur as a result of a hardware failure.

 

[brianoz]

Re kernel panics, there's a flag you can set to restart the server automatically after a kernel panic. Without this, the box will panic and just sit there hoping you or someone else will come along and look at the console messages and restart it. This could be a long wait overnight!

To setup the panic reboot timer:

echo kernel.panic = 120 >> /etc/sysctl.conf

This will ensure that the system will reboot 120 seconds after a panic rather than just staying shut down.

You can change the live kernel setting on the fly by reading or writing to /proc/sys/kernel/panic. For instance, "cat /proc/sys/kernel/panic" will display 0 usually, "echo 120 > /proc/sys/kernel/panic" will modify the current value.

The above works for Centos, I haven't tried it anywhere else, and unfortunately since setting it up a month ago I haven't had a system crash.

Also, there's a thing called netdump available that should be able to show why a server crashed by logging it to another server when the crash occurs. I don't know the details of setting it up yet, my initial half-arsed attempt failed and I haven't been able to try again. This would work well coupled with the above panic timer as you should then be able to see why the crash occurred, and then recover from it automatically.

 

[AnthonyCaesar]

does the server panic timer work for fedora

 

[brianoz]

does the server panic timer work for fedora

Does this command list a value for kernel.panic?

sysctl -a | grep -a panic

Does the file /proc/sys/kernel/panic exist with a value in it?

If the answer to both is yes, then probably it does. Checking these two things will probably show you whether your particular variant supports something like this very quickly, even if you're not on Fedora. I'm not sure whether the BSD family support this or not, but they're likely to have a mechanism that functions similarly.

One thought/caution - I have a script that emails me on reboot. You should probably have a script like that in place if you set kernel.panic as otherwise the system could crash and reboot regularly and you'd have little or no idea that it's happening.

BTW, to add to what AndyReed said before - I have a machine failing every month or so, I suspect it's a problem with memory, ie that's another possible cause for panics. My panics only happen during the nightly cpbackup session where there's loads of disk I/O and lots of memory usage.

 

[AnthonyCaesar]

should this script be in php cgi or a c language im starting to become very uneasy about my data center i have choosen it has happened again that twice in three days. I have not done any backups or anything in that time. Nothing that would have caused this.

 

[AnthonyCaesar]

I ran
sysctl -a | grep -a panic
and got
kernel.panic_on_oops = 0
kernel.panic = 0

I ran
pico /proc/sys/kernel/panic
and it had "0" in it

and the I ran
echo kernel.panic = 120 >> /etc/sysctl.conf
this changed changed nothing.

but when I
pico etc/sysctl.conf
kernel.panic = 120

but when I
sysctl -a | grep -a panic
I still get
kernel.panic_on_oops = 0
kernel.panic = 0

 

[brianoz]

I ran
sysctl -a | grep panic
and got
kernel.panic_on_oops = 0
kernel.panic = 0

That tells you kernel.panic isn't set, as well as telling you that it's likely your kernel supports it.

I ran
pico /proc/sys/kernel/panic
and it had "0" in it

Better to use cat /proc/sys/kernel/panic to just see what it's currently set to, or use the above sysctl command .

and the I ran
echo kernel.panic = 120 >> /etc/sysctl.conf
this changed changed nothing.

That wasn't meant to change anything, other than the default for next time you reboot! Ie: it did exactly what it should have!

but when I
pico etc/sysctl.conf
kernel.panic = 120

I would have preferred: echo 120 > /proc/sys/kernel/panic

Perhaps the /proc files don't like pico; they're not normal files despite looking like they are.

You mostly want to use "echo" to write to them (or cat) and to use "cat" to display the contents of a /proc file.

 

[gorilla]

Brian, thanks for the interesting explanation of this topic

 

[AnthonyCaesar]

I apreciate everyones help on this i did all that was explained working on the script to tell me when i panic. This last week of panics have caused my uptime to drop to 99.52%.

Any suggestions on the script. Im a webdeveloper that has been put (thrown) in charge of our server (unwillingly) and I am still learning about Linux Apache an WHM.