Kernel Paniced HELP!!

Think of it like a Windows Blue Screen Of Death. A system error that caused the system to essentially choke (or panic) and freeze.
It doesn't happen very often with Linux, but it does happen. See if you can find any triggers in your system logs that may have set it off.

 

Kernel panic happens when the server restart after the kernel upgrade , server will load the linux kernel, if the kernel does not have matching hardware support it will panic,

For example if the kernel configured with different cpu then your server cpu it will panic. thats one of the example but it can panic for many reason.

Normally happen if you upgrade your kenrel and your new kernel missing some hardware support for the server,

hope that helps

 

A kernel panic is a message displayed by an operating system upon detecting an internal system error from which it cannot recover. Kernel panics often provide cryptic debugging information that is useful only to the developers of the operating system.

Attempts by the operating system to read an invalid or unpermitted memory address are a common source of kernel panics. A panic may also occur as a result of a hardware failure.

 

Re kernel panics, there's a flag you can set to restart the server automatically after a kernel panic. Without this, the box will panic and just sit there hoping you or someone else will come along and look at the console messages and restart it. This could be a long wait overnight!

To setup the panic reboot timer:

Code:

echo kernel.panic = 120 >> /etc/sysctl.conf

This will ensure that the system will reboot 120 seconds after a panic rather than just staying shut down.

You can change the live kernel setting on the fly by reading or writing to /proc/sys/kernel/panic. For instance, "cat /proc/sys/kernel/panic" will display 0 usually, "echo 120 > /proc/sys/kernel/panic" will modify the current value.

The above works for Centos, I haven't tried it anywhere else, and unfortunately since setting it up a month ago I haven't had a system crash.

Also, there's a thing called netdump available that should be able to show why a server crashed by logging it to another server when the crash occurs. I don't know the details of setting it up yet, my initial half-arsed attempt failed and I haven't been able to try again. This would work well coupled with the above panic timer as you should then be able to see why the crash occurred, and then recover from it automatically.

 

Does this command list a value for kernel.panic?

Code:

sysctl -a | grep -a panic

Does the file /proc/sys/kernel/panic exist with a value in it?

If the answer to both is yes, then probably it does. Checking these two things will probably show you whether your particular variant supports something like this very quickly, even if you're not on Fedora. I'm not sure whether the BSD family support this or not, but they're likely to have a mechanism that functions similarly.

One thought/caution - I have a script that emails me on reboot. You should probably have a script like that in place if you set kernel.panic as otherwise the system could crash and reboot regularly and you'd have little or no idea that it's happening.

BTW, to add to what AndyReed said before - I have a machine failing every month or so, I suspect it's a problem with memory, ie that's another possible cause for panics. My panics only happen during the nightly cpbackup session where there's loads of disk I/O and lots of memory usage.

 

That tells you kernel.panic isn't set, as well as telling you that it's likely your kernel supports it.

Better to use cat /proc/sys/kernel/panic to just see what it's currently set to, or use the above sysctl command .

That wasn't meant to change anything, other than the default for next time you reboot! Ie: it did exactly what it should have!

I would have preferred: echo 120 > /proc/sys/kernel/panic

Perhaps the /proc files don't like pico; they're not normal files despite looking like they are.

You mostly want to use "echo" to write to them (or cat) and to use "cat" to display the contents of a /proc file.

 

Brian, thanks for the interesting explanation of this topic