Recovery from Server Kernel Update corruption after WHM update

[martin MHC]

Hello
I received a WHM update notification this morning (28th April) and this ran and then offered to WHM update the Kernel and reboot as usual

I ran the update/reboot and the server lost contact -- SSH, Web, FTP all lost connection. After contacting the Hosting providers (Fasthosts) they manaully entered the server and established that the Kernel that was used to boot was corrupted and so then they reset the Server to load from an earlier Kernel.

The server now loads and runs ok, but:-

1) I need to ensure that the bad/corrupt kernel has been removed and/or cleanly reinstalled.
2) I had previously found an incomplete yum transaction which I cleaned up. Re running yum update + tidy up does not claim to need to do anything new now.
3) Fasthosts offered me a Server kernel updating system (El Repo) which is not supported by cPanel so I do not want to install it: https://www.howtoforge.com/tutorial/how-to-upgrade-kernel-in-centos-7-server/
4) The current install of WHM still asks me to reboot the system but the Fasthost engineer states that rebooting the server will revert to the most recent but corrupt kernel.

Kernel version currently operating ok:
3.10.0-1062.4.3.el7.x86_64

Kernel versions currently on the system:
kernel-3.10.0-1062.12.1.el7.x86_64
kernel-3.10.0-1062.9.1.el7.x86_64
kernel-3.10.0-1062.7.1.el7.x86_64
kernel-3.10.0-1127.el7.x86_64
kernel-3.10.0-1062.18.1.el7.x86_64
kernel-3.10.0-1062.4.3.el7.x86_64

yum reinstall kernel
wants to reinstall kernel-3.10.0-1127.el7.x86_64

How can I ensure that the current "current" kernel is stable and usable before running the WHM update process as outlined in (3)?

It might all be fine, but I don't want to risk loosing remote access to the server again. Would appreciate guidance on next steps. I am not signed up with KernelCare.

Cheers

 

[cPanelLauren]

Keep in mind while cPanel and WHM will let you know when you have a new kernel available it does not have control over the kernel or version.

You could reinstall the kernel, the likelihood that you'd receive a corrupted kernel is minimal, you could also wait until the next kernel version is released. You need to be able to update the kernel though, so I'd suggest identifying what the issue actually was, in many cases, this happens as a result of a kernel panic which is in many cases the result of an incompatibility with some process running on the server. In the event the kernel version you were installing was corrupt you think that others would be experiencing the issue as well.

 

[martin MHC]

Keep in mind while cPanel and WHM will let you know when you have a new kernel available it does not have control over the kernel or version.

You could reinstall the kernel, the likelihood that you'd receive a corrupted kernel is minimal, you could also wait until the next kernel version is released. You need to be able to update the kernel though, so I'd suggest identifying what the issue actually was, in many cases, this happens as a result of a kernel panic which is in many cases the result of an incompatibility with some process running on the server. In the event the kernel version you were installing was corrupt you think that others would be experiencing the issue as well.

Yes, you remind me that the Fasthost engineer said there was a "Kernel Panic" . I don't know anything about such things so will read up on this today.

Does it sound reasonable (as far as you know) that by simply correcting the yum incomplete transaction this would have ironed out the original issue? Or is the yum transaction also a symptom rather than a cause?

Thanks in advance.

 

[dalem]

Does it sound reasonable (as far as you know) that by simply correcting the yum incomplete transaction this would have ironed out the original issue? Or is the yum transaction also a symptom rather than a cause?

Is possible but no real way to know unless you reboot it . It could a be panic error specific to your hardware & Red hat has not addressed the issue yet.

if you have KVM/IPMI access you could try it and just reboot in back to the working kernel

Best to reinstall the effected kernel first then try

 

[cPanelLauren]

Exactly this ^^

Alternatively you could wait 1 version but it might be best to work with your provider once you reinstall the kernel (have them available when you reboot if necessary)

 

[keat63]

I had exactly this issue about 6 months ago, and also with FastHosts, although I'm not sure FastHosts is relevent.
Was told that it was a kernel panic.
I never got a true answer as to what caused it, other than to wait it out until the next kernel version.

Nerves got the better of me, so I avoided any kernel updates after that.

All due respect to FastHosts, they were on the ball and had the server back up within about 10 minutes of my call.

 

[cPanelLauren]

Yea, I don't think the hosting provider is as relevant. I do think that the reason for a kernel panic should be investigated and diagnosed @keat63 I do hope you're updating your kernel now :O

 

[dalem]

Like I said usually is a hardware compatibility issue but on occasion
as
cPanelLauren said it can be software related

kernel updates have been 99.9 % reliable over the last many years but occasionally you get the one that mucks up

 

[martin MHC]

I have just rebooted the server with the WHM kernel update as per the suggestion on the top right of the WHM screen. The reboot worked successfully, so the diagnosis was that for some reason the yum update (that was previously incomplete for some reason yet to be explored) had caused a Kernel Panic with the system trying to use the corrupted/incomplete kernel downloaded via yum.

Thanks all for your help and guidance on this topic.

Cheers

p.s> The Kernel this relates to is: 3.10.0-1127.el7.x86_64

 

[martin MHC]

All due respect to FastHosts, they were on the ball and had the server back up within about 10 minutes of my call.

Yes, Fasthosts had KVM access to the server that I did not have, they operated quickly and gave me useful feedback.

HOWEVER: It needs to be noted that the Fasthosts user KVM system is absolutely insecure and unsafe (invalid TLS certification, various additional certification flaws) and should not be used. If you need KVM access, contact the Fasthosts support team directly.

 

[piyushmaheswari]

I will suggest to contact the support of Operating System .
Some days ago my Kernal and OS got fully corrupted , I just contacted the support of OS ( CloudLinux ) they solved it .

 

[360webfirm]

Good day to you all.

After a security advisor scan, it mentioned I should update the kernel as I ahve done time and time again without no issues. This last update ( see below ) caused an issue where I could not get the kernel up after I yum update and rebooted server.

This is what I followed:

The system kernel is at version “3.10.0-1127.10.1.el7.x86_64”, but an update is available: 3.10.0-1127.13.1.el7.x86_64
Update the system (run “yum -y update” on the command line), and reboot the system.

I did an update like I have done several times before and this time I cannot load my server after I did a Yum Update and then rebooted server and now it will not reload. I cannot even ping the server.

I believe it's an issue with Grub not recognizing the new kernel. I am not the best with this stuff and I am still learning all the time.

I have tried a few things but I cannot get it to load.

If anyone has a fix for this or a solution, it would be greatly appreciated. I know some, but I am no expert with system admin. I really appreciate anyone who can help me.

Thanks so much in advance.

 

[andrew.n]

Do you have KVM or console access to the server? Do you know why the new kernel is not booting? Is it a kernel panic? You can hire cPanel certified system admins from here System Administration Services

 

[cPanelLauren]

If you don't have console access you'll need to talk to your Datacenter to find out if they can provide you assistance with booting into a supported kernel.