SOLVED 'You must reboot the server to apply kernel updates' persists after multiple reboots

[aztopdavid]

My VPS is running v86.0.18 (RELEASE) on CENTOS 7.8 and I have a persistent "You must reboot the server to apply kernel updates" warning that doesn't go away after rebooting (several times). The latest "yum -y update" results in "No packages marked for update' and "whmapi1 system_needs_reboot" produces:

kernel:
boot_version: 3.10.0-1062.18.1.el7.x86_64
running_version: 3.10.0-862.14.4.el7.x86_64
needs_reboot: 1
metadata:
command: system_needs_reboot
reason: OK
result: 1
version: 1

"rpm -qa |grep kernel" results:
kernel-3.10.0-1062.18.1.el7.x86_64
kernel-3.10.0-957.27.2.el7.x86_64
kernel-headers-3.10.0-1127.el7.x86_64
kernel-devel-3.10.0-862.14.4.el7.x86_64
kernel-3.10.0-862.14.4.el7.x86_64
kernel-devel-3.10.0-1062.12.1.el7.x86_64
kernel-3.10.0-1062.12.1.el7.x86_64
kernel-devel-3.10.0-1062.18.1.el7.x86_64
kernel-3.10.0-1127.el7.x86_64
kernel-devel-3.10.0-1127.el7.x86_64
kernel-devel-3.10.0-1062.9.1.el7.x86_64
texlive-l3kernel-svn29409.SVN_4469-45.el7.noarch

and finally "uname -r" produces:
3.10.0-862.14.4.el7.x86_64

I'm not sure if this issue will just go away on its own or if I need to do something to make the warning disappear? Thanks!

 

[GOT]

My guess is you have an lxc based vm. Known issue. You can turn this notice off in tweak settings but we haven't found a fix. Seems endemic with lxc.

 

[aztopdavid]

I just ran "systemd-detect-virt" and the result was "kvm" so it doesn't look like my host is using lxc

 

[GOT]

Looking closer at your initial post, your system is not booting into the latest kernel, that is why you are getting the message. Latest is

1062.18.1

Whereas your running is

862.14.4

I would take a look here

How to set default boot kernel on Linux ( CentOS / RHEL 7 ) | GoLinuxCloud

How to set default boot kernel in linux? Change the default boot entry for kernel ? Boot with old kernel version ? Revert to previous kernel version ?

www.golinuxcloud.com

To see about setting your system to boot the latest kernel.

 

[aztopdavid]

Thanks, I saw that in an older thread, but wasn't sure if that would be an advisable solution for me. Perhaps I should just ask the host if/when they plan on doing system maintenance involving rebooting the node?

 

[GOT]

Doesn't really involve rebooting the node. For some reason your system is not flagging new kernels to be utilized, which is normally the default behavior, but this does not require a node reboot to do so.

 

[aztopdavid]

OK, I'm working my way through the steps in "How to set default boot kernel on Linux" but it's not going all that smoothly. When I ran:

# awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg

I got:

CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)
CentOS Linux 7 (Core)

However, when I poked around in the grub2.cfg file, looking at the details of the "menuentry" values in the "/etc/grub.d/10_linux" section, I found these, which include eight different "submenu" choices for that generic "CentOS Linux 7" value from the output above:

menuentry 'CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)'
    linux16 /boot/vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/sda1 ro LANG=en_US.UTF-8
    initrd16 /boot/initramfs-3.10.0-1127.el7.x86_64.img
menuentry 'CentOS Linux 7 (Core)'
    linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
    initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img
submenu 'Advanced options for CentOS Linux 7 (Core)'
   menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core)'
        linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
        initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img
   menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core) (recovery mode)'
        linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single
        initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img
   menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core)'
        linux16 /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
        initrd16 /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img
   menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core) (recovery mode)'
        linux16 /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single
        initrd16 /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img
   menuentry 'CentOS Linux (3.10.0-957.27.2.el7.x86_64) 7 (Core)'
        linux16 /boot/vmlinuz-3.10.0-957.27.2.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
        initrd16 /boot/initramfs-3.10.0-957.27.2.el7.x86_64.img
   menuentry 'CentOS Linux (3.10.0-957.27.2.el7.x86_64) 7 (Core) (recovery mode)'
        linux16 /boot/vmlinuz-3.10.0-957.27.2.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single
        initrd16 /boot/initramfs-3.10.0-957.27.2.el7.x86_64.img
   menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core)'
        linux16 /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
        initrd16 /boot/initrd-3.10.0-862.14.4.el7.x86_64.img
   menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core) (recovery mode)'
        linux16 /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single
        initrd16 /boot/initrd-3.10.0-862.14.4.el7.x86_64.img

So to my untrained eye, it appears that the first entry results in booting 1127, while the second option basically refers to the following three submenu choices:

1062.18.1
957.27.2
862.14.4

and as you mentioned before, there's a mismatch between the boot and the running version that's not currently being resolved by rebooting:

boot_version: 3.10.0-1062.18.1.el7.x86_64
running_version: 3.10.0-862.14.4.el7.x86_64

But if I successfully apply the procedures mentioned in "How to set default boot kernel?" might that result in running 1127 instead of 1062.18.1? Also, "/etc/default/grub" (mentioned in the article) doesn't exist. I'm not sure the article covers a setup with all the different "submenu" choices present on my system -- it appears to deal with a configuration with discrete "menuentry" values to choose from.

 

[GOT]

1127 is the very newest kernel for Cent7 so no suprise there. You've got a lot of kernels installed. It may be less confusing for you if you run

package-cleanup -y --oldkernels --count=3

Which will remove all but the latest kernels.

 

[aztopdavid]

That appears only to have removed 957.27.2

 

[GOT]

Does your host provide you a kvm console?

 

[aztopdavid]

They have what looks to be a proprietary cloud dashboard. I've just logged in there and am rebooting using that interface, instead of from inside WHM.

 

[aztopdavid]

So, that didn't do it, after rebooting in my host's dashboard and logging back into WHM, I still had the "You must reboot..." warning, so I also did a Graceful Reboot in WHM and it's still there. I suppose I could sign up for KernelCare.

 

[GOT]

Does your host provide a console where you can see the server rebooting as if you were in front of it, if so you could select the kernel it boots into.

Output of this please:

cat /etc/sysconfig/kernel

 

[GOT]

I suspect a few things, but the article should help you. Assuming you have

grep GRUB_DEFAULT /etc/default/grub
GRUB_DEFAULT=saved

Then running

grub2-set-default 0

Then run

grub2-mkconfig -o /boot/grub2/grub.cfg

Then reboot.

If the grub_default setting above is not "saved" then change it before running the other commands.

 

[aztopdavid]

cat /etc/sysconfig/kernel = "No such file or directory"

I also already mentioned that /etc/default/grub doesn't exist, which is partly why I'm stuck.

There's a "console" button in the host interface and I'm running it, but it appears to give me the same view as my PuTTY session with su root

 

[GOT]

if you have the console open while its booting you might (might) see a centos boot menu that lets you pick your kernel.

If you are missing those files something is wacky with your set up and I could not help you further without risking a non-bootable system.

 

[aztopdavid]

Yes, a non-bootable system is what I'm trying to avoid. Here's what I saw in the console during rebooting - note that there's only the one choice:

 

[GOT]

Yeah that makes more sense, the lernels are not getting installed correctly.

I don't really know how to advise you here. You could ask your host it could be something they are doing on deployment for stability reasons.

If you don't want to move servers, get kernelcare to keep the kernel patched and turn off the reboot notices in tweak settings.

 

[aztopdavid]

Thanks! I'm actually shopping for a new host -- I'd be happy for anyone's suggestions, as long as that doesn't violate the forum policies.

 

[GOT]

Others can. I cannot.