I keep getting the "You must reboot the server to apply software updates"

Operating System & Version
AlmaLinux 8
cPanel & WHM Version
108.0.13

Hefin

Member
Feb 23, 2021
17
3
3
London
cPanel Access Level
Website Owner
I have a fresh install of WHM on AlmaLinux 8.

Prior to installation, I ran the command something like 'sudo dnf update' to update the packages on the box.

After the WHM install, now in the WHM dashboard, I get two notifications to 'You must reboot the server to apply software updates'

1677738974274.png

I have rebooted the server twice but the notifications remain.
 
Last edited by a moderator:

quietFinn

Well-Known Member
Feb 4, 2006
1,899
465
438
Finland
cPanel Access Level
Root Administrator
When you ran 'sudo dnf update' did it update the kernel?
You could run that command again.
 

Hefin

Member
Feb 23, 2021
17
3
3
London
cPanel Access Level
Website Owner
Yes! @cPRex! The Man, The Myth, The Legend! Never Fear cPRex is here! :cool: Hope you're well.

I have run the command `sudo dnf list kernel` to check the list of available kernel versions on the server

Output
Installed Packages
kernel.x86_64 4.18.0-372.9.1.el8 @baseos
kernel.x86_64 4.18.0-425.10.1.el8_7 @baseos
kernel.x86_64 4.18.0-425.13.1.el8_7 @baseos

Running `sudo grubby --default-kernel` in order to check which kernel which is selected to be run on server boot:

Output
`/boot/vmlinuz-4.18.0-425.13.1.el8_7.x86_64`

which is the latest kernel that I'm expecting to be running.

Running the command `sudo dnf updateinfo`

Output
Security: kernel-core-4.18.0-425.13.1.el8_7.x86_64 is an installed security update
Security: kernel-core-4.18.0-425.10.1.el8_7.x86_64 is the currently running version

It appears the server is booting to run: * `kernel-core-4.18.0-425.10.1.el8_7.x86_64`

instead of the latest kernel * `kernel-core-4.18.0-425.13.1.el8_7.x86_64`

Issuing the command `uname -r` to get the running kernel I get

`4.18.0-425.10.1.el8_7.x86_64` :confused:

I'm expecting `kernel-core-4.18.0-425.13.1.el8_7.x86_64` to be running on reboot. Any help is much-appreciated cPRex
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
15,280
2,434
363
cPanel Access Level
Root Administrator
Thanks for investigating that. That would confirm why the "needs reboot" message is still appearing in WHM.

I'm not sure why the new kernel wouldn't have been automatically applied to the server as part of your previous updates and reboots. To avoid potential issues with booting into the wrong or incompatible kernel, I recommend contacting your hosting provider or data center for assistance with investigating this particular issue.
 

Hefin

Member
Feb 23, 2021
17
3
3
London
cPanel Access Level
Website Owner
Thanks @cPRex. Unfortunately, the hosting provider's responsibility lies only with the hardware for the server, occasionally my hosting support team will help with software issues/config and troubleshooting. But with this server, the hardware appears to be functioning correctly, and the OS software issue is fairly low level. At the moment the issue to appears to lie with AlmaLinux OS not booting with the latest confined kernel. Hoping a forum member or staff might be able to suggest a workaround/solution.

At the moment I'm at a loss as to how to remedy this, as I'm unsure about the maintainability of the server for any future updates and new kernel updates f the server appear stuck on an older Kernel version. A support ticket for this issue has been raised too. cPanel Support team may be able to propose a solution.
 

Hefin

Member
Feb 23, 2021
17
3
3
London
cPanel Access Level
Website Owner
Thanks for the follow up @cPRex. You're awesome! :cool:

This was a weird one, at the moment still troubleshooting.

So at the moment, I'm in the process of setting up this dedicated server, I thought I'd simplify this issue and while I have the opportunity its not in product I thought to myself let's just re-image the server with a nice clean AlmaLinux 8 OS . That will remove any ambiguity of WHM/cPanel from the equation, (it didn't look like WHM/cPanel was doing anything to get in the way of OS kernel updates anyhow but if I can simplify this I thought it better too). You prompted earlier it could be a server issue, which at the time I thought unlikely, now I think is more likely.

So the server was given a brand spanking new copy of AlmaLinux 8 this morning, untouched by my grubby hands (or anyone else's for that matter)

Following the new image of AlamaLinux 8 that was applied, these are the only commands that have been run on this box

Get currently running Kernel version:

COMMAND: uname -r
---------------------------
OUTPUT: 4.18.0-425.10.1.el8_7.x86_64


Get the list of default kernel selected to load from the grublist/bootloader

COMMAND: sudo grubby --default-kernel
-----------------------------
OUTPUT: /boot/boot/vmlinuz-4.18.0-425.10.1.el8_7.x86_64


Get list of Kernel versions for OS

COMMAND:
sudo dnf list kernel
-----------------------
OUTPUT:
Last metadata expiration check: 0:23:20 ago on Mon 06 Mar 2023 10:25:43 AM UTC.
Installed Packages
kernel.x86_64 4.18.0-348.12.2.el8_5 @baseos
kernel.x86_64 4.18.0-372.9.1.el8 @baseos
kernel.x86_64 4.18.0-425.10.1.el8_7 @baseos
Available Packages
kernel.x86_64 4.18.0-425.13.1.el8_7 @baseos **kernel version, to update to**


Update Packages and Kernels

COMMAND:
sudo dnf update

OUTPUT:
Last metadata expiration check: 0:31:34 ago on Mon 06 Mar 2023 10:25:43 AM UTC.
Dependencies resolved.
=====================================================================================================================================
Package Architecture Version Repository Size
=====================================================================================================================================
Installing:
kernel x86_64 4.18.0-425.13.1.el8_7 baseos 8.8 M
kernel-core x86_64 4.18.0-425.13.1.el8_7 baseos 41 M
kernel-devel x86_64 4.18.0-425.13.1.el8_7 baseos 22 M
kernel-modules x86_64 4.18.0-425.13.1.el8_7 baseos 33 M


Check the default kernal again due to load to load the new one on reboot
COMMAND: sudo grubby --default-kernel
OUTPUT: /boot/vmlinuz-4.18.0-425.13.1.el8_7.x86_64 - Yes new Kernel due to load


REBOOT THE BOX


Check what kernel is running
COMMAND:
uname -r
---------------------------
OUTPUT: 4.18.0-425.10.1.el8_7.x86_64 - OLD Kernel still! :eek::mad: Expecting 4.18.0-425.13.1.el8_7.x86_64


Check the default kernal due to load

COMMAND: sudo grubby --default-kernel
OUTPUT: /boot/vmlinuz-4.18.0-425.13.1.el8_7.x86_64 - This is correct


Check the grub list/bootloader. Should be loading 4.18.0-425.13 kernel

COMMAND: sudo awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
OUTPUT:
0 : AlmaLinux (4.18.0-425.13.1.el8_7.x86_64) 8.7 (Stone Smilodon)
1 : AlmaLinux (4.18.0-425.10.1.el8_7.x86_64) 8.7 (Stone Smilodon)
2 : AlmaLinux (4.18.0-372.9.1.el8.x86_64) 8.7 (Stone Smilodon)
3 : AlmaLinux (0-rescue-5b3d368fc52a4cf7ad96fb517a77eb3b) 8.7 (Stone Smilodon)

Grub list has the new kernel at index 0 of the list.


Check the default kernel selected to run

COMMAND: sudo grubby --info=DEFAULT
OUTPUT:
index=0
kernel="/boot/vmlinuz-4.18.0-425.13.1.el8_7.x86_64"
args="ro crashkernel=auto resume=/dev/mapper/almalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap $tuned_params"
root="/dev/mapper/almalinux-root"
initrd="/boot/initramfs-4.18.0-425.13.1.el8_7.x86_64.img $tuned_initrd"
title="AlmaLinux (4.18.0-425.13.1.el8_7.x86_64) 8.7 (Stone Smilodon)"
id="5b3d368fc52a4cf7ad96fb517a77eb3b-4.18.0-425.13.1.el8_7.x86_64"


Check the selection index of the default kernel selected to run
COMMAND: sudo grubby --default-index
OUTPUT: 0


Check the index list of the grub loader
COMMAND: sudo grubby --info=ALL | grep -E '^(index|kernel)='
OUTPUT:
index=0
kernel="/boot/vmlinuz-4.18.0-425.13.1.el8_7.x86_64"
index=1
kernel="/boot/boot/vmlinuz-4.18.0-425.10.1.el8_7.x86_64"
index=2
kernel="/boot/boot/vmlinuz-4.18.0-372.9.1.el8.x86_64"
index=3
kernel="/boot/boot/vmlinuz-0-rescue-5b3d368fc52a4cf7ad96fb517a77eb3b"


I don't know why the bootloader on this server is not accepting to boot with the updated Kernel which has been installed, it should be as simple as running `sudo dnf update` this will update the grub list and defaults all this appears to have been done and rebooting to load the updates. But server on boot simply seems to ignore the grub list.

I also spun up a small test instance of AlmaLinux 8 via AWS on a VPS . The COMMAND: sudo dnf update is simply enough for it to update the packages and kernels and a reboot, boots the new kernel with no issues. Obviously, kernel updates should be this simple.

Having gone through all this with you/support and gathering all this data think it’s a weird datacentre/server configuration issue that the server/hosting company I have the server leased with. I've raised it with the datacentre company now, as everything on the OS is saying boot the latest installed kernel, but the server is not doing this.


My latest response from my server provider
We have been investigating the issue and in an attempt to replicate this, we have spun up a test Server with Alma Linux 8 and we found that this image/server actually deployed with the latest kernel version straight away. It may be a possibility that the issue lies with the specific image for Bare Metal servers, however these take a little longer to deploy. We are currently in the process of deploying a Bare Metal server with Alma Linux 8 so that we can attempt to replicate this again. If the issue persists we will then deploy this to our build team for investigation.

To help any other people experiencing similar issues like this there is an article on a similar issue which support pointed me towards ( I already rebooted but WHM still says I need to reboot ) This article is basically making sure that the grublist/bootloader for the OS has the correct Kernel selected to load. So at the moment, my money is some weird issue with how the AlmaLinux 8 OS is playing with the server hardware, as it appears for all intents and purposes the server boot is ignoring the grublist/bootloader on boot which is set correctly.

Fingers crossed my provider will be able have a solution here. Will let you know of any updates. As always thanks for your help @cPRex
 
  • Like
Reactions: cPRex