SOLVED 'You must reboot the server to apply kernel updates' persists after multiple reboots

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston

aztopdavid

Well-Known Member
Jan 1, 2016
53
9
58
Arizona
cPanel Access Level
Root Administrator
Yes, thanks, you'll see that was provided earlier in the thread and I went through that very thoroughly, but it doesn't appear to be an option given some specifics of my server config. Please take a look at the details I provided earlier, as I got stuck.
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston
Ok...usually that's a symlink to /boot/grub/grub.conf which is what I was really looking for - does that exist?

If not what is the output of the following:

Code:
ls -lah /boot/ |grep grub
Code:
rpm -qa |grep grub
 

aztopdavid

Well-Known Member
Jan 1, 2016
53
9
58
Arizona
cPanel Access Level
Root Administrator
Yes - I provided some fairly detailed excerpts of the conf file earlier in the thread -- the relevant "menuentry" values.

drwxr-xr-x 2 root root 4.0K Oct 29 2018 grub
drwx------ 2 root root 4.0K May 1 08:10 grub2

grub2-pc-modules-2.02-0.81.el7.centos.noarch
grub2-pc-2.02-0.81.el7.centos.x86_64
grub2-tools-minimal-2.02-0.81.el7.centos.x86_64
grub2-common-2.02-0.81.el7.centos.noarch
grub2-tools-extra-2.02-0.81.el7.centos.x86_64
grub2-2.02-0.81.el7.centos.x86_64
grubby-8.28-26.el7.x86_64
grub2-tools-2.02-0.81.el7.centos.x86_64
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston
I don't want the menuentry, I want the whole file which is why I asked for it. Can you please provide what I am requesting, I did read the thread - I'm attempting to help here but I can't do it if you're not providing me with exactly what I am requesting. I have no access to your system and no access to one *LIKE* yours so it places anyone attempting to provide assistance in a precarious position.
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston
One huge difference is the fact you have grub2-pc added:

My test environment runs the following so it makes sense why I don't' have the same items as you and I'm running on KVM as well:
Code:
[[email protected] default]# rpm -qa |grep grub
grub-0.97-94.el6.x86_64
grubby-8.28-8.el7.x86_64
So that helps guide the direction. Do you then have /etc/grub.d present?
 

aztopdavid

Well-Known Member
Jan 1, 2016
53
9
58
Arizona
cPanel Access Level
Root Administrator
Yes, /etc/grub.d is present, and here are the contents of /boot/grub2/grub.cfg

Code:
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="1"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_msdos
insmod ext2
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
else
  search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
fi
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_output gfxterm
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-simple-a44fb57f-8c52-7db3-f787-3f47c406178e' {
    load_video
    set gfxpayload=keep
    insmod gzio
    insmod part_msdos
    insmod ext2
    set root='hd0,msdos1'
    if [ x$feature_platform_search_hint = xy ]; then
      search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
    else
      search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
    fi
    linux16 /boot/vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/sda1 ro LANG=en_US.UTF-8
    initrd16 /boot/initramfs-3.10.0-1127.el7.x86_64.img
}
menuentry 'CentOS Linux 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-simple-a44fb57f-8c52-7db3-f787-3f47c406178e' {
    load_video
    set gfxpayload=keep
    insmod gzio
    insmod part_msdos
    insmod ext2
    set root='hd0,msdos1'
    if [ x$feature_platform_search_hint = xy ]; then
      search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
    else
      search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
    fi
    linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
    initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img
}
submenu 'Advanced options for CentOS Linux 7 (Core)' $menuentry_id_option 'gnulinux-advanced-a44fb57f-8c52-7db3-f787-3f47c406178e' {
    menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.18.1.el7.x86_64-advanced-a44fb57f-8c52-7db3-f787-3f47c406178e' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
        else
          search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
        fi
        linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
        initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img
    }
    menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core) (recovery mode)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.18.1.el7.x86_64-recovery-a44fb57f-8c52-7db3-f787-3f47c406178e' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
        else
          search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
        fi
        linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single
        initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img
    }
    menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.12.1.el7.x86_64-advanced-a44fb57f-8c52-7db3-f787-3f47c406178e' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
        else
          search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
        fi
        linux16 /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
        initrd16 /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img
    }
    menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core) (recovery mode)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.12.1.el7.x86_64-recovery-a44fb57f-8c52-7db3-f787-3f47c406178e' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
        else
          search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
        fi
        linux16 /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single
        initrd16 /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img
    }
    menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-862.14.4.el7.x86_64-advanced-a44fb57f-8c52-7db3-f787-3f47c406178e' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
        else
          search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
        fi
        linux16 /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro
        initrd16 /boot/initrd-3.10.0-862.14.4.el7.x86_64.img
    }
    menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core) (recovery mode)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-862.14.4.el7.x86_64-recovery-a44fb57f-8c52-7db3-f787-3f47c406178e' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='hd0,msdos1'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  a44fb57f-8c52-7db3-f787-3f47c406178e
        else
          search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e
        fi
        linux16 /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single
        initrd16 /boot/initrd-3.10.0-862.14.4.el7.x86_64.img
    }
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
 
Last edited by a moderator:

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston
Awesome, this tells us everything!

- where to edit it:

Code:
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
- It also tells us your default right now is 1:
Code:
   set default="1"
This explains why you're not booting into the new kernel - it should be 0

Give me a sec to find out where you should be editing /etc/grub.d/ - if you could provide me an ls -lah of that directory in the meantime it would be fantastic. I believe I can also tell you how to disable the submenus/advanced config :)
 

aztopdavid

Well-Known Member
Jan 1, 2016
53
9
58
Arizona
cPanel Access Level
Root Administrator
Thanks! Here's the ls -lah of /etc/grub.d/

drwx------ 2 root root 4.0K Apr 28 01:32 .
drwxr-xr-x 102 root root 24K May 1 13:30 ..
-rwxr-xr-x 1 root root 8.5K Apr 3 10:24 00_header
-rwxr-xr-x 1 root root 232 Apr 3 10:24 01_users
-rwxr-xr-x 1 root root 11K Apr 3 10:24 10_linux
-rwxr-xr-x 1 root root 11K Apr 3 10:24 20_linux_xen
-rwxr-xr-x 1 root root 2.5K Apr 3 10:24 20_ppc_terminfo
-rwxr-xr-x 1 root root 11K Apr 3 10:24 30_os-prober
-rwxr-xr-x 1 root root 214 Apr 3 10:24 40_custom
-rwxr-xr-x 1 root root 216 Apr 3 10:24 41_custom
-rw-r--r-- 1 root root 483 Apr 3 10:24 README
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston
So without /etc/default/grub (which the system I just spun up doesn't have either) you should be able to do the following:

(some of this overlaps with what you've done already but the steps are important to have all in one place I believe)

View the menuentries:
Code:
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
VIew the default/what the system is set to boot into:
Code:
grub2-editenv list
Compare the above and determine which menuentry starting at 0 -> X you want to boot into for example I have the following (I just installed the new kernel and didn't reboot in order to have this example)

Code:
-bash-4.2# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
0 : CentOS Linux 7 Rescue a1772cbb489b4174a47ccf9deca6e297 (3.10.0-1127.el7.x86_64)
1 : CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)
2 : CentOS Linux (3.10.0-1062.9.1.el7.x86_64) 7 (Core)
3 : CentOS Linux (0-rescue-87206729330b4fdda4e2599815d07128) 7 (Core)
Based on this menuentry 0 is a rescue so for my system I'd want to pick 1

Code:
-bash-4.2# grub2-editenv list
saved_entry=CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)

Now to edit the default order (example is using my menuentry) :

Code:
grub2-set-default 1
view the change:
Code:
-bash-4.2# grub2-editenv list
saved_entry=1
This will be different on your system since I believe the saved menuentry you want is 0


For the submenus to be disabled I'm struggling with a way to do this without /etc/default/grub existing, grub2-mkconfig does this when you have multiple kernels installed. Without the setting to disable in /etc/default/grub I'm running into dead ends for how to manipulate it. I believe you need to edit /etc/grub.d/40_custom but because I'm not 100% on that I don't want to recommend it.
 

aztopdavid

Well-Known Member
Jan 1, 2016
53
9
58
Arizona
cPanel Access Level
Root Administrator
Mine already appears to be set to boot to 0

Code:
# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
0 : CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)
1 : CentOS Linux 7 (Core)

# grub2-editenv list
saved_entry=0
Unfortunately, that doesn't appear to be happening.
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston
My assumption is in that case, something is happening when you're rebooting. Funny question but what's in /etc/fstab?

the output of the following would be useful:

Code:
cat /etc/fstab
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston
Ok, well that rules out that weird anomaly - I saw one case where this was happening due to /boot being commented out in /etc/fstab but since you have everything mounted on / that wouldn't be relevant.

What I think is happening is grub2's failsafe booting where if it can't update to that kernel it tries one other.

You're running a few versions behind even with 3.10.0-862.14.4.el7.x86_64 so this has been going on for a bit and every kernel installed is newer than the one you're running.

You could reinstall the newest kernel:

Code:
yum -y reinstall kernel-3.10.0-1127.el7.x86_64
But I'm not sure at this point that would help.

You could try pointing to the next newer kernel: kernel-3.10.0-957.27.2.el7.x86_64 which is I believe within a submenu

These articles are some of the things I've been looking through:


It's really interesting that the boot menu only shows the currently running kernel. Does grubby show the default kernel? What does it report for these?

Code:
grubby --default-kernel
Code:
grubby --default-index
 

aztopdavid

Well-Known Member
Jan 1, 2016
53
9
58
Arizona
cPanel Access Level
Root Administrator
# grubby --default-kernel
/boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64
# grubby --default-index
1

(and 957.27.2 got removed earlier in this thread by a command suggested by GOT)
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,268
313
Houston
Wait a minute here though. It seems grubby is seeing entirely different settings.

I forgot about the cleaning old packages my fault, there's a lot of info here.

The menuentry/index it's loading is indeed 1 from your /boot/grub2/grub.cfg my test system updates this automatically and it seems that somewhere months ago yours stop synchronizing with the grub2 config?

this may be interesting to see the output of:
Code:
grubby --info=ALL
You can set the default index with:
Code:
grubby --set-default-index=0
You can specify a kernel version with:

Code:
grubby --set-default /boot/vmlinuz-3.10.0-1127.el7.x86_64
I'd be really curious to see if setting the kernel or setting the index with grubby resolves this.