SOLVED Service failures after upgrading system to CentOS version 7.6

[cPanelMichael]

The issue appears to only pop up on systems where selinux-policy is updated.

An updated Bind package is included as part of the CentOS 7.6 upgrade and comes with a hard dependency for the selinux-policy RPM. Thus, upgrading to CentOS 7.6 will automatically install the selinux-policy RPM on systems where it was previously uninstalled, leading to the issue described in this thread.

These are the contents of my selinux config, what should I do?

The contents you pasted show that SELinux is not configured with enforcing mode enabled, so you should not experience any problems after rebooting the server. Are reboot attempts failing? Or, are you experiencing any specific issues?

Thank you.

 

[Adamfynd]

Hi
An hour ago I clicked on the updated WHM
Unfortunately, the entire server stopped working
The server is now in rescue mode
How do I fix the problem in rescue mode?
Please I want a quick fix

 

[JayFromEpic]

These are the contents of my selinux config, what should I do?

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

You would want your configuration file updated to the following:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

Once you have finished updating this file, reboot the server. As cPanelMichael mentioned, it will take some time for the command to process most likely because of the relabeling process. I was able to force reboot one of our servers via the control panel of our server provider but I don't normally recommend this due to the risk of data loss.

 

[cPanelMichael]

The server is now in rescue mode. How do I fix the problem in rescue mode?

You'll need console access to the server to solve the problem. If you have console access, you can perform the steps noted under "Solution 2" on this post. If you don't have console access, contact your provider to have them perform those steps on your behalf, or see if they can facilitate setting up console access for you. You may also want to reach out to a system administrator if you don't have experience performing this type of action on your own.

Once you have finished updating this file, reboot the server. As cPanelMichael mentioned, it will take some time for the command to process most likely because of the relabeling process.

As long as SELinux is disabled before the server the server is rebooted, the relabeling process should not occur at boot time.

Thank you.

 

[Webdew]

So OVH 'intervened'

recommendations:
Further action is required by the customer to fix the root cause of the grub/kernel issues

Received a follow up from OVH.

"
After some recent feedback from other client’s, we can confirm that there’s likely an issue with our installation template for Centos 7 with the Host lineup of servers. Since the issue at hand has to do with the software, we wouldn’t be able to provide a fix on our end but it’s possible that booting the server using a network kernel can remedy the issue. Below is a link with more information on booting from a network kernel and there’s another link below it with more information on how to update the kernel if you wanted to implemented your own fix.

Starting your server on an OVH kernel

Updating the kernel on a dedicated server

"


So I'm still not sure how to be sure I have fixed it. I'm not wanting to reboot as I'm still not sure how OVH managed to get it started again when it stopped before. Will future cPanel patches be able to revert / check and ensure this is now OK?

 

[cPanelMichael]

So I'm still not sure how to be sure I have fixed it. I'm not wanting to reboot as I'm still not sure how OVH managed to get it started again when it stopped before.

They have IPMI java console. I access the machine through this and I get

grub>

?

Hello @Webdew,

Once you're at the "grub" boot prompt, you could try following the steps provided by @WorkinOnIt earlier in this thread. See a quote below:

2. You will see a GRUB boot prompt - press E to edit the first boot option. (If you do not see the GRUB prompt, you may need to press any key to bring it up before the machine boots)

3. Find the kernel line (it starts with "linux16"), REPLACE ro with rw init=/sysroot/bin/sh and leave the rest of the line unchanged.

4. Press CTRL+X or F10 to boot into single user mode.

5. Access the system with the command: chroot /sysroot

6. Edit the file with vi: eg;
#vi /etc/selinux/config

and change the line "enforcing" to "disabled"

7. reboot or restart the machine at host control panel.

See also the advice from @LucasRolff as well:

When booted into the rescue image, did you go to /etc/selinux/config on SSH, or did you mount your partition on /mnt and changed it there? the /etc/selinux/config file in rescue, is the selinux config for the rescue image, and not your server.

Let me know if this helps.

Thank you.

 

[omaniyat]

Hello,

I'm on OVH i have same issue.

Can someone help me and make the server start again please?

Thank you,

 

[greektranslator]

Hi, I had to assign this to a freelance sysadmin. The KVM was not working, neither the IPMI! OVH had to intervene to fix the IPMI. It took the sysadmin several hours to resolve the problem, quoting:

The grub-efi package was never installed or removed. (Probably when setting up the server you chose for OVH kernel so they didn't install it).
That explains why it needed a command to find the config file on each boot.

 

[WorkinOnIt]

Hi, I had to assign this to a freelance sysadmin. The KVM was not working, neither the IPMI! OVH had to intervene to fix the IPMI. It took the sysadmin several hours to resolve

Ouch! Painful - it never rains but it pours!! When I first spin up a machine, I always run through the console checks to make sure I will always have access in an emergency.

Glad you got it sorted though.

 

[cPanelMichael]

Hi, I had to assign this to a freelance sysadmin. The KVM was not working, neither the IPMI! OVH had to intervene to fix the IPMI. It took the sysadmin several hours to resolve the problem, quoting:

The grub-efi package was never installed or removed. (Probably when setting up the server you chose for OVH kernel so they didn't install it).
That explains why it needed a command to find the config file on each boot.

Hello @greektranslator,

Thanks for sharing the outcome. Do you have any additional details to share, such as the specific commands that were ran, in-case other OVH users face the same problem?

Thank you.

 

[greektranslator]

No sorry, as I said, this was assigned to a sysadmin who struggled for a considerable amount of time, I can get people in touch with him if they so desire.

 

[cPanelMichael]

I can get people in touch with him if they so desire.

Hi @greektranslator,

It's not urgent, but I think the other OVH users facing the same issue would find it helpful should something like this happen again in the future.

Thanks!

 

[greektranslator]

Indeed, I would also have found it useful if I had such information some days ago, but the OVH were saying it is not their responsibility and Cpanel were saying the same and that I should contact a sysadmin to do the job. So, this is what I also recommend.

 

[marcuszan]

I am facing similar issues. I am on OVH. Right now I have the netboot kernel from OVH so my server is up again, but the problem is not solved.
I did an upgrade to CentOS 7.6 and then it got stuck on reboot in grub.

I did disable SElinux, so thats not an issue in my case.


When I check my grub entries, it is like this >

grep vmlinuz /boot/grub2/grub.cfg
        linuxefi /vmlinuz-4.4.166-1.el7.elrepo.x86_64 root=/dev/md3 ro crashkernel=auto rhgb quiet vga=normal nomodeset rd.auto=1 rd.md.uuid=55fba025:dcfa45b1:a4d2adc2:26fd5302 rd.md.uuid=4f2ff053:53ba30e8:a4d2adc2:26fd5302 rootdelay=10 rootdelay=10 noquiet nosplash net.ifnames=0 biosdevname=0
        linuxefi /vmlinuz-4.4.165-1.el7.elrepo.x86_64 root=/dev/md3 ro crashkernel=auto rhgb quiet vga=normal nomodeset rd.auto=1 rd.md.uuid=55fba025:dcfa45b1:a4d2adc2:26fd5302 rd.md.uuid=4f2ff053:53ba30e8:a4d2adc2:26fd5302 rootdelay=10 rootdelay=10 noquiet nosplash net.ifnames=0 biosdevname=0
        linuxefi /vmlinuz-0-rescue-c1a63cb41f3a079c801badf65b464d62 root=/dev/md3 ro crashkernel=auto rhgb quiet vga=normal nomodeset rd.auto=1 rd.md.uuid=55fba025:dcfa45b1:a4d2adc2:26fd5302 rd.md.uuid=4f2ff053:53ba30e8:a4d2adc2:26fd5302 rootdelay=10 rootdelay=10 noquiet nosplash net.ifnames=0 biosdevname=0

I am wondering if the linuxefi is causing the problem. I cant recall if this was like this before.
Reading the info from @greektranslator and his sysadmin, I checked if grub-efi was installed. This was the case.

Installed:
  grub2-efi-x64.x86_64 1:2.02-0.76.el7.centos                      grub2-efi-x64-modules.noarch 1:2.02-0.76.el7.centos

I have no clue what is wrong or missing in grub that prevents my box from booting.

Any help is appreciated.

Thanks

 

[Webdew]

@greektranslator

2. You will see a GRUB boot prompt - press E to edit the first boot option.

I still haven't rebooted my server since it went back up .. but I did see that post at the time and remember the above command on grub did nothing - neither 'e' or 'E' does this mean like a comment above "The grub-efi package was never installed or removed. " ?

I'm basically in the same situation as @marcuszan above except i don't know how to check for grub efi .. and I'm not wanting to reboot the server now to tool around to find out if I can or can not get it started again without being fairly sure of what I need to do.

 

[marcuszan]

@greektranslator

Can you post the output of this, so we can compare ?

grep vmlinuz /boot/grub2/grub.cfg

Maybe your sysadmin changed something in your grub config file and this might give us some clue.

thanks

 

[Yoann]

I make the update on my server
I reboot
and now I have no server, a problem on systemcrtl

I'm at OVH, I'm on rescue boot and it's a catastrophe for me.

 

[marcuszan]

I make the update on my server
I reboot
and now I have no server, a problem on systemcrtl

I'm at OVH, I'm on rescue boot and it's a catastrophe for me.

For a temp fix go to your OVH manager and change to netboot instead of hdd boot, select OVH kernel image and reboot. Server will then boot up using OVH net kernel.
It is a workaround but at least your server is up.

 

[marcuszan]

I think I found a solution. For me it works and I am now able to boot using my HDD kernel in CentOS 7.6
Working with the info from @greektranslator
Also looking in /var/log/grubby I found some info on missing uuid
So I figured I need to rebuild grub2. I did this before using this command :

grub2-mkconfig -o /boot/grub2/grub.cfg

But that didnt work for me
But I have the idea that since CentOS 7.6 some uefi things were added to grub2
So I found some info on rebuilding grub2 for uefi systems.

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

This seems to solve the issue for me . I did a reboot and now all is good. Will do some more testing and report if I occur any more problems.

 

[Webdew]

grep vmlinuz /boot/grub2/grub.cfg

gives me nothing.