SOLVED Service failures after upgrading system to CentOS version 7.6

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,913
2,200
363
The issue appears to only pop up on systems where selinux-policy is updated.
An updated Bind package is included as part of the CentOS 7.6 upgrade and comes with a hard dependency for the selinux-policy RPM. Thus, upgrading to CentOS 7.6 will automatically install the selinux-policy RPM on systems where it was previously uninstalled, leading to the issue described in this thread.

These are the contents of my selinux config, what should I do?
The contents you pasted show that SELinux is not configured with enforcing mode enabled, so you should not experience any problems after rebooting the server. Are reboot attempts failing? Or, are you experiencing any specific issues?

Thank you.
 

Adamfynd

Registered
Nov 2, 2018
3
0
1
Sverige
cPanel Access Level
Website Owner
Hi
An hour ago I clicked on the updated WHM
Unfortunately, the entire server stopped working
The server is now in rescue mode
How do I fix the problem in rescue mode?
Please I want a quick fix
 

JayFromEpic

Well-Known Member
Apr 2, 2011
218
8
68
Scottsdale
cPanel Access Level
Root Administrator
Twitter
These are the contents of my selinux config, what should I do?
Code:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

You would want your configuration file updated to the following:

Code:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
Once you have finished updating this file, reboot the server. As cPanelMichael mentioned, it will take some time for the command to process most likely because of the relabeling process. I was able to force reboot one of our servers via the control panel of our server provider but I don't normally recommend this due to the risk of data loss.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,913
2,200
363
The server is now in rescue mode. How do I fix the problem in rescue mode?
You'll need console access to the server to solve the problem. If you have console access, you can perform the steps noted under "Solution 2" on this post. If you don't have console access, contact your provider to have them perform those steps on your behalf, or see if they can facilitate setting up console access for you. You may also want to reach out to a system administrator if you don't have experience performing this type of action on your own.

Once you have finished updating this file, reboot the server. As cPanelMichael mentioned, it will take some time for the command to process most likely because of the relabeling process.
As long as SELinux is disabled before the server the server is rebooted, the relabeling process should not occur at boot time.

Thank you.
 
Last edited:

Webdew

Member
Jun 18, 2013
15
0
51
cPanel Access Level
Root Administrator
So OVH 'intervened'

recommendations:
Further action is required by the customer to fix the root cause of the grub/kernel issues
Received a follow up from OVH.

"
After some recent feedback from other client’s, we can confirm that there’s likely an issue with our installation template for Centos 7 with the Host lineup of servers. Since the issue at hand has to do with the software, we wouldn’t be able to provide a fix on our end but it’s possible that booting the server using a network kernel can remedy the issue. Below is a link with more information on booting from a network kernel and there’s another link below it with more information on how to update the kernel if you wanted to implemented your own fix.

Starting your server on an OVH kernel

Updating the kernel on a dedicated server

"


So I'm still not sure how to be sure I have fixed it. I'm not wanting to reboot as I'm still not sure how OVH managed to get it started again when it stopped before. Will future cPanel patches be able to revert / check and ensure this is now OK?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,913
2,200
363
So I'm still not sure how to be sure I have fixed it. I'm not wanting to reboot as I'm still not sure how OVH managed to get it started again when it stopped before.
They have IPMI java console. I access the machine through this and I get

grub>

?
Hello @Webdew,

Once you're at the "grub" boot prompt, you could try following the steps provided by @WorkinOnIt earlier in this thread. See a quote below:

2. You will see a GRUB boot prompt - press E to edit the first boot option. (If you do not see the GRUB prompt, you may need to press any key to bring it up before the machine boots)

3. Find the kernel line (it starts with "linux16"), REPLACE ro with rw init=/sysroot/bin/sh and leave the rest of the line unchanged.

4. Press CTRL+X or F10 to boot into single user mode.

5. Access the system with the command: chroot /sysroot

6. Edit the file with vi: eg;
#vi /etc/selinux/config

and change the line "enforcing" to "disabled"

7. reboot or restart the machine at host control panel.
See also the advice from @LucasRolff as well:

When booted into the rescue image, did you go to /etc/selinux/config on SSH, or did you mount your partition on /mnt and changed it there? the /etc/selinux/config file in rescue, is the selinux config for the rescue image, and not your server.
Let me know if this helps.

Thank you.
 

omaniyat

Registered
Nov 1, 2006
1
0
151
Hello,

I'm on OVH i have same issue.

Can someone help me and make the server start again please?

Thank you,
 

greektranslator

Well-Known Member
Jun 5, 2011
71
0
56
Greece
cPanel Access Level
Root Administrator
Hi, I had to assign this to a freelance sysadmin. The KVM was not working, neither the IPMI! OVH had to intervene to fix the IPMI. It took the sysadmin several hours to resolve the problem, quoting:

The grub-efi package was never installed or removed. (Probably when setting up the server you chose for OVH kernel so they didn't install it).
That explains why it needed a command to find the config file on each boot.
 

WorkinOnIt

Well-Known Member
Aug 3, 2016
193
27
28
UK
cPanel Access Level
Root Administrator
Hi, I had to assign this to a freelance sysadmin. The KVM was not working, neither the IPMI! OVH had to intervene to fix the IPMI. It took the sysadmin several hours to resolve
Ouch! Painful - it never rains but it pours!! When I first spin up a machine, I always run through the console checks to make sure I will always have access in an emergency.

Glad you got it sorted though.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,913
2,200
363
Hi, I had to assign this to a freelance sysadmin. The KVM was not working, neither the IPMI! OVH had to intervene to fix the IPMI. It took the sysadmin several hours to resolve the problem, quoting:

The grub-efi package was never installed or removed. (Probably when setting up the server you chose for OVH kernel so they didn't install it).
That explains why it needed a command to find the config file on each boot.
Hello @greektranslator,

Thanks for sharing the outcome. Do you have any additional details to share, such as the specific commands that were ran, in-case other OVH users face the same problem?

Thank you.
 

greektranslator

Well-Known Member
Jun 5, 2011
71
0
56
Greece
cPanel Access Level
Root Administrator
Indeed, I would also have found it useful if I had such information some days ago, but the OVH were saying it is not their responsibility and Cpanel were saying the same and that I should contact a sysadmin to do the job. So, this is what I also recommend.
 

marcuszan

Well-Known Member
Apr 19, 2018
63
5
8
Netherlands
cPanel Access Level
Root Administrator
I am facing similar issues. I am on OVH. Right now I have the netboot kernel from OVH so my server is up again, but the problem is not solved.
I did an upgrade to CentOS 7.6 and then it got stuck on reboot in grub.

I did disable SElinux, so thats not an issue in my case.


When I check my grub entries, it is like this >
Code:
grep vmlinuz /boot/grub2/grub.cfg
        linuxefi /vmlinuz-4.4.166-1.el7.elrepo.x86_64 root=/dev/md3 ro crashkernel=auto rhgb quiet vga=normal nomodeset rd.auto=1 rd.md.uuid=55fba025:dcfa45b1:a4d2adc2:26fd5302 rd.md.uuid=4f2ff053:53ba30e8:a4d2adc2:26fd5302 rootdelay=10 rootdelay=10 noquiet nosplash net.ifnames=0 biosdevname=0
        linuxefi /vmlinuz-4.4.165-1.el7.elrepo.x86_64 root=/dev/md3 ro crashkernel=auto rhgb quiet vga=normal nomodeset rd.auto=1 rd.md.uuid=55fba025:dcfa45b1:a4d2adc2:26fd5302 rd.md.uuid=4f2ff053:53ba30e8:a4d2adc2:26fd5302 rootdelay=10 rootdelay=10 noquiet nosplash net.ifnames=0 biosdevname=0
        linuxefi /vmlinuz-0-rescue-c1a63cb41f3a079c801badf65b464d62 root=/dev/md3 ro crashkernel=auto rhgb quiet vga=normal nomodeset rd.auto=1 rd.md.uuid=55fba025:dcfa45b1:a4d2adc2:26fd5302 rd.md.uuid=4f2ff053:53ba30e8:a4d2adc2:26fd5302 rootdelay=10 rootdelay=10 noquiet nosplash net.ifnames=0 biosdevname=0
I am wondering if the linuxefi is causing the problem. I cant recall if this was like this before.
Reading the info from @greektranslator and his sysadmin, I checked if grub-efi was installed. This was the case.

Code:
Installed:
  grub2-efi-x64.x86_64 1:2.02-0.76.el7.centos                      grub2-efi-x64-modules.noarch 1:2.02-0.76.el7.centos
I have no clue what is wrong or missing in grub that prevents my box from booting.

Any help is appreciated.

Thanks
 

Webdew

Member
Jun 18, 2013
15
0
51
cPanel Access Level
Root Administrator
@greektranslator

2. You will see a GRUB boot prompt - press E to edit the first boot option.

I still haven't rebooted my server since it went back up .. but I did see that post at the time and remember the above command on grub did nothing - neither 'e' or 'E' does this mean like a comment above "The grub-efi package was never installed or removed. " ?

I'm basically in the same situation as @marcuszan above except i don't know how to check for grub efi .. and I'm not wanting to reboot the server now to tool around to find out if I can or can not get it started again without being fairly sure of what I need to do.
 

Yoann

Registered
Aug 21, 2011
1
0
51
I make the update on my server
I reboot
and now I have no server, a problem on systemcrtl

I'm at OVH, I'm on rescue boot and it's a catastrophe for me.
 

marcuszan

Well-Known Member
Apr 19, 2018
63
5
8
Netherlands
cPanel Access Level
Root Administrator
I make the update on my server
I reboot
and now I have no server, a problem on systemcrtl

I'm at OVH, I'm on rescue boot and it's a catastrophe for me.
For a temp fix go to your OVH manager and change to netboot instead of hdd boot, select OVH kernel image and reboot. Server will then boot up using OVH net kernel.
It is a workaround but at least your server is up.
 

marcuszan

Well-Known Member
Apr 19, 2018
63
5
8
Netherlands
cPanel Access Level
Root Administrator
I think I found a solution. For me it works and I am now able to boot using my HDD kernel in CentOS 7.6
Working with the info from @greektranslator
Also looking in /var/log/grubby I found some info on missing uuid
So I figured I need to rebuild grub2. I did this before using this command :
Code:
grub2-mkconfig -o /boot/grub2/grub.cfg
But that didnt work for me
But I have the idea that since CentOS 7.6 some uefi things were added to grub2
So I found some info on rebuilding grub2 for uefi systems.
Code:
grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
This seems to solve the issue for me . I did a reboot and now all is good. Will do some more testing and report if I occur any more problems.
 
  • Like
Reactions: cPanelMichael