SOLVED Service failures after upgrading system to CentOS version 7.6

greektranslator

Well-Known Member
Jun 5, 2011
71
0
56
Greece
cPanel Access Level
Root Administrator
Please see this post for the official cPanel response with available solutions.

I was prompted for a server graceful reboot for software updates and server did not come back.
Edit: it came up after host intervention after 15 minutes but with some notes regarding correcting software config.
Cpanel Ticket id for this: 10885171
 
Last edited by a moderator:

ES - George

Well-Known Member
PartnerNOC
Jun 12, 2011
179
25
68
UK
cPanel Access Level
DataCenter Provider
Twitter
Glad to hear it came back online! :)

If you haven't done so already, I'd strongly recommend checking that you have working IPMI / KVM / Console access available to you. In a situation like this, it's the one (and only) hope available to revive a server that's inaccessible via normal means like SSH, WHM, etc. Personally speaking, I like to audit such out-of-band access systems every month or so, just to ensure they still work properly.
 

greektranslator

Well-Known Member
Jun 5, 2011
71
0
56
Greece
cPanel Access Level
Root Administrator
Thanks, looks like the host intervened automatically. Not sure though what really I need to change on the server.

Here are the details of this operation:
Software diagnosis
The server is on the Grub rescue.

Action:
Restarting the server in bzimage (Kernel OVH)

Result:
Boot: OK
Ping: OK
Services started; Server on login.

Recommendation:
Software configuration to be corrected by the customer.
 

WorkinOnIt

Well-Known Member
Aug 3, 2016
188
26
28
UK
cPanel Access Level
Root Administrator
I've opened a support ticket 10885695

I am having issue similar to these threads:

[Removed Links - these threads were merged into this one]

Can't restart the server after applying update. I am using the kernel care free simlink protection patch on this server and when I click on restart I see the message:

kcarectl --info reporting unknown patch type: free:15xxxxxx
at /usr/local/cpanel/Cpanel/KernelCare.pm line 80.
Cpanel::KernelCare::get_kernelcare_state() called at /usr/local/cpanel/Cpanel/KernelCare/Suggest.pm line 65
Cpanel::KernelCare::Suggest::get_suggestion() called at whostmgr/bin/whostmgr.pl line 5204
main::graceful_reboot_landing("graceful_reboot_landing") called at /usr/local/cpanel/Whostmgr/Dispatch.pm line 259
Whostmgr::Dispatch::_do_call("graceful_reboot_landing", HASH(0x2b9d000), HASH(0x2b9d988)) called at /usr/local/cpanel/Whostmgr/Dispatch.pm line 157
Whostmgr::Dispatch::dispatch("graceful_reboot_landing", 1, ARRAY(0x2b9cf88), HASH(0x2b9d988)) called at whostmgr/bin/whostmgr.pl line 393


I performed a graceful restart and the server failed to reload.
Then I restarted server at host - server won't restart.

In host console I get the message: !!!!!failed to load selinux policy

The solution is to modify the Root config in the GRUB boot (you need to be administrator)

I did this at my server host console (NOTE: use at your own risk. This worked for me (CentOS KVM) - it might be a different process for you depending on your machine / OS / Host. Please check with cPanel your host if you are not 100%


1. Access host console and click the send CTRL+ALT+DEL button on the top right, or click [RESTART] to restart the server. As soon as the boot process starts, press ESC to bring up the GRUB boot prompt. You may need to stop the server first, then restart it to reach the GRUB boot prompt.

2. You will see a GRUB boot prompt - press E to edit the first boot option. (If you do not see the GRUB prompt, you may need to press any key to bring it up before the machine boots)

3. Find the kernel line (it starts with "linux16"), REPLACE ro with rw init=/sysroot/bin/sh and leave the rest of the line unchanged.

4. Press CTRL+X or F10 to boot into single user mode.

5. Access the system with the command: chroot /sysroot

6. Edit the file with vi: eg;
#vi /etc/selinux/config

and change the line "enforcing" to "disabled"

7. reboot or restart the machine at host control panel.


I hope this helps someone !

There is also a tutorial here asafshoval.wordpress.com/2014/11/18/overcome-fail-to-load-selinux-policy-freezing-error-message-while-booting-linux (I didn't use it, but it is similar instruction)


 
Last edited by a moderator:

WorkinOnIt

Well-Known Member
Aug 3, 2016
188
26
28
UK
cPanel Access Level
Root Administrator
I also have several other servers - awaiting cPanel tech support to provide an update on how to mitigate this issue before updating other servers.

I guess if you are running SELinux (I didn't know I was!) you can disable it via this instruction before updating your server
 
Last edited:

JayFromEpic

Well-Known Member
Apr 2, 2011
218
8
68
Scottsdale
cPanel Access Level
Root Administrator
Twitter
I can confirm this is an active problem as well, atleast for servers running on Vultr OS Templates.

For what ever reason, when system updates are automatically updated and the machine is rebooted as needed, selinux somehow becomes enabled which is definitely not supposed to happen being a cpanel server.

Those are the same instructions vultr support confirmed with my when I brought this to their attention after having 15 servers crap out at once with them. I was able to just disable selinux and it resolved the issue.

Ill be shooting in a ticket as well with access to a now fixed machine for further investigation if needed.


EDIT: Ticket submitted with access to server that was affected: 10888269
 
Last edited:
  • Like
Reactions: WorkinOnIt

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,914
2,195
363
Hello Everyone,

We've received multiple reports of service failures and failed reboot attempts after a system is upgraded to CentOS version 7.6.

The upgrade to CentOS 7.6 can occur automatically as part of nightly cPanel & WHM updates on CentOS 7 systems with Operating System Package Updates set to Automatic in WHM Home » Server Configuration » Update Preferences, or when a system administrator manually runs the yum update command.

Upon investigating the issue, it appears the issue is isolated to systems missing the /etc/selinux/config file that's part of the selinux-policy RPM. This RPM and file exist by default on stock CentOS 7 installations, however it looks like some providers intentionally exclude the /etc/selinux/config file in the CentOS and cPanel & WHM images provided to their customers.

Furthermore, an updated Bind package is included as part of the CentOS 7.6 upgrade which includes a hard dependency for the selinux-policy RPM. This leads to the automatic installation of the selinux-policy RPM on systems where it was previously uninstalled. The selinux-policy RPM includes the following post-script:

Code:
# rpm -q --scripts selinux-policy-3.13.1-229.el7_6.6.noarch
postinstall scriptlet (using /bin/sh):
if [ ! -s /etc/selinux/config ]; then
#
#     New install so we will default to targeted policy
#
echo "
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
" > /etc/selinux/config
This post script automatically creates the /etc/selinux/config file with the SELINUX=enforcing value enabled if the file did not previously exist. Once enforcing mode is enabled, issues such as service failures and failed reboot attempts will occur.

Available solutions:

Modify the following value in the /etc/selinux/config file:

Code:
SELINUX=enforcing
To:

Code:
SELINUX=disabled
Once you've modified the /etc/selinux/config file, reboot the system to enable the change.

You will need to connect to your server's console using the method documented by your server provider. If console access is not available, contact your provider and ask them to review and perform the workaround steps provided by @WorkinOnIt below:

1. Access host console and click the send CTRL+ALT+DEL button on the top right, or click [RESTART] to restart the server. As soon as the boot process starts, press ESC to bring up the GRUB boot prompt. You may need to stop the server first, then restart it to reach the GRUB boot prompt.

2. You will see a GRUB boot prompt - press E to edit the first boot option. (If you do not see the GRUB prompt, you may need to press any key to bring it up before the machine boots)

3. Find the kernel line (it starts with "linux16"), REPLACE ro with rw init=/sysroot/bin/sh and leave the rest of the line unchanged.

4. Press CTRL+X or F10 to boot into single user mode.

5. Access the system with the command: chroot /sysroot

6. Edit the file with vi: eg;
#vi /etc/selinux/config

and change the line "enforcing" to "disabled"

7. reboot or restart the machine at host control panel.

The following steps explain how it's possible for reboot attempts to take longer than normal on affected systems:
  1. The /etc/selinux/config file does not exist on the system (SELinux defaults to disabled in this state).
  2. The server was rebooted at least once in the past (with /etc/selinux/config missing).
  3. As part of the initial reboot, the /.autorelabel touch file is created during the shutdown phase.
  4. With SELinux disabled, the reboot process finishes normally and the relabeling process is not initiated.
  5. Later, the /etc/selinux/config is created and SELinux is enabled (see the beginning of this post for how this can happen).
  6. Upon the next reboot, the /.autorelabel touch file initiates the relabeling process this time because it detects SELinux as enabled.
  7. The relabeling process can take anywhere from a few minutes to several hours depending on the server hardware and usage.
  8. Once the relabeling process is complete, the server will automatically reboot again.

If you have not yet upgraded your system to CentOS 7.6, and the /etc/selinux/config file does not exist on your system, ensure that it's created with the SELINUX=disabled value before performing the upgrade as seen in the output below:

Code:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
Let us know if you have any questions.

Thank you.
 
Last edited:

Webdew

Member
Jun 18, 2013
12
0
1
cPanel Access Level
Root Administrator
I'm on OVH and have just had this happen.. naturally its exactly 20min before phone support closed and I can not get through!

Also after booting to a rescue image and looking for the
SELINUX=enforcing flag only to discover it was already set to
SELINUX=disabled I'm totally lost as to what to do and now looking at 12 hours until phone support comes online - I have emailed them so hopefully they can work out what has happened and get it back online.

WHM/cPanel auto-update is enabled.

I've received message that after a new kernel update the server needs to reboot. I've logged into WHM and clicked the "You must reboot the server to apply software updates."

Then the server never comes back.
I'm on OVH as a physical machine.

They have IPMI java console. I access the machine through this and I get

grub>

?

This is not my area of expertise. But after I find the link above and boot into a rescue image to get disk access, I check the raid is OK and SSH to look for the
SELINUX=enforcing
line in /etc/selinux/config and it was already set to 'disabled' now I am at a loss as to how to get the server back up. I have ticketed OVH as it is just after hours for phone support... I'm hoping now they can resolve it an understand my notes (I'm also hoping they will not discard my request if they see me connected by the IPMI).
 
Last edited by a moderator:

Webdew

Member
Jun 18, 2013
12
0
1
cPanel Access Level
Root Administrator
So OVH 'intervened'

Here are the details of this operation:

Diagnosis interface boot (rescue)
Date 2018-12-05 18:52:16 AEDT (UTC +11:00),Diagnosis interface boot (rescue):


Here are the details of the operation performed:
The server was stuck on the grub.

We have seen this issue on other servers. A restart with the OVH kernel (bzimage) allows the server to start.

Actions:
restart with the OVH kernel (bzimage)

result:
Boot OK. Server pings and is on login

recommendations:
Further action is required by the customer to fix the root cause of the grub/kernel issues


.....

Awesome it is back up. I'm not sure what I need to do to fix the "grub issue" - certainly I'm extremely hesitant to restart this server right now.

Any help or direction appreciated now.
 

LucasRolff

Well-Known Member
May 27, 2013
133
76
28
cPanel Access Level
Root Administrator
I'm on OVH and have just had this happen.. naturally its exactly 20min before phone support closed and I can not get through!

Also after booting to a rescue image and looking for the
SELINUX=enforcing flag only to discover it was already set to
SELINUX=disabled I'm totally lost as to what to do and now looking at 12 hours until phone support comes online - I have emailed them so hopefully they can work out what has happened and get it back online.
When booted into the rescue image, did you go to /etc/selinux/config on SSH, or did you mount your partition on /mnt and changed it there? the /etc/selinux/config file in rescue, is the selinux config for the rescue image, and not your server.
 
  • Like
Reactions: cPanelMichael

WorkinOnIt

Well-Known Member
Aug 3, 2016
188
26
28
UK
cPanel Access Level
Root Administrator
Last edited:
  • Like
Reactions: cPanelMichael

WorkinOnIt

Well-Known Member
Aug 3, 2016
188
26
28
UK
cPanel Access Level
Root Administrator
Glad you got it solved. Check this out - it worked for me.

Installation Guide - System Requirements - Version 76 Documentation - cPanel Documentation

Disable SELinux
If your server runs an operating system from a source other than the cPanel & WHM installer, you must disable SELinux to make your system compatible with cPanel & WHM.

To disable SELinux security features, use one of the following methods:
  • Use the graphical interface to disable SELinux while you configure your operating system, and then reboot the server.
  • Edit the /etc/selinux/config file to set the SELINUX parameter to disabled, and then reboot the server.

    The contents of the /etc/selinux/config file should resemble the following example...
Code:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted
Warning:

  • To run cPanel & WHM on your server, SELinux must remain disabled.
    • SELinux in enforcing mode does not allow cPanel & WHM to function properly. For more information about SELinux modes, read the SELinux Mode documentation.
    • While cPanel & WHM can function with SELinux in permissive mode, we recommend that you do not use it. Permissive mode generates a large number of log entries.
    • To check the status of SELinux on your server, run the sestatus command.
  • Do not transfer the SELinux configuration file between computers. It may destroy the file's integrity.
Also read @cPanelMichael's thread here that might help to explain.
 
Last edited by a moderator:

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,914
2,195
363
Hello Everyone,

I updated my previous post with some additional information for hosting providers and system administrators explaining why rebooting a system affected by this issue can take longer than normal if SELinux is not disabled prior to the reboot. I've also quoted this information below:

The following steps explain how it's possible for reboot attempts to take longer than normal on affected systems:
  1. The /etc/selinux/config file does not exist on the system (SELinux defaults to disabled in this state).
  2. The server was rebooted at least once in the past (with /etc/selinux/config missing).
  3. As part of the initial reboot, the /.autorelabel touch file is created during the shutdown phase.
  4. With SELinux disabled, the reboot process finishes normally and the relabeling process is not initiated.
  5. Later, the /etc/selinux/config is created and SELinux is enabled (see the beginning of this post for how this can happen).
  6. Upon the next reboot, the /.autorelabel touch file initiates the relabeling process this time because it detects SELinux as enabled.
  7. The relabeling process can take anywhere from a few minutes to several hours depending on the server hardware and usage.
  8. Once the relabeling process is complete, the server will automatically reboot again.
Thank you.
 

sparek-3

Well-Known Member
Aug 10, 2002
1,966
206
343
cPanel Access Level
Root Administrator
Does this affect OpenVZ?

As far as I know, OpenVZ does not support SELinux, and so as such /etc/selinux/config does not exist.

And OpenVZ system do not install the selinux-policy rpm on the CentOS 7.6 update, so I believe they would be unaffected.
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,914
2,195
363
Does this affect OpenVZ?

As far as I know, OpenVZ does not support SELinux, and so as such /etc/selinux/config does not exist.

And OpenVZ system do not install the selinux-policy rpm on the CentOS 7.6 update, so I believe they would be unaffected.
Hello @sparek-3,

I don't have an OpenVZ system available for immediate testing, but as I understand the /etc/selinux/config file exists by default with any stock CentOS 7 installation (with the SELINUX=disabled value). Are you using a particular image or template to provision the servers?

Thank you.
 

sparek-3

Well-Known Member
Aug 10, 2002
1,966
206
343
cPanel Access Level
Root Administrator
I just happen to have 1 OpenVZ CentOS 7 server. It does not have an /etc/selinux/config file and yum check-update does not list an selinux-policy package to be installed (I haven't yet updated to CentOS 7.6).

I'm not sure how CentOS 7 was installed on this particular OpenVZ VPS. Perhaps others might chime in?

I suspect I can add an /etc/selinux/config with SELINUX=disabled just to be safe, but I don't think it would make any difference.

The issue appears to only pop up on systems where selinux-policy is updated.
 

greektranslator

Well-Known Member
Jun 5, 2011
71
0
56
Greece
cPanel Access Level
Root Administrator
These are the contents of my selinux config, what should I do?
Code:
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted
 

greektranslator

Well-Known Member
Jun 5, 2011
71
0
56
Greece
cPanel Access Level
Root Administrator
This was the host reply (whom I pointed out to this thread):

Your server got updated and with the new update is not able to boot correctly. Since the monitoring was enabled our system detected that and putted an intervention to check why the server was offline. The message you have received is the result of that intervention advising you to fix the software problems preventing your server from correctly booting into the OS.

I checked the link that you have send to me and it refers to a configuration file that does not exist. You can reboot the server in rescue mode and create the missing configuration in order for the server to boot.

From what I saw the server now is online and you are using one of the kernels of OVH. I suggest in this case you are able to log in to your server and start the configuration of the kernel in order to boot from your kernel and have the services running.