Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

SOLVED Service failures after upgrading system to CentOS version 7.6

Discussion in 'General Discussion' started by greektranslator, Dec 4, 2018.

Tags:
  1. greektranslator

    greektranslator Well-Known Member

    Joined:
    Jun 5, 2011
    Messages:
    58
    Likes Received:
    0
    Trophy Points:
    56
    Location:
    Greece
    cPanel Access Level:
    Root Administrator
    Please see this post for the official cPanel response with available solutions.

    I was prompted for a server graceful reboot for software updates and server did not come back.
    Edit: it came up after host intervention after 15 minutes but with some notes regarding correcting software config.
    Cpanel Ticket id for this: 10885171
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #1 greektranslator, Dec 4, 2018
    Last edited by a moderator: Dec 5, 2018
  2. ES - George

    ES - George Well-Known Member PartnerNOC

    Joined:
    Jun 12, 2011
    Messages:
    177
    Likes Received:
    23
    Trophy Points:
    68
    Location:
    UK
    cPanel Access Level:
    DataCenter Provider
    Twitter:
    Glad to hear it came back online! :)

    If you haven't done so already, I'd strongly recommend checking that you have working IPMI / KVM / Console access available to you. In a situation like this, it's the one (and only) hope available to revive a server that's inaccessible via normal means like SSH, WHM, etc. Personally speaking, I like to audit such out-of-band access systems every month or so, just to ensure they still work properly.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. greektranslator

    greektranslator Well-Known Member

    Joined:
    Jun 5, 2011
    Messages:
    58
    Likes Received:
    0
    Trophy Points:
    56
    Location:
    Greece
    cPanel Access Level:
    Root Administrator
    Thanks, looks like the host intervened automatically. Not sure though what really I need to change on the server.

    Here are the details of this operation:
    Software diagnosis
    The server is on the Grub rescue.

    Action:
    Restarting the server in bzimage (Kernel OVH)

    Result:
    Boot: OK
    Ping: OK
    Services started; Server on login.

    Recommendation:
    Software configuration to be corrected by the customer.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. WorkinOnIt

    WorkinOnIt Well-Known Member

    Joined:
    Aug 3, 2016
    Messages:
    168
    Likes Received:
    24
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    I've opened a support ticket 10885695

    I am having issue similar to these threads:

    [Removed Links - these threads were merged into this one]

    Can't restart the server after applying update. I am using the kernel care free simlink protection patch on this server and when I click on restart I see the message:

    kcarectl --info reporting unknown patch type: free:15xxxxxx
    at /usr/local/cpanel/Cpanel/KernelCare.pm line 80.
    Cpanel::KernelCare::get_kernelcare_state() called at /usr/local/cpanel/Cpanel/KernelCare/Suggest.pm line 65
    Cpanel::KernelCare::Suggest::get_suggestion() called at whostmgr/bin/whostmgr.pl line 5204
    main::graceful_reboot_landing("graceful_reboot_landing") called at /usr/local/cpanel/Whostmgr/Dispatch.pm line 259
    Whostmgr::Dispatch::_do_call("graceful_reboot_landing", HASH(0x2b9d000), HASH(0x2b9d988)) called at /usr/local/cpanel/Whostmgr/Dispatch.pm line 157
    Whostmgr::Dispatch::dispatch("graceful_reboot_landing", 1, ARRAY(0x2b9cf88), HASH(0x2b9d988)) called at whostmgr/bin/whostmgr.pl line 393


    I performed a graceful restart and the server failed to reload.
    Then I restarted server at host - server won't restart.

    In host console I get the message: !!!!!failed to load selinux policy

    The solution is to modify the Root config in the GRUB boot (you need to be administrator)

    I did this at my server host console (NOTE: use at your own risk. This worked for me (CentOS KVM) - it might be a different process for you depending on your machine / OS / Host. Please check with cPanel your host if you are not 100%


    1. Access host console and click the send CTRL+ALT+DEL button on the top right, or click [RESTART] to restart the server. As soon as the boot process starts, press ESC to bring up the GRUB boot prompt. You may need to stop the server first, then restart it to reach the GRUB boot prompt.

    2. You will see a GRUB boot prompt - press E to edit the first boot option. (If you do not see the GRUB prompt, you may need to press any key to bring it up before the machine boots)

    3. Find the kernel line (it starts with "linux16"), REPLACE ro with rw init=/sysroot/bin/sh and leave the rest of the line unchanged.

    4. Press CTRL+X or F10 to boot into single user mode.

    5. Access the system with the command: chroot /sysroot

    6. Edit the file with vi: eg;
    #vi /etc/selinux/config

    and change the line "enforcing" to "disabled"

    7. reboot or restart the machine at host control panel.


    I hope this helps someone !

    There is also a tutorial here asafshoval.wordpress.com/2014/11/18/overcome-fail-to-load-selinux-policy-freezing-error-message-while-booting-linux (I didn't use it, but it is similar instruction)


     
    #4 WorkinOnIt, Dec 4, 2018
    Last edited by a moderator: Dec 4, 2018
  5. WorkinOnIt

    WorkinOnIt Well-Known Member

    Joined:
    Aug 3, 2016
    Messages:
    168
    Likes Received:
    24
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Are you on an OVH managed server? I had a similar issue - I believe due to selinux fault
     
  6. WorkinOnIt

    WorkinOnIt Well-Known Member

    Joined:
    Aug 3, 2016
    Messages:
    168
    Likes Received:
    24
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    I also have several other servers - awaiting cPanel tech support to provide an update on how to mitigate this issue before updating other servers.

    I guess if you are running SELinux (I didn't know I was!) you can disable it via this instruction before updating your server
     
    #6 WorkinOnIt, Dec 4, 2018
    Last edited: Dec 4, 2018
  7. greektranslator

    greektranslator Well-Known Member

    Joined:
    Jun 5, 2011
    Messages:
    58
    Likes Received:
    0
    Trophy Points:
    56
    Location:
    Greece
    cPanel Access Level:
    Root Administrator
    Indeed. Thanks for the instructions on that thread. I think I will not do anything for now, I will just wait for Cpanel techs to check out my ticket and take the appropriate action.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #7 greektranslator, Dec 4, 2018
    Last edited: Dec 4, 2018
  8. JayFromEpic

    JayFromEpic Well-Known Member

    Joined:
    Apr 2, 2011
    Messages:
    218
    Likes Received:
    7
    Trophy Points:
    68
    Location:
    Scottsdale
    cPanel Access Level:
    Root Administrator
    Twitter:
    I can confirm this is an active problem as well, atleast for servers running on Vultr OS Templates.

    For what ever reason, when system updates are automatically updated and the machine is rebooted as needed, selinux somehow becomes enabled which is definitely not supposed to happen being a cpanel server.

    Those are the same instructions vultr support confirmed with my when I brought this to their attention after having 15 servers crap out at once with them. I was able to just disable selinux and it resolved the issue.

    Ill be shooting in a ticket as well with access to a now fixed machine for further investigation if needed.


    EDIT: Ticket submitted with access to server that was affected: 10888269
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #8 JayFromEpic, Dec 4, 2018
    Last edited: Dec 4, 2018
    WorkinOnIt likes this.
  9. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello Everyone,

    We've received multiple reports of service failures and failed reboot attempts after a system is upgraded to CentOS version 7.6.

    The upgrade to CentOS 7.6 can occur automatically as part of nightly cPanel & WHM updates on CentOS 7 systems with Operating System Package Updates set to Automatic in WHM Home » Server Configuration » Update Preferences, or when a system administrator manually runs the yum update command.

    Upon investigating the issue, it appears the issue is isolated to systems missing the /etc/selinux/config file that's part of the selinux-policy RPM. This RPM and file exist by default on stock CentOS 7 installations, however it looks like some providers intentionally exclude the /etc/selinux/config file in the CentOS and cPanel & WHM images provided to their customers.

    Furthermore, an updated Bind package is included as part of the CentOS 7.6 upgrade which includes a hard dependency for the selinux-policy RPM. This leads to the automatic installation of the selinux-policy RPM on systems where it was previously uninstalled. The selinux-policy RPM includes the following post-script:

    Code:
    # rpm -q --scripts selinux-policy-3.13.1-229.el7_6.6.noarch
    postinstall scriptlet (using /bin/sh):
    if [ ! -s /etc/selinux/config ]; then
    #
    #     New install so we will default to targeted policy
    #
    echo "
    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    #     enforcing - SELinux security policy is enforced.
    #     permissive - SELinux prints warnings instead of enforcing.
    #     disabled - No SELinux policy is loaded.
    SELINUX=enforcing
    # SELINUXTYPE= can take one of three values:
    #     targeted - Targeted processes are protected,
    #     minimum - Modification of targeted policy. Only selected processes are protected.
    #     mls - Multi Level Security protection.
    SELINUXTYPE=targeted
    " > /etc/selinux/config
    
    This post script automatically creates the /etc/selinux/config file with the SELINUX=enforcing value enabled if the file did not previously exist. Once enforcing mode is enabled, issues such as service failures and failed reboot attempts will occur.

    Available solutions:

    Modify the following value in the /etc/selinux/config file:

    Code:
    SELINUX=enforcing
    To:

    Code:
    SELINUX=disabled
    Once you've modified the /etc/selinux/config file, reboot the system to enable the change.

    You will need to connect to your server's console using the method documented by your server provider. If console access is not available, contact your provider and ask them to review and perform the workaround steps provided by @WorkinOnIt below:

    1. Access host console and click the send CTRL+ALT+DEL button on the top right, or click [RESTART] to restart the server. As soon as the boot process starts, press ESC to bring up the GRUB boot prompt. You may need to stop the server first, then restart it to reach the GRUB boot prompt.

    2. You will see a GRUB boot prompt - press E to edit the first boot option. (If you do not see the GRUB prompt, you may need to press any key to bring it up before the machine boots)

    3. Find the kernel line (it starts with "linux16"), REPLACE ro with rw init=/sysroot/bin/sh and leave the rest of the line unchanged.

    4. Press CTRL+X or F10 to boot into single user mode.

    5. Access the system with the command: chroot /sysroot

    6. Edit the file with vi: eg;
    #vi /etc/selinux/config

    and change the line "enforcing" to "disabled"

    7. reboot or restart the machine at host control panel.

    The following steps explain how it's possible for reboot attempts to take longer than normal on affected systems:
    1. The /etc/selinux/config file does not exist on the system (SELinux defaults to disabled in this state).
    2. The server was rebooted at least once in the past (with /etc/selinux/config missing).
    3. As part of the initial reboot, the /.autorelabel touch file is created during the shutdown phase.
    4. With SELinux disabled, the reboot process finishes normally and the relabeling process is not initiated.
    5. Later, the /etc/selinux/config is created and SELinux is enabled (see the beginning of this post for how this can happen).
    6. Upon the next reboot, the /.autorelabel touch file initiates the relabeling process this time because it detects SELinux as enabled.
    7. The relabeling process can take anywhere from a few minutes to several hours depending on the server hardware and usage.
    8. Once the relabeling process is complete, the server will automatically reboot again.

    If you have not yet upgraded your system to CentOS 7.6, and the /etc/selinux/config file does not exist on your system, ensure that it's created with the SELINUX=disabled value before performing the upgrade as seen in the output below:

    Code:
    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    #     enforcing - SELinux security policy is enforced.
    #     permissive - SELinux prints warnings instead of enforcing.
    #     disabled - No SELinux policy is loaded.
    SELINUX=disabled
    # SELINUXTYPE= can take one of three values:
    #     targeted - Targeted processes are protected,
    #     minimum - Modification of targeted policy. Only selected processes are protected.
    #     mls - Multi Level Security protection.
    SELINUXTYPE=targeted
    Let us know if you have any questions.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #9 cPanelMichael, Dec 4, 2018
    Last edited: Dec 5, 2018
  10. Webdew

    Webdew Member

    Joined:
    Jun 18, 2013
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I'm on OVH and have just had this happen.. naturally its exactly 20min before phone support closed and I can not get through!

    Also after booting to a rescue image and looking for the
    SELINUX=enforcing flag only to discover it was already set to
    SELINUX=disabled I'm totally lost as to what to do and now looking at 12 hours until phone support comes online - I have emailed them so hopefully they can work out what has happened and get it back online.

    WHM/cPanel auto-update is enabled.

    I've received message that after a new kernel update the server needs to reboot. I've logged into WHM and clicked the "You must reboot the server to apply software updates."

    Then the server never comes back.
    I'm on OVH as a physical machine.

    They have IPMI java console. I access the machine through this and I get

    grub>

    ?

    This is not my area of expertise. But after I find the link above and boot into a rescue image to get disk access, I check the raid is OK and SSH to look for the
    SELINUX=enforcing
    line in /etc/selinux/config and it was already set to 'disabled' now I am at a loss as to how to get the server back up. I have ticketed OVH as it is just after hours for phone support... I'm hoping now they can resolve it an understand my notes (I'm also hoping they will not discard my request if they see me connected by the IPMI).
     
    #10 Webdew, Dec 5, 2018
    Last edited by a moderator: Dec 5, 2018
  11. Webdew

    Webdew Member

    Joined:
    Jun 18, 2013
    Messages:
    11
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    So OVH 'intervened'

    Here are the details of this operation:

    Diagnosis interface boot (rescue)
    Date 2018-12-05 18:52:16 AEDT (UTC +11:00),Diagnosis interface boot (rescue):


    Here are the details of the operation performed:
    The server was stuck on the grub.

    We have seen this issue on other servers. A restart with the OVH kernel (bzimage) allows the server to start.

    Actions:
    restart with the OVH kernel (bzimage)

    result:
    Boot OK. Server pings and is on login

    recommendations:
    Further action is required by the customer to fix the root cause of the grub/kernel issues


    .....

    Awesome it is back up. I'm not sure what I need to do to fix the "grub issue" - certainly I'm extremely hesitant to restart this server right now.

    Any help or direction appreciated now.
     
  12. LucasRolff

    LucasRolff Well-Known Member

    Joined:
    May 27, 2013
    Messages:
    62
    Likes Received:
    48
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    When booted into the rescue image, did you go to /etc/selinux/config on SSH, or did you mount your partition on /mnt and changed it there? the /etc/selinux/config file in rescue, is the selinux config for the rescue image, and not your server.
     
    cPanelMichael likes this.
  13. WorkinOnIt

    WorkinOnIt Well-Known Member

    Joined:
    Aug 3, 2016
    Messages:
    168
    Likes Received:
    24
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    #13 WorkinOnIt, Dec 5, 2018
    Last edited: Dec 5, 2018
    cPanelMichael likes this.
  14. WorkinOnIt

    WorkinOnIt Well-Known Member

    Joined:
    Aug 3, 2016
    Messages:
    168
    Likes Received:
    24
    Trophy Points:
    18
    Location:
    UK
    cPanel Access Level:
    Root Administrator
    Glad you got it solved. Check this out - it worked for me.

    Installation Guide - System Requirements - Version 76 Documentation - cPanel Documentation

    Also read @cPanelMichael's thread here that might help to explain.
     
    #14 WorkinOnIt, Dec 5, 2018
    Last edited by a moderator: Dec 5, 2018
  15. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello Everyone,

    I updated my previous post with some additional information for hosting providers and system administrators explaining why rebooting a system affected by this issue can take longer than normal if SELinux is not disabled prior to the reboot. I've also quoted this information below:

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  16. sparek-3

    sparek-3 Well-Known Member

    Joined:
    Aug 10, 2002
    Messages:
    1,923
    Likes Received:
    177
    Trophy Points:
    343
    cPanel Access Level:
    Root Administrator
    Does this affect OpenVZ?

    As far as I know, OpenVZ does not support SELinux, and so as such /etc/selinux/config does not exist.

    And OpenVZ system do not install the selinux-policy rpm on the CentOS 7.6 update, so I believe they would be unaffected.
     
  17. cPanelMichael

    cPanelMichael Technical Support Community Manager Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    47,528
    Likes Received:
    2,180
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @sparek-3,

    I don't have an OpenVZ system available for immediate testing, but as I understand the /etc/selinux/config file exists by default with any stock CentOS 7 installation (with the SELINUX=disabled value). Are you using a particular image or template to provision the servers?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. sparek-3

    sparek-3 Well-Known Member

    Joined:
    Aug 10, 2002
    Messages:
    1,923
    Likes Received:
    177
    Trophy Points:
    343
    cPanel Access Level:
    Root Administrator
    I just happen to have 1 OpenVZ CentOS 7 server. It does not have an /etc/selinux/config file and yum check-update does not list an selinux-policy package to be installed (I haven't yet updated to CentOS 7.6).

    I'm not sure how CentOS 7 was installed on this particular OpenVZ VPS. Perhaps others might chime in?

    I suspect I can add an /etc/selinux/config with SELINUX=disabled just to be safe, but I don't think it would make any difference.

    The issue appears to only pop up on systems where selinux-policy is updated.
     
  19. greektranslator

    greektranslator Well-Known Member

    Joined:
    Jun 5, 2011
    Messages:
    58
    Likes Received:
    0
    Trophy Points:
    56
    Location:
    Greece
    cPanel Access Level:
    Root Administrator
    These are the contents of my selinux config, what should I do?
    Code:
    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    #     enforcing - SELinux security policy is enforced.
    #     permissive - SELinux prints warnings instead of enforcing.
    #     disabled - No SELinux policy is loaded.
    SELINUX=permissive
    # SELINUXTYPE= can take one of three two values:
    #     targeted - Targeted processes are protected,
    #     minimum - Modification of targeted policy. Only selected processes are protected.
    #     mls - Multi Level Security protection.
    SELINUXTYPE=targeted
    
    
    
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  20. greektranslator

    greektranslator Well-Known Member

    Joined:
    Jun 5, 2011
    Messages:
    58
    Likes Received:
    0
    Trophy Points:
    56
    Location:
    Greece
    cPanel Access Level:
    Root Administrator
    This was the host reply (whom I pointed out to this thread):

    Your server got updated and with the new update is not able to boot correctly. Since the monitoring was enabled our system detected that and putted an intervention to check why the server was offline. The message you have received is the result of that intervention advising you to fix the software problems preventing your server from correctly booting into the OS.

    I checked the link that you have send to me and it refers to a configuration file that does not exist. You can reboot the server in rescue mode and create the missing configuration in order for the server to boot.

    From what I saw the server now is online and you are using one of the kernels of OVH. I suggest in this case you are able to log in to your server and start the configuration of the kernel in order to boot from your kernel and have the services running.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice