Server Was Offline After Kernel Upgrade and Server Reboot

[Archmactrix]

My VPS server was offline yesterday after upgrading the kernel and rebooting so I would like to know what might have been the cause of it.

The system is:
CentOS 7.2 x86_64 kvm
cPanel 54.0 (build 15)

There was automatic system upgrade yesterday with critical security update for vulnerability glibc CVE-2015-7547. I ran yum update too to upgrade the kernel as it wasn't downloaded and installed at the same time as the automatic upgrade.

But the server didn't come online again after rebooting it until I choosed Virtio network card in the SolusVM control panel. I have upgraded the kernel and rebooted few times before on the same installation this winter so this is different.

I also see that NetworkManager has been disabled now, which is perhaps not bad, and I would like to keep it that way.

 

[syslint]

There is a chance that your NIC kernel module is missing from the latest kernel. Can you get the boot logs ?

 

[Archmactrix]

There is a chance that your NIC kernel module is missing from the latest kernel. Can you get the boot logs ?

I have a boot.log file with few hundred lines and can insert it here.

I see these two lines here (there is no other failed or colored entries besides a mailman entry which is disabled):

[ INFO ] Network Manager is not active.
[DEPEND] Dependency failed for Network Manager Wait Online.

I have these messages log error and warn entries from this week:

The failed reboot was initated at 23:4x and the successful boot at
03:2x.

Feb 17 05:13:10 serverhostname kdumpctl: cat: write error: Broken pipe
Feb 17 23:43:48 serverhostname p0f: [!] WARNING: User-initiated shutdown.
Feb 17 23:43:48 serverhostname NetworkManager[31823]: <warn>  disconnected by the system bus.
Feb 17 23:44:08 serverhostname network: Bringing up interface eth0:  ERROR    : [/etc/sysconfig/network-scripts/ifup-eth] Device eth0 does not seem to be present, delaying initialization.
Feb 17 23:44:54 serverhostname kdumpctl: cat: write error: Broken pipe
Feb 17 23:51:36 serverhostname network: Bringing up interface eth0:  ERROR    : [/etc/sysconfig/network-scripts/ifup-eth] Device eth0 does not seem to be present, delaying initialization.
Feb 17 23:53:53 serverhostname p0f: [!] WARNING: User-initiated shutdown.
Feb 17 23:54:17 serverhostname network: Bringing up interface eth0:  ERROR    : [/etc/sysconfig/network-scripts/ifup-eth] Device eth0 does not seem to be present, delaying initialization.
Feb 18 00:54:29 serverhostname network: Bringing up interface eth0:  ERROR    : [/etc/sysconfig/network-scripts/ifup-eth] Device eth0 does not seem to be present, delaying initialization.
Feb 18 03:21:06 serverhostname network: Bringing up interface eth0:  ERROR    : [ipv6_add_route] 'No route to host' adding route '::/0' via gateway 'xxxx:xxxx:xxxx:0000:0000:0000:0000:0001' through device ''
Feb 18 03:21:16 serverhostname cpipv6: Cpanel::Exception::ProcessFailed::Error/(XID e3q8wz) “ip” reported error code “1” when it ended.
Feb 18 03:21:16 serverhostname cpipv6: Cpanel::Exception::new("Cpanel::Exception::ProcessFailed::Error", HASH(0x1f8c798)) called at /usr/local/cpanel/Cpanel/Exception.pm line 75
Feb 18 03:21:16 serverhostname cpipv6: Cpanel::Exception::__create("ProcessFailed::Error", HASH(0x1f8c798)) called at /usr/local/cpanel/Cpanel/ChildErrorStringifier.pm line 141

 

[cPanelMichael]

Hello

The change in the Network Manager package stems from this case, implemented in cPanel version 54.0.15:

Fixed case CPANEL-4140: Require NetworkManager on all C7 systems so network.online precedes multi-user.

Are you able to reproduce the issue in subsequent reboots?

Thank you.

 

[syslint]

Hi,
Also please note that eth0 is not the default name of interface in centos 7 . Even though the cpanel is setting it as eth0 without updating the grub. So it may some times take little time to restart the services with eth0 names too. Also check the network manager service too.

 

[syslint]

The following code may help too,

# sed -i~ '/^GRUB_CMDLINE_LINUX/s/"$/ net.ifnames=0"/' /etc/sysconfig/grub
# grub2-mkconfig -o /boot/grub2/grub.cfg

 

[Archmactrix]

Hello

The change in the Network Manager package stems from this case, implemented in cPanel version 54.0.15:

Fixed case CPANEL-4140: Require NetworkManager on all C7 systems so network.online precedes multi-user.

Are you able to reproduce the issue in subsequent reboots?

Thank you.

I didn't reboot to reproduce this, but I updated the grub file now with:

net.ifnames=0

and rebooted now.

I found the below messages log entry for the time when I rebooted the OS yesterday and it didn't come online again:

Feb 17 23:44:06 myhostname systemd: Job NetworkManager-wait-online.service/start failed with result 'dependency'.

The following code may help too,

# sed -i~ '/^GRUB_CMDLINE_LINUX/s/"$/ net.ifnames=0"/' /etc/sysconfig/grub
# grub2-mkconfig -o /boot/grub2/grub.cfg

Yes thank you, I updated the grub file using the above commands and rebooted.

 

[syslint]

Happy to here your issue is resolved . I think the cpanel developers should consider updating grub to avoid this slow reboot problem while renaming the NIC

 

[marciolinuxforce]

The following code may help too,

# sed -i~ '/^GRUB_CMDLINE_LINUX/s/"$/ net.ifnames=0"/' /etc/sysconfig/grub
# grub2-mkconfig -o /boot/grub2/grub.cfg

I had the same problem on the production server and it worked perfectly. Thank you very much.