Locked out from server

[Yanuar Baskoro]

Hello fellow cPanel User.

This occur yesterday January 19th, 2018 6.45AM (+7GMT) I've got info from my fellow Tech Support that one of our server got "Starting sshd: Privilege separation user sshd does not exist FAILED" messages. Shortly all function goes down except WHM.
So I managed to access our server via IPMI console, but with no luck. I can't login with its credential, so I decide to reboot it.
In the process I found that my "/" mounted as -ro, I still cant login to it via console, but strangely I can access it via WHM interface and there's no active process user except dbus and nscd

What I've done is :
full fsck
- If I load Cloudlinux kernel 2.6.32-896.16.1.lve1.4.49.el6.x86_64, dracut will mount "/" as ro, indeed I can't login to console via IPMI and I can't login to WHM either because WHM tells me that its license is invalid/expired

- dracut didn't mount the "/" as ro with Cloudlinux kernel “2.6.32-673.26.1.lve1.4.30.el6.x86_64” . but I still can't logon to console via IPMI. WHM login is doable but I can't do anything further than submitting cPanel ticket, but they can't help further due to lack of access since we can't bring sshd up

 

[rpvw]

Have you tried the script that will provide a minimal, default configuration of SSH to allow you to log in. To use this script, you append the following to the url for WHM: /scripts2/doautofixer?autofix=safesshrestart

eg https://YourServerIP:2087/scripts2/doautofixer?autofix=safesshrestart

Take note of the output from running that script, as it may restart ssh on a port other than 22 if somehow the previous ssh instance is still running.

(Please note: If you are thinking about upgrading the Kernel to current, if you are running on Xen PV, the CloudLinux Kernels written to mitigate the Meltdown/Spectre exploits will not work, and you should migrate to Xen HVM)

 

[Yanuar Baskoro]

I have tried that but no luck, sshd refuse to start. I've contacted Cloudlinux about this issue. I'll update this thread since they haven't seen issue like this

Thank you for replying anyway

 

[cPanelMichael]

I've contacted Cloudlinux about this issue. I'll update this thread since they haven't seen issue like this

Hello,

Were they able to help you solve the issue?

Thank you.

 

[Yanuar Baskoro]

nope, they say my case is one of kind. A lot of package has been corrupted by unknown reason.

Now I manage to transfer our server to another server in rescue layer. I'll let they continue after we finished, so they can continue investigating the case. I have another 2 weeks before datacenter suspend my server.

Since mysql service refuses to start, I transfer all database as FRM, MYI and MYD. I'll let you know when they found something.

6TB of data are pain in the @## to move thou