cPanel Root Password screw-up

[bizzy]

A multiple account transfer between cPanel servers suddenly stopped. The problem was that the root password (I was using the su escalation) suddenly was not recognised. Oh well, WHM has a change root password options so I did that from a wheel account. It did not work. No matter what I changed it to - I could not login to WHM as root or SSH in directly or by su-ing from another account. Restarted SSH of course to no effect.

It is at these times you think you have a borked server.

Luckily I had key authentication from another server and was able to get into root that way. Did a passwd, tokens were updated but it didn't actually deliver a working password. I was really panicking now. I eventually solved the issue by creating (as root) another user then giving it root powers AND UID 0. I then SSH into this other account and passwd root. It worked!!!

Password access is back on both SSH and WHM. How crazy was that? And without key authentication the server would have been effectively bricked.

Any ideas on what went wrong and hence how can I avoid a repetition? I'm guess its really a Linux/Centos 6.3 issue but cPanel was probably the cause and cPanel Root password change could not bring it back.

Meanwhile I'm going to have a few stiff whiskies and preach key authentication to anybody who will listen. Saved my day!

 

[Jeff Shotnik]

Without showing a command history it would be hard to say what happened. I've setup/migrated hundreds of cpanel systems with rhel/centos and have never had the issue you described.

"And without key authentication the server would have been effectively bricked."
>> Not exactly. You can always reset the root password by rebooting into single user mode and setting a new password from that run level.

 

[bizzy]

Without showing a command history it would be hard to say what happened. I've setup/migrated hundreds of cpanel systems with rhel/centos and have never had the issue you described.

There is no relevant command history in root. Simply the stuff i did long before the incident and then my attempts to extricate myself from the issue and after.

I also checked /var/cpanel/logs on the other server. It shows a successful transfer of accounts up to the breakage and after the fix. Nothing in /var/cpanel/logs on the damaged server. Where might I find a trail of failed attempts?

"And without key authentication the server would have been effectively bricked."
>> Not exactly. You can always reset the root password by rebooting into single user mode and setting a new password from that run level.

I didn't think this an option. This was a production server (which is why I was backing up accounts to another server) in which root had lost the ability to set a password. This solution is usually proposed when the password has just been 'lost' and needs to be reset.

That's the key. How can root lose the ability to set its own password? It still had root privileges (UID=0) otherwise I could not have been able to setup the pseudo root to change root's password.

Trawling through this forum from the suggested links below there are instances where loss of root password has been associated with account transfers. Like you - I have done it hundreds of times without issue. The ease and reliability is the one reason I continue to specify cPanel for our servers!

So I'm badly shaken and feel I am rooting on eggshells atm. Which is why i need to understand what exactly happened. Please help ...

 

[cPanelMichael]

Hello

Did you check to see if the "root" user was blocked by cPHulk brute force detection? You can find this option at:

"WHM Home » Security Center » cPHulk Brute Force Protection"

The log file is located at:

/usr/local/cpanel/logs/cphulkd.log

Thank you.

 

[bizzy]

Nope, checked nothing blocked by cpHulk since Feb 3rd when I turned SSH off. I had only re-enabled it about half an hour earlier to enable Account Transfer.

Also I could SSH into another account but su-ing was blocked by password problem. And it doesn't explain why, when in root, I could not change password even though passwd did get the usual 'tokens updated' response. AFAIK its an internal root password verification issue that is at fault and not an external connection.

cPanel is guilty by association in that it was presumably rapidly logging in during escalation when transferring accounts and could be contributory to corrupting password verification. I've been through /var/log/message on both servers and not found anything interesting or coincidental. Still totally mystified.

I'm speculating that the password was not the issue at all but rather root could not verify it for some reason. NB other users' passwords verified correctly. And this verifying issue was sorted by pseudo root resetting root's password. Weird.