All accounts suddenly jailshells and passwords dont work??

[wzd]

Ok here is a pretty puzzle for everyone:

I wake up today and try logging in with my "wizard" account into my server. SSH keeps saying Access Denied even though i'm certain i'm typing in the correct password (I even type it one finger at a time). - I get some coffee and go about unblocking myself from the firewall (brute force protection)

After a long amount of digging and managing to use the ConfigServer Explorer (THANK GOD FOR IT) http://www.configserver.com/cp/cse.html
. I was able to re-edit the sshd_config and re-enable direct root access.

ALL the accounts in /etc/password were like this one:

sharp:x:32070:32071::/home/sharp:/usr/local/cpanel/bin/jailshell

This is INCLUDING my "wizard" account which wasn't accepting my password. (It was a bash account before)

I then used passwd to reset the password for "wizard" account - Still couldn't su from the wizard account (access denied) until i also modded the account to be using /bin/bash even though it was part of wheel

-- What is the meaning of this? How did ALL THE ACCOUNTS suddenly covert to jailshell? and none of the passwords work for anyone! (They are unable to get in via Cpanel or SSH or FTP) I've tested one or two of my own personal Cpanel accounts on the same server and i have to reset the password ...

It now looks like we have to reset every single password for every single account - POP3 passwords and everything else are working fine. It's just the main users password.

Looking forward to finding out how the hell this happened?


Some MISC system information below:

Linux 2.6.9-42.0.10.EL #1 Tue Feb 27 09:24:42 EST 2007 i686 athlon i386 GNU/Linux

-- A debug code from trying to login from another shell

OpenSSH_3.9p1, OpenSSL 0.9.7a Feb 19 2003
debug2: ssh_connect: needpriv 0
debug1: Connecting to coder.devb0x.net [216.32.75.90] port 22.
debug1: Connection established.
debug1: identity file /home/wizard/.ssh/identity type -1
debug1: identity file /home/wizard/.ssh/id_rsa type -1
debug1: identity file /home/wizard/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_3.9p1
debug1: match: OpenSSH_3.9p1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_3.9p1

[brendanrtg]

You server have probably been hacked.

Do some screening for changed files, there are many scripts at out there that can detect root kit and whatnots.

[wzd]

That's what i thought at first as well but it's not so - The server was locked down quite well by ConfigServer people before and i had RK hunter running on it plus reports of all sorts.

All the latest software has been installed and i would have received notifications via email of major file changes in the important directories - I had directory watching on system directories and i have SSH logins for all users notifications...

Also no funny accounts found - Root password was the same

I strongly suspect it's something else but what it is i have no idea,

[Koreru]

Anyone ever find the reason for this?

We just had it happen on one of our servers.

(Root password is the same/No rootkits detected)

[cpanelkenneth]

Please open a support ticket at https://tiockets.cpanel.net/submit/ so we can investigate and resolve the issue.