Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

SSHD Rootkit

Discussion in 'Security' started by prajithp13, Feb 16, 2013.

Thread Status:
Not open for further replies.
  1. chrismfz

    chrismfz Well-Known Member

    Joined:
    Jul 4, 2007
    Messages:
    125
    Likes Received:
    1
    Trophy Points:
    68
    Location:
    Greece
    cPanel Access Level:
    DataCenter Provider
    Check the question again. Didn't say what. But how.

    That is known. And they say it again in the link you gave:
    At this point in time we still do not know what the initial attack vector is – it is unknown how the attackers get root access on the compromised servers
     
  2. Tam

    Tam Well-Known Member

    Joined:
    Jul 31, 2004
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    168
    What is missing from the message is that if using keys you should delete the private key from the server straight after you have downloaded it. Note what ISC Diary says "Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys"
     
  3. chrismfz

    chrismfz Well-Known Member

    Joined:
    Jul 4, 2007
    Messages:
    125
    Likes Received:
    1
    Trophy Points:
    68
    Location:
    Greece
    cPanel Access Level:
    DataCenter Provider
    Should also give attention to:

     
  4. Tam

    Tam Well-Known Member

    Joined:
    Jul 31, 2004
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    168
    Create the keys locally using PuttyGen and upload the public key only to the server should prevent any private key theft.
     
  5. Tam

    Tam Well-Known Member

    Joined:
    Jul 31, 2004
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    168
    Woah! Now WHT is down:

     
  6. tomfrog

    tomfrog Registered

    Joined:
    Feb 22, 2013
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    DirectAdmin and Plesk server were exploited because someone logged into those servers from a cPanel server. All those username and passwords were stored in memory.

    @Steven from WHT knows this rootkit in and out. He did a fantastic research. If you read the thread at WHT, you will find a lot of relevant information. Most of what you need to know is in that thread.

    Let's support cPanel!
     
  7. Tam

    Tam Well-Known Member

    Joined:
    Jul 31, 2004
    Messages:
    108
    Likes Received:
    1
    Trophy Points:
    168
    If you read the entire thread (which I wouldn't recommend given it's extreme length), although I have because I started following it just after it started: you will discover that is not true. Please re-read it. Infection came via local rootkits so far as is currently known, not just via cPanel.

    You've missed a lot of it, understandable because it is so huge.

    You see the contradiction there? With your first sentence you incorrectly state that it spread through a cPanel server, that is not true.
     
    #67 Tam, Feb 22, 2013
    Last edited: Feb 22, 2013
  8. tomfrog

    tomfrog Registered

    Joined:
    Feb 22, 2013
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator

    Read this again...

    ISC Diary | SSHD rootkit in the wild

    Especially the UPDATE part...

    I don't speculate. It mentions unSpawn and Steven. Read the memory part carefully...
     
  9. tomfrog

    tomfrog Registered

    Joined:
    Feb 22, 2013
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator

    You're probably right. I didn't read the entire thread.

    I'm here to support cPanel. I sometimes forget that not everyone has access to the same information. Based on the information you read at WHT, I guess you're on track...

    Have a great day!
     
  10. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    306
    Likes Received:
    1
    Trophy Points:
    316
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    Very disappointed that cPanel is not currently an active participant in any public discussion threads that I'm aware of.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. biodocs

    biodocs Member

    Joined:
    Jun 27, 2008
    Messages:
    7
    Likes Received:
    0
    Trophy Points:
    51
    Location:
    Regina
    I second that as loudly as i can ....
     
  12. Infopro

    Infopro cPanel Sr. Product Evangelist Staff Member

    Joined:
    May 20, 2003
    Messages:
    16,931
    Likes Received:
    485
    Trophy Points:
    583
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    cPanelJeff is in on this thread with some very valuable information, on the cPanel forums:
    http://forums.cpanel.net/f185/sshd-rootkit-323962.html#post1324161

    Not exactly sure what you're looking for here.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  13. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    306
    Likes Received:
    1
    Trophy Points:
    316
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    Those posts are four and five days old respectively with no further activity since, whereas everything has since changed.

    I'd be interested in knowing more details about what was found relating to the internal exploit, and if cPanel is not willing to release that information, then that could be stated as well. It would further be helpful if cPanel provided its own guidance/recommendations for temporarily cleanup measures rather than relying on users to stumble across scripts and different variant recommendations within the 85-page WHT thread, many of which have caused less experienced admins to crash their servers.

    If the cPanel forum is to be any bit useful, it would be good if for one of the most critical issues we've seen in some time, for users to know that they could visit the cPanel forums and get the needed guidance versus reading through a very complex 85-page thread on WHT and hopefully understanding enough details to implement the appropriate measures.

    I've read the entire 85-page (and growing) thread and you can see that many less experienced admins have been confused.

    We'd also, of course, like to know that cPanel will be taking measures to enhance security in some way through the support process without making it cumbersome, and it would productive to hear or collaborate on ideas as to how that can be accomplished.

    You guys have always offered the highest quality ticket-based support in the business. That should show through here as well.

    Thanks.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. chrismfz

    chrismfz Well-Known Member

    Joined:
    Jul 4, 2007
    Messages:
    125
    Likes Received:
    1
    Trophy Points:
    68
    Location:
    Greece
    cPanel Access Level:
    DataCenter Provider
    Credits should also go to spender from grsecurity who got a copy of the lib from Steven and managed to find out exactly how it works.

    grsec kernels couldn't even been infected btw. :D nice one
     
  15. JeffP.

    JeffP. Well-Known Member

    Joined:
    Sep 28, 2010
    Messages:
    164
    Likes Received:
    15
    Trophy Points:
    68
    Brad Spengler (spender) [ grsecurity.net ] truly is one of the most brilliant minds around today. His passion for finding things that are terribly broken and fixing them is a dire necessity. He deserves a huge thanks from all of us (you, me, etc) for his efforts not only with grsec, but for providing invaluable info about this latest libkeyutils mess. Thank you, Brad. Your efforts do not go unnoticed.

    Unfortunately there are no additional details at this time. This is from an excerpt of the email sent out to customers last night (not everyone received the email, so I am pasting this part of it here):

    As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.

    There is no safe way to clean up a rooted server, other than to reload the OS (and even then, you still have to trust your installation media, so that's not 100% foolproof either).

    The code is compiled, and intentionally obfuscated to some degree. There are also variants of the malicious library. There are also rpms that may be installed or may have been installed which (edit: may) relate to the issue. Finally - and by far most importantly - there is no way to search for every possible backdoor that may have been set up by the attacker(s). I should also note that any steps to get rid of known malicious software on one server may have no effect or even disastrous effects on another.

    The reasons above are why no officially recommended cleanup steps are mentioned.

    If there were more information to be shared at this time, I would see to it that it were shared. What I can do personally is recommend steps to help prevent something like this from occurring again (and steps are being taken internally, though that information is internal at this time):

    1. Restrict which hosts can access sshd on your server
    2. Disallow root logins to sshd
    3. Use ssh keys instead of passwords for authentication
    4. Use a wheel or sudo user that can escalate to root privileges after logging in
    5. Use a temporary account with a unique password when providing login information to a third party, and remove the account when the task is complete

    As a result of this particular issue, it is imperative to change passwords, and never reuse any of the old passwords again. Additionally, reusing the same password for more than 1 account (no matter what the account is, what server it exists on, what it's used for, etc) is one of the easiest ways for an attacker to gain access to other resources that you utilize. So, don't reuse passwords, not for anything.

    There are actions being taken at this time which are being tested. Good things take time for proper implementation, review, and testing. I cannot elaborate on this (yet, anyway), but changes are being made accordingly. More info to come.
     
  16. ffeingol

    ffeingol Well-Known Member PartnerNOC

    Joined:
    Nov 9, 2001
    Messages:
    215
    Likes Received:
    1
    Trophy Points:
    318
    cPanel Access Level:
    DataCenter Provider
    To me one of the 'issues' is that there does not seem to be any way to actually determine if you are affected or not. From what I've read in the various thread changing passwords/keys is useless if you are affected.
     
  17. quietFinn

    quietFinn Well-Known Member

    Joined:
    Feb 4, 2006
    Messages:
    997
    Likes Received:
    11
    Trophy Points:
    168
    Location:
    Finland
    cPanel Access Level:
    Root Administrator
    There never is. Either you know you are affected, or you believe (and hope) that you are not.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  18. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    306
    Likes Received:
    1
    Trophy Points:
    316
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    Hey Frank - long time - nice to see you here : ) The recommendations of changing passwords/keys is not a recommendation for dealing with the aftermath of the attack. Those are recommendations for minimizing risk of a future compromise. If you are looking to cleanup, there are temporary measures to get the known rootkit out of your system and then it is recommended to do a full clean OS install.

    To temporarily remove the rootkit, I used the following steps, modified from a post by Hivelocity in the WHT thread -

    Code:
    1. SSH to the server
    2. Go into WHM -> Service Manager and disable sshd - that will keep your current session up, but prevent any new sessions
    3. cd /lib64/
    4. rm libkeyutils.so.1.9 (this may be different on your machine - you've got to look at which variant you have)
    5. rm libkeyutils.so.1
    6. ln -s libkeyutils.so.1.3 libkeyutils.so.1
    7. Restart ssh
    8. Reboot to close any active connections
    In some cases files were made immutible, so you may have to do chattr -i on the file in order to delete it.

    You should be able to check which variant you have by simply doing ls -al /lib64/libkey*

    If using the above steps, before you reboot, things should look like this -

    [email protected]<yourserver> [/lib64]# ls -al libkey*
    lrwxrwxrwx 1 root root 18 Feb 21 22:56 libkeyutils.so.1 -> libkeyutils.so.1.3*
    -rwxr-xr-x 1 root root 12592 Jun 22 2012 libkeyutils.so.1.3*

    If not, you will encounter some serious problems when you reboot.

    If you are not running 64-bit OS, then you simply substitute the above /lib64 with just /lib

    Hope this helps.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  19. ffeingol

    ffeingol Well-Known Member PartnerNOC

    Joined:
    Nov 9, 2001
    Messages:
    215
    Likes Received:
    1
    Trophy Points:
    318
    cPanel Access Level:
    DataCenter Provider
    We don't actually believe we have any affected servers. Our problem is we have a lot of cPanel servers to check, so we were just looking for a more automated way to do it and no one seems to agree on how to verify if you are infected. I've seen checking for utils.so.1.9 file, running rpm verification commands etc.
     
  20. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    306
    Likes Received:
    1
    Trophy Points:
    316
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    Well, AVG is now detecting this rootkit, so if you run AVG on these servers and scan your lib and/or lib64 directories, you should detect the ones with the exploit. Make sure not to connect from one server to another because you will cross-contaminate your clean servers.

    I'd suggest posting on the WHT thread re: a more automated way of checking. I know that CloudLinux also posted a detection script.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...
Thread Status:
Not open for further replies.

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice