Status
Not open for further replies.

Tam

Well-Known Member
Jul 31, 2004
109
2
168
Just received this:
What is missing from the message is that if using keys you should delete the private key from the server straight after you have downloaded it. Note what ISC Diary says "Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys"
 

chrismfz

Well-Known Member
Jul 4, 2007
125
1
68
Greece
cPanel Access Level
DataCenter Provider
Should also give attention to:

We all have been so worried on the sshd aspect to this.
But we forgot to take a look at 'ssh'.. this little lonely binary used to initiate ssh connections with other servers.

Well...

You go and login to another server using an infected machine (which you may not know is infected).

Guess what happens, our well known friend here:

Quote:
5297 connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("72.156.139.154")}, 16) = 0
shows up again, sending your other servers login details out.

So really all you need is 1 compromised server, to have multiple.. if you use your server to login to other servers.

They are in memory too.
 

Tam

Well-Known Member
Jul 31, 2004
109
2
168
What is missing from the message is that if using keys you should delete the private key from the server straight after you have downloaded it. Note what ISC Diary says "Notice that the rootkit can steal username and password pairs as well as RSA and DSA private keys"
Create the keys locally using PuttyGen and upload the public key only to the server should prevent any private key theft.
 

tomfrog

Registered
Feb 22, 2013
4
0
1
cPanel Access Level
Root Administrator
DirectAdmin and Plesk server were exploited because someone logged into those servers from a cPanel server. All those username and passwords were stored in memory.

@Steven from WHT knows this rootkit in and out. He did a fantastic research. If you read the thread at WHT, you will find a lot of relevant information. Most of what you need to know is in that thread.

Let's support cPanel!
 

Tam

Well-Known Member
Jul 31, 2004
109
2
168
DirectAdmin and Plesk server were exploited because someone logged into those servers from a cPanel server. All those username and passwords were stored in memory.
If you read the entire thread (which I wouldn't recommend given it's extreme length), although I have because I started following it just after it started: you will discover that is not true. Please re-read it. Infection came via local rootkits so far as is currently known, not just via cPanel.

@Steven from WHT knows this rootkit in and out. He did a fantastic research. If you read the thread at WHT, you will find a lot of relevant information. Most of what you need to know is in that thread.
You've missed a lot of it, understandable because it is so huge.

Let's support cPanel!
You see the contradiction there? With your first sentence you incorrectly state that it spread through a cPanel server, that is not true.
 
Last edited:

tomfrog

Registered
Feb 22, 2013
4
0
1
cPanel Access Level
Root Administrator
If you read the entire thread (which I wouldn't recommend given it's extreme length), although I have because I started following it just after it started: you will discover that is not true. Please re-read it. Infection came via local rootkits so far as is currently known, not just via cPanel.



You've missed a lot of it, understandable because it is so huge.



You see the contradiction there? With your first sentence you incorrectly state that it spread through a cPanel server, that is not true.

Read this again...

ISC Diary | SSHD rootkit in the wild

Especially the UPDATE part...

I don't speculate. It mentions unSpawn and Steven. Read the memory part carefully...
 

tomfrog

Registered
Feb 22, 2013
4
0
1
cPanel Access Level
Root Administrator
If you read the entire thread (which I wouldn't recommend given it's extreme length), although I have because I started following it just after it started: you will discover that is not true. Please re-read it. Infection came via local rootkits so far as is currently known, not just via cPanel.

You've missed a lot of it, understandable because it is so huge.

You see the contradiction there? With your first sentence you incorrectly state that it spread through a cPanel server, that is not true.

You're probably right. I didn't read the entire thread.

I'm here to support cPanel. I sometimes forget that not everyone has access to the same information. Based on the information you read at WHT, I guess you're on track...

Have a great day!
 

Infopro

Well-Known Member
May 20, 2003
17,113
513
613
Pennsylvania
cPanel Access Level
Root Administrator
Twitter

lbeachmike

Well-Known Member
Dec 27, 2001
306
1
316
Long Beach, NY
cPanel Access Level
Root Administrator
cPanelJeff is in on this thread with some very valuable information, on the cPanel forums:
http://forums.cpanel.net/f185/sshd-rootkit-323962.html#post1324161

Not exactly sure what you're looking for here.
Those posts are four and five days old respectively with no further activity since, whereas everything has since changed.

I'd be interested in knowing more details about what was found relating to the internal exploit, and if cPanel is not willing to release that information, then that could be stated as well. It would further be helpful if cPanel provided its own guidance/recommendations for temporarily cleanup measures rather than relying on users to stumble across scripts and different variant recommendations within the 85-page WHT thread, many of which have caused less experienced admins to crash their servers.

If the cPanel forum is to be any bit useful, it would be good if for one of the most critical issues we've seen in some time, for users to know that they could visit the cPanel forums and get the needed guidance versus reading through a very complex 85-page thread on WHT and hopefully understanding enough details to implement the appropriate measures.

I've read the entire 85-page (and growing) thread and you can see that many less experienced admins have been confused.

We'd also, of course, like to know that cPanel will be taking measures to enhance security in some way through the support process without making it cumbersome, and it would productive to hear or collaborate on ideas as to how that can be accomplished.

You guys have always offered the highest quality ticket-based support in the business. That should show through here as well.

Thanks.
 

chrismfz

Well-Known Member
Jul 4, 2007
125
1
68
Greece
cPanel Access Level
DataCenter Provider
Credits should also go to spender from grsecurity who got a copy of the lib from Steven and managed to find out exactly how it works.

grsec kernels couldn't even been infected btw. :D nice one
 

JeffP.

Well-Known Member
Sep 28, 2010
164
15
68
Brad Spengler (spender) [ grsecurity.net ] truly is one of the most brilliant minds around today. His passion for finding things that are terribly broken and fixing them is a dire necessity. He deserves a huge thanks from all of us (you, me, etc) for his efforts not only with grsec, but for providing invaluable info about this latest libkeyutils mess. Thank you, Brad. Your efforts do not go unnoticed.

I'd be interested in knowing more details about what was found relating to the internal exploit, and if cPanel is not willing to release that information, then that could be stated as well.
Unfortunately there are no additional details at this time. This is from an excerpt of the email sent out to customers last night (not everyone received the email, so I am pasting this part of it here):

As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.

It would further be helpful if cPanel provided its own guidance/recommendations for temporarily cleanup measures rather than relying on users to stumble across scripts and different variant recommendations within the 85-page WHT thread, many of which have caused less experienced admins to crash their servers.
There is no safe way to clean up a rooted server, other than to reload the OS (and even then, you still have to trust your installation media, so that's not 100% foolproof either).

The code is compiled, and intentionally obfuscated to some degree. There are also variants of the malicious library. There are also rpms that may be installed or may have been installed which (edit: may) relate to the issue. Finally - and by far most importantly - there is no way to search for every possible backdoor that may have been set up by the attacker(s). I should also note that any steps to get rid of known malicious software on one server may have no effect or even disastrous effects on another.

The reasons above are why no officially recommended cleanup steps are mentioned.

If the cPanel forum is to be any bit useful, it would be good if for one of the most critical issues we've seen in some time, for users to know that they could visit the cPanel forums and get the needed guidance versus reading through a very complex 85-page thread on WHT and hopefully understanding enough details to implement the appropriate measures.
If there were more information to be shared at this time, I would see to it that it were shared. What I can do personally is recommend steps to help prevent something like this from occurring again (and steps are being taken internally, though that information is internal at this time):

  1. Restrict which hosts can access sshd on your server
  2. Disallow root logins to sshd
  3. Use ssh keys instead of passwords for authentication
  4. Use a wheel or sudo user that can escalate to root privileges after logging in
  5. Use a temporary account with a unique password when providing login information to a third party, and remove the account when the task is complete

As a result of this particular issue, it is imperative to change passwords, and never reuse any of the old passwords again. Additionally, reusing the same password for more than 1 account (no matter what the account is, what server it exists on, what it's used for, etc) is one of the easiest ways for an attacker to gain access to other resources that you utilize. So, don't reuse passwords, not for anything.

We'd also, of course, like to know that cPanel will be taking measures to enhance security in some way through the support process without making it cumbersome, and it would productive to hear or collaborate on ideas as to how that can be accomplished.
There are actions being taken at this time which are being tested. Good things take time for proper implementation, review, and testing. I cannot elaborate on this (yet, anyway), but changes are being made accordingly. More info to come.
 

ffeingol

Well-Known Member
PartnerNOC
Nov 9, 2001
532
160
343
cPanel Access Level
DataCenter Provider
To me one of the 'issues' is that there does not seem to be any way to actually determine if you are affected or not. From what I've read in the various thread changing passwords/keys is useless if you are affected.
 

quietFinn

Well-Known Member
Feb 4, 2006
1,222
87
178
Finland
cPanel Access Level
Root Administrator
There never is. Either you know you are affected, or you believe (and hope) that you are not.
 

lbeachmike

Well-Known Member
Dec 27, 2001
306
1
316
Long Beach, NY
cPanel Access Level
Root Administrator
To me one of the 'issues' is that there does not seem to be any way to actually determine if you are affected or not. From what I've read in the various thread changing passwords/keys is useless if you are affected.
Hey Frank - long time - nice to see you here : ) The recommendations of changing passwords/keys is not a recommendation for dealing with the aftermath of the attack. Those are recommendations for minimizing risk of a future compromise. If you are looking to cleanup, there are temporary measures to get the known rootkit out of your system and then it is recommended to do a full clean OS install.

To temporarily remove the rootkit, I used the following steps, modified from a post by Hivelocity in the WHT thread -

Code:
1. SSH to the server
2. Go into WHM -> Service Manager and disable sshd - that will keep your current session up, but prevent any new sessions
3. cd /lib64/
4. rm libkeyutils.so.1.9 (this may be different on your machine - you've got to look at which variant you have)
5. rm libkeyutils.so.1
6. ln -s libkeyutils.so.1.3 libkeyutils.so.1
7. Restart ssh
8. Reboot to close any active connections
In some cases files were made immutible, so you may have to do chattr -i on the file in order to delete it.

You should be able to check which variant you have by simply doing ls -al /lib64/libkey*

If using the above steps, before you reboot, things should look like this -

[email protected]<yourserver> [/lib64]# ls -al libkey*
lrwxrwxrwx 1 root root 18 Feb 21 22:56 libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 12592 Jun 22 2012 libkeyutils.so.1.3*

If not, you will encounter some serious problems when you reboot.

If you are not running 64-bit OS, then you simply substitute the above /lib64 with just /lib

Hope this helps.
 

ffeingol

Well-Known Member
PartnerNOC
Nov 9, 2001
532
160
343
cPanel Access Level
DataCenter Provider
We don't actually believe we have any affected servers. Our problem is we have a lot of cPanel servers to check, so we were just looking for a more automated way to do it and no one seems to agree on how to verify if you are infected. I've seen checking for utils.so.1.9 file, running rpm verification commands etc.
 

lbeachmike

Well-Known Member
Dec 27, 2001
306
1
316
Long Beach, NY
cPanel Access Level
Root Administrator
We don't actually believe we have any affected servers. Our problem is we have a lot of cPanel servers to check, so we were just looking for a more automated way to do it and no one seems to agree on how to verify if you are infected. I've seen checking for utils.so.1.9 file, running rpm verification commands etc.
Well, AVG is now detecting this rootkit, so if you run AVG on these servers and scan your lib and/or lib64 directories, you should detect the ones with the exploit. Make sure not to connect from one server to another because you will cross-contaminate your clean servers.

I'd suggest posting on the WHT thread re: a more automated way of checking. I know that CloudLinux also posted a detection script.
 
Status
Not open for further replies.