Status
Not open for further replies.

lbeachmike

Well-Known Member
Dec 27, 2001
307
4
318
Long Beach, NY
cPanel Access Level
Root Administrator
Well, AVG is now detecting this rootkit, so if you run AVG on these servers and scan your lib and/or lib64 directories, you should detect the ones with the exploit. Make sure not to connect from one server to another because you will cross-contaminate your clean servers.

I'd suggest posting on the WHT thread re: a more automated way of checking. I know that CloudLinux also posted a detection script.
Frank - Are you not running CSF on these servers? The current CSF version detects the issue and emails you an alert.
 

Tam

Well-Known Member
Jul 31, 2004
112
10
168
Read this again...

ISC Diary | SSHD rootkit in the wild

Especially the UPDATE part...

I don't speculate. It mentions unSpawn and Steven. Read the memory part carefully...
I have been following that thread, I don't see your point though?

- - - Updated - - -

Brad Spengler (spender) [ grsecurity.net ] truly is one of the most brilliant minds around today. His passion for finding things that are terribly broken and fixing them is a dire necessity. He deserves a huge thanks from all of us (you, me, etc) for his efforts not only with grsec, but for providing invaluable info about this latest libkeyutils mess. Thank you, Brad. Your efforts do not go unnoticed.



Unfortunately there are no additional details at this time. This is from an excerpt of the email sent out to customers last night (not everyone received the email, so I am pasting this part of it here):

As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel's security team is continuing to investigate the nature of this security issue.



There is no safe way to clean up a rooted server, other than to reload the OS (and even then, you still have to trust your installation media, so that's not 100% foolproof either).

The code is compiled, and intentionally obfuscated to some degree. There are also variants of the malicious library. There are also rpms that may be installed or may have been installed which (edit: may) relate to the issue. Finally - and by far most importantly - there is no way to search for every possible backdoor that may have been set up by the attacker(s). I should also note that any steps to get rid of known malicious software on one server may have no effect or even disastrous effects on another.

The reasons above are why no officially recommended cleanup steps are mentioned.



If there were more information to be shared at this time, I would see to it that it were shared. What I can do personally is recommend steps to help prevent something like this from occurring again (and steps are being taken internally, though that information is internal at this time):

  1. Restrict which hosts can access sshd on your server
  2. Disallow root logins to sshd
  3. Use ssh keys instead of passwords for authentication
  4. Use a wheel or sudo user that can escalate to root privileges after logging in
  5. Use a temporary account with a unique password when providing login information to a third party, and remove the account when the task is complete

As a result of this particular issue, it is imperative to change passwords, and never reuse any of the old passwords again. Additionally, reusing the same password for more than 1 account (no matter what the account is, what server it exists on, what it's used for, etc) is one of the easiest ways for an attacker to gain access to other resources that you utilize. So, don't reuse passwords, not for anything.



There are actions being taken at this time which are being tested. Good things take time for proper implementation, review, and testing. I cannot elaborate on this (yet, anyway), but changes are being made accordingly. More info to come.
Since the rootkit hooks the PEM_write_RSAPrivateKey and PEM_write_DSAPrivateKey functions (and potentially the ssh-keygen utility), could you please update your advice to everyone to advise that they generate key pairs locally (using something like PuttyGen for instance) and upload the public key only? I believe that this is VERY important.
 
Last edited:

kitchin

Member
Sep 18, 2011
24
0
51
cPanel Access Level
Root Administrator
The defining feature seems to be SSH using shared memory. NetworkPanda at WHT suggests running this shell command to look for processes doing that: www.webhostingtalk.com/showpost.php?p=8571571&postcount=1316

(In case the link is lost, it filters "ps aux" through the entries in "ipcs -mp", hence:
Code:
for i in `ipcs -mp | grep -v cpid | awk {'print $3'} | uniq`; do ps aux | grep $i | grep -v grep;done
but go over to WHT to read NetworkPanda's full explanation first.)
 
Last edited:

hicom

Well-Known Member
May 23, 2003
296
7
168
Just curious, what is the conclusion on how these hacks are occurring? is this the result of a vulnerability or a stolen password/key?
 

fcbinfo

Well-Known Member
Dec 10, 2006
111
3
168
cPanel Access Level
Root Administrator
Im going to install a new clean cpanel instalation on a new server now.

With this new instalation we have the same problem or i can ignore it?

Thanks.
 

SageBrian

Well-Known Member
Jun 1, 2002
413
2
318
NY/CT (US)
cPanel Access Level
Root Administrator
Im going to install a new clean cpanel instalation on a new server now.

With this new instalation we have the same problem or i can ignore it?

Thanks.
Let us know how you make out. I was planning on getting new server anyway, but without knowing how they acquired access, I don't want to go through the trouble of moving all to a new server only to find it follows me.
 

patchwork

Well-Known Member
Nov 2, 2001
95
0
316
Here is an email copied from another forum, for some reason I did not get this email myself even though I had 2 servers hacked.

cPanel, Inc. Announces Additional Internal Security Enhancements

This is a follow up on the status of the security compromise that cPanel, Inc. experienced on Thursday, February 21, 2013.

As mentioned in our email sent to cPanel Server Administrators who’ve opened a ticket with us in the past 6 months, on February 21 we discovered that one of the proxy servers we utilize in the technical support department had been compromised. The cPanel Security Team’s investigation into this matter is ongoing.

We’d like to relay additional details about the intrusion that we have gathered with you, and we want to explain what preventative measures we’re putting in place that will introduce additional layers of security to our new and existing systems, already in place. How the server was accessed and compromised is not clear, but we know a few key facts that we’re sharing.

Here’s what we know:

* The proxy machine compromised in this incident was, at the time, utilized to access customer servers by some of our Technical Analysts. It's intent was to provide a layer of security between local & remote workstations and customer servers.

* This proxy machine was compromised by a malicious third-party by compromising a single workstation used by one of our Technical Analysts.

* Only a small group of our Technical Analysts uses this particular machine for logins.

* There is no evidence that any sensitive customer data was exposed and there is no evidence that the actual database was compromised.
Here’s what we’re doing about it:

Documentation is now provided at: Determine Your System's Status which we encourage system administrators to use to determine the status of their machine.

We have restructured the process used to access customer servers to significantly reduce the risk of this type of sophisticated attack in the future. We have also been working on implementing multiple changes to our internal support systems and procedures as outlined for your information below.

* Our system will now generate and provide you with a unique SSH key for each new support ticket submitted.

* We are providing tools to authorize and de-authorize SSH keys and instructions on how to use them whenever you submit a ticket.

* Our system will generate a single-use username and password credentials for accessing WebHost Manager that are only valid while our staff is logged into your server.

* Additional enhancements are also planned behind the scene that should be transparent to our customers.

With these new layers of security in place, it is now possible for our Technical Analysts to service your support requests without you providing your server’s password for nearly all requests involving machines running our cPanel & WHM product going forward. However, we will still offer the ability to provide your password for server migrations, or in the event you cannot use SSH keys.

cPanel’s Internal Development Team has been working on an automated solution with the end goal of eliminating the need for our Technical Analysts to view any passwords you provide during the ticket submission process. We are testing this solution right now, and hope to have it fully implemented in the next few days.

cPanel, Inc. understands your concerns expressed over the last few days, and we very much appreciate the cooperation and patience you have provided us during this time as we work through all of this.

Thank you.
 

waddy

Member
Aug 26, 2008
10
0
51
We got hacked by this, rebuilt the server and changed ssh port. We still get heaps of hits on the old port. The Ip's are below if you want to block them at firewall for extra protection.

212.58.0.195
50.7.221.34
178.33.232.117
 
Status
Not open for further replies.