Strange Virus On cPanel server

[rpmws]

Over the past few months ..almost a year I have seen servers with logs of a particular IP address logging into FTP across several accounts in a matter of seconds and defacing them. The logins do NOT fail ..not even once. Then you go in and change any of the passwords on any of the accounts and you wind up with failing logins. This clearly indicates that these are NOT compromises in the security of a server. It means that someone has the keys (user and passwd) or at one point had the master keys to root or a reseller password with access to many accounts.

The fact that this has gone undetected in many cases explains this. I believe that in 99% of the cases the passwords to either reseller accounts, root accounts or simgle accounts are being collected and later used. I have NEVER felt that the breaking of software or services on these servers are how these defacements happen. Of course there are the few that have happened that way ..but these where no one can find the obvious answer and only find evidence of what looks like a real user doing his or her thing is present is obviously coming from real credentials being passed to gain access. This can happen months later. This sort of thing happens all the time and I can bet someone in here has had it happen in the following way. Back in 2005 I had a reseller signup with me. He moved over about 50 accounts and was running with out company just fine. Out of the blue ..all of his sites were defaced and some of them deleted completely. Of course we thought the box had been compromised, hacked ..whatever. I couldn't find a thing to that affect. Later i decided to look theough the cpanel logs and found a particular user (this reseller) had logged in and used the file manager of each cpanel to upload new index files. Turns out ..the old host had a point to make. None of the passwords had been changed when we did the copy account from the other hosts's servers. This is EXACTALLY the kind of thing that can happen and go almost undectected. So many of us and so used to seeing php scripts and opens source phpBB and other "exploits" scare the crap out of us we seem to forget about the good old username and password method of access is not so hard to do when everyone is looking in the wrong place.

having said this ..it's certainly possible there is a gaping hole someone has found ..but so far ..many of these can be explaind with the old fashioned way of crooks using the damn keys they bought from another dishonest creeps with no morals.

[rs-freddo]

Most users don't use TLS or SSL to secure their FTP password. So it's no wonder that their accounts are compromised. We had one account compromised because the user visited Sth America and used a public internet cafe to upload. Their account was used to send spam for a short while.

[ChadE]

@mta: I meant that, since people were theorizing that the attacks were occuring through SSH (either bypassing SSH's authentication OR through an exploit, whether it's on SSH or another service), I modified my firewall to only serve my SSH, WHM and other ports that only the server admins need (WHM for general purposes) to my, and trusted IPs.

Considering what rpmws has said, it could be possible - but the people compromised would need to tell us if they recently changed their login information (passwords, etc) BEFORE the attack. However, let's say it is one or two, or three datacenters that this is revolving around, then the servers could have been compromised in advance with no actions taken (silent trojan timebomb), with the attacks occuring simultaneously at a given date. Thus, irregardless of whether or not the person changed their credentials, they're already infected.

I have servers at two DCs right now, and one of them sent an email to their clients informing them that attackers had gained access to their support ticket database and ran off with thousands of passwords. It's feasible the attackers didn't stop at one DC, particularly if it was a software exploit and the other targets used the same software. I changed my passwords quickly and sure enough, a few weeks later, a bot was trying to access my servers with the old login information (I had changed my SU username as well).

EDIT: I read what freddo wrote - yeah, that's true too. People quickly login via the non-encrypted methods and do so from public computers. Even if it's not related to this, God knows how many corporations or people have compromised computers (or shared computers), or disgruntled employees that record and run off with this information to wreak havoc later.

[rpmws]

something else to consider ..let's say badguy1 get's his hand on the "1001 root passwords" list. He decides to write a script that runs from any of the proxy boxes or any of the boxes he has access to. let's say that script logs into each SSH as root and then grabs the password files from the server ..then logs out to never be seen again ..then he hits the rest of the servers on the lists. His work is done. He can pack up and move his operation.

next he cracks the passwords or at least some of them locally and already has the usernames and the primary IPs of each box. Now instead of "hacking" anything ..he can run bots that simply FTP and deface servers. No hacking required. and his "bot" can run from other servers he has passwords to all the user accounts on. This cycle can continue on and on ..box to box ..account to account. Changing root passwords and all the reseller and separate account passwords has fixed servers from being hit over and over again. If it is an exploit on the server to server level then that wouldn't matter. They should get right back in.

Of course there are some that have been "rooted" and those we may not want to put in the same catagory.

[DigitalN]

let's say that script logs into each SSH as root and then grabs the password files from the server ..then logs out to never be seen again ..then he hits the rest of the servers on the lists. His work is done. He can pack up and move his operation.

You missed the bit out where they login as root and install a rootkit for later usage as well
I can't see a hacker spending all the time to crack users passwords from hundreds of boxes, but I guess they could if they had enough time and processors - more likely to leave a way back in than that imho.

[rpmws]

You missed the bit out where they login as root and install a rootkit for later usage as well
I can't see a hacker spending all the time to crack users passwords from hundreds of boxes, but I guess they could if they had enough time and processors - more likely to leave a way back in than that imho.

well that's the other part ..the rootkit and I understand that. HOWEVER if you have a ton of account passwords, that's one thing that most hosts are NOT going to change wholesale fassion on all their resellers and end users and even if the host did a massive PW change ..many users will just change them back. So after a OS reload and all that normal "cleaning" admins do , without changing ALL the passwords ..here we go again. This whole thing has a been a bunch of different people using login data in different ways and linger on and on and haunt us for a long time. We may never see this go away.

[DigitalN]

Agreed, maybe it's time to start setting time limits (30 days for eg) on password expiries etc.

I've also seen ex-hosts of resellers attack our servers too, I always ask clients to change at least their reseller account password as a result.

[ChadE]

Continuing from the IFRAME/Javascript hacks post -

1. Some of the servers rooted have been reimaged, passwords changed, and rooted again

2. I believe some of these only had one or two websites loaded at the time they were rooted (if any at all), with new passwords

3. There's a mass of FTP and SSH logins with no failures..but in rpm's experiences, he changed passwords and the logins started failing. However, if they grabbed the passwords from the passwd file and logged in normally, this would be the reason. Presumably the script doesn't try to re-root the server upon FTP failure, at least not right away. Let's assume if an issue exists, it only applies to root logins.

4. Is it cPanel-limited, or affecting other panels? Is it RedHat-based distros only, or including Gentoo, Debian, etc?

Let's say it HAS affected OTHER control panels but NOT other distros, then I say it might be a safe assumption that it is a kernel issue. If it has affected every distro, every panel, every unique piece of software - then there's a serious issue. Complete and catastrophic flaw in Linux? In the principles behind web services applications? I sincerely hope we can root out the cause, and preferably, it's something simple, minor and easy to fix. Either way, I hope it's identified before it rips the virtual floor of the internet from under us.

[claudio]

Continuing from the IFRAME/Javascript hacks post -

1. Some of the servers rooted have been reimaged, passwords changed, and rooted again

2. I believe some of these only had one or two websites loaded at the time they were rooted (if any at all), with new passwords

3. There's a mass of FTP and SSH logins with no failures..but in rpm's experiences, he changed passwords and the logins started failing. However, if they grabbed the passwords from the passwd file and logged in normally, this would be the reason. Presumably the script doesn't try to re-root the server upon FTP failure, at least not right away. Let's assume if an issue exists, it only applies to root logins.

4. Is it cPanel-limited, or affecting other panels? Is it RedHat-based distros only, or including Gentoo, Debian, etc?

Let's say it HAS affected OTHER control panels but NOT other distros, then I say it might be a safe assumption that it is a kernel issue. If it has affected every distro, every panel, every unique piece of software - then there's a serious issue. Complete and catastrophic flaw in Linux? In the principles behind web services applications? I sincerely hope we can root out the cause, and preferably, it's something simple, minor and easy to fix. Either way, I hope it's identified before it rips the virtual floor of the internet from under us.

it all makes sense, sounds like kernel or cpanel vulnerability more than trojan or sniffer case....

[cPanelTodd]

To clear up most of these questions that keep reappearing over and over again...

This is not related directly to cPanel. Many other control panels have been affected and this is in no way related to a specific control panel.

It has only been reported on Redhat based systems. (RHEL, CentOS, Fedora) We have yet to see any other distro affected.

It affects both 2.4 and 2.6 kernels.

Our post at http://www.cpanel.net/security/notes...s_toolkit.html explains the details of the rootkit and whats going on on the server. It's still being researched to gather more information. However, there appears to be a couple different points of entry being used including password compromises, but it doesn't seem to be related to a single widespread exploit or vulnerability. As I said, it's still being investigated and we'll update everyone as we gather more information. The best thing to do is to ensure you have a secure system in all aspects and a solid set of password and security policies to limit the affect of this.

[gregk5]

I wish to relate my experience with this style of "exploit" as I think it contradicts some of the assumptions that are being made. I'm particularly concerned about how entry was made to my system and what variations of the rootkit are known about.

My system runs Fedora 8 with SSH, Apache, PHP, sendmail, dovecot IMAP and a single user that can connect via ftp (although it appears to have never been used).

When I first read about the rootkit in Slashdot I did the numeric directory create test and had no issue creating the directory. I then ran the tcpdump command and pointed my browser at my home page - I got output from tcpdump suggesting that I had the exploit. I tried from other IP addresses, both within my network and from outside my network and was not able to get another instance of tcpdump output. I searched my entire file system for the offending javascipt file and did not find it (which is what I expected).

I then reloaded my system and booted from a live CD. There were no files in /sbin or /bin with trailing random characters. There were also no zero length files.

I reloaded the system back to the normal image and setup a permanent tcpdump with grep capture. Over the next few days there were no further instances of the random javascript filename being served.

I also checked for an offending Apache module which was mentioned in some of the forum threads, but it was not found.

For years I have only ever allowed keyed access to ssh and only to one non-root account. So there has never been any ssh password access to my system. Also, I'm the only one that administers the system so I'm the only one who knows any passwords to privileged accounts. No users have shell access to the system. As far as the logs indicate, there was never any access to ftp except for the initial test when I set the single ftp user's account up.

Details of the exploit, or exploits, have not been made available to the level that I can definitely determine whether I have suffered a known exploit. It is also not possible in my situation that root was compromised by someone knowing the password. It has not been disclosed, to my knowledge, how the exploit causes the numeric directory creation to fail. Maybe a system call hook, but why? What's the point of denying numeric directory creation, especially if it turns out to be a symptom of an exploit.

My guess on the sequence of events is: I had a variation of the exploit running in memory, but I had at some stage done a system update which removed it from any binaries; after reloading my system the memory copy of the exploit was gone and since the binaries had been replaced with good ones, there was nothing in place to reinstall the exploit. Still doesn't explain why I don't have any oddly named files in /sbin and /bin.

I fear there is an exploit in one of the services that I mentioned at the start of this message; nothing to do with system access via passwords.

[dragon2611]

What's the point of denying numeric directory creation, especially if it turns out to be a symptom of an exploit.

at a guess to Stop you building a patched kernel ?

[sparek-3]

My system runs Fedora 8 with SSH, Apache, PHP, sendmail, dovecot IMAP and a single user that can connect via ftp (although it appears to have never been used).

What version of these softwares were you running? This might be helpful. Especially if there is someone else that was also running Fedora 8 and they were running the same version of one of these pieces of software. For example if two people were using OpenSSH 4.7 but different versions of Apache and both were exploited, then this would seem to point the finger at OpenSSH 4.7 (though not necessarily).

cPanel systems use exim instead of sendmail and Courier instead of Dovecot so I would doubt that either of those would be the entry point.

[sitekeeper]

This may be "way out there" but has anyone looked into what ssh program people are using to login to their servers? Since many seem to use Putty and it is opensource I could see where someone with knowledge could think up something like this.

Bob

[ChadE]

Indeed...but the rootkit seems to have been spotted on RedHat-based servers ONLY. Not Gentoo, not Debian, Slackware, etc. Although, it's possible they've been affected but not as widely. I'll re-read the Webhostingtalk and cPanel page on it here in a minute. It's NOT cPanel because Plesk and Ensim servers have been affected.

If it turns out it really is a RedHat only problem, then something in the kernel is causing the problem...

CentOS 4/5, various Fedora Cores, and RHEL have been affected - what's a common software and or version in all of them?

If it really is only RedHat, then I say it's not Apache, PHP, BIND, OpenSSH, etc.