Server hacked.. Maybe a new 0day

[zigzam]

One of my highly secured servers was hacked today. All software was up to date and the server was running:

- The latest centos 5 kernel
- suPHP on PHP 5
- Apache 2.2
- Mod_security with advanced ruleset
- CSF firewall
- Cpanel latest release version
- Mysql Root password and root password very secure and impossible to guess

This is what I know so far. The hacker created an account out of the sky named: "localhost" and escalated the permissions to superuser (UID 0)

New account [localhost] has been created with uid:[0] gid:[0] login:[/home/localhost] shell:[/bin/bash]

Account [localhost] password has changed

Account [root] password has changed


Once on the server he ran:

 root     20745  0.0  0.0  10160  2996 ?        Ss   10:54   0:00  \_ sshd: [email protected]/0,pts/2,pts/3,pts/4
 root     20800  0.0  0.0   5396  1392 pts/0    Ss   10:54   0:00      \_ -bash
 root     23079  4.1  0.1  12056  7112 pts/0    D+   12:26   0:10      |   \_ perl mass.pl -d /home -f index. -n /tmp/index.html
 root      6630  0.0  0.0   6640  1708 ?        Ss   11:16   0:00      \_ /usr/libexec/openssh/sftp-server
 root     20460  0.0  0.0   5392  1396 pts/2    Ss   11:37   0:00      \_ -bash
 root     23842  0.4  0.0   4592   472 pts/2    D+   12:27   0:00      |   \_ rm -rf /var/cpanel
 root     32001  0.0  0.0   5400  1396 pts/3    Ss   11:51   0:00      \_ -bash
 root     24116  5.1  0.0   4596   500 pts/3    D+   12:28   0:07      |   \_ rm -rf daily
 root     14510  0.0  0.0   6492  1592 ?        Ss   12:13   0:00      \_ /usr/libexec/openssh/sftp-server
 root     24536  0.0  0.0   5392  1392 pts/4    Ss   12:29   0:00      \_ -bash

Thoughts?

 

[sirotex]

lol

Yep...how can you say you're secure when you do not use ssh keys? honestly..

If you disabled logins via passwords in SSHD and used keys then he/she won't have been able to gain access to SSH..

Ahwell it's a harsh reality but it might be a good idea to start using keys

 

[zigzam]

Yep...how can you say you're secure when you do not use ssh keys? honestly..

If you disabled logins via passwords in SSHD and used keys then he/she won't have been able to gain access to SSH..

Ahwell it's a harsh reality but it might be a good idea to start using keys

SSH keys or not he still would have done what he wanted to do. The question is how was he able to gain superuser permissions.

 

[sirotex]

Without SSH access the account is useless. There are a number of ways, you should check your websites haven't been infected with a shell script like c99. c99: http://shellci.biz/c99.txt.

 

[zigzam]

With the server running suPHP those scripts should be harmless to the server as long as everything is up to date?

I have disabled root login all together via the public network.

 

[brianoz]

Without SSH access the account is useless.

What about WHM? Doesn't that allow root logins? Not sure if they function as root if not called root though?

 

[zigzam]

Or couldnt the hacker create a ssh key from whm? Then access SSH?

 

[brianoz]

I really don't want to distract from the original poster's question, how did they actually gain access in the first place?

(He says, running "useradd -m localhost; passwd -l localhost" on all his boxes ...)

 

[sirotex]

So basically they had root or sudo axx to the adduser function to add there account anyways..

 

[brianoz]

Yes, they had root somehow - although the unusual choice of username may not be a coincidence. Possibly they thought 'localhost' would go under the radar, so chose it for that reason; possibly the hack does that.

 

[dwykofka]

You still haven't figured out where they came in?

I suggest you track down their ip and start greping domlogs to find the original intrusion point.

 

[Solokron]

I have to say, Chirpy's CSF with a modsecurity trigger of five is an excellent way to deter many of these attackers. Stick SSH on a very high port and fend off port scans and invalid logins with CSF as well.

I am interested in how he got in.

 

[sirotex]

Hense why using SSH keys would fix this as they would not be able to try and bruteforce the password.. IF thats how they got in, you really should think about your passwords.

 

[Solokron]

As I mentioned, CSF blocks brute forces rather quickly.

Hense why using SSH keys would fix this as they would not be able to try and bruteforce the password.. IF thats how they got in, you really should think about your passwords.

 

[zigzam]

I have CSF to block IPs after 5 failed login attempts and the server ssh port is on a high port number.

It was not a password guess, it was a hack through some software on the server that was the latest version.

 

[weetabix]

it's equally likely you are compromised on a personal level. look for trojans and keyloggers.

 

[zigzam]

it's equally likely you are compromised on a personal level. look for trojans and keyloggers.

My PC is not compromised. I manage over 100 servers and nothing else was compromised.

As stated already this was not a password login. He created his own account then changed the root password then logged in.

 

[LiNUxG0d]

With the server running suPHP those scripts should be harmless to the server as long as everything is up to date?

I have disabled root login all together via the public network.

Hey there,

suPHP is not a failsafe at all, it's a container. The great thing about suPHP is it runs your scripts as the user who's directory you're in. The good thing about suPHP is it doesn't allow 7's or 6's across the board, doesn't allow nobody write-access.

The bad thing about suPHP is it doesn't STOP hacks from happening, at all. I'm not sure why you'd believe that either. It just limits the hacks infection to the user account. This still doesn't mean the hacker can't log in via shell or elevate privileges. As soon as a hacker has access to a c99 or r57, it's open territory to get in and hook in.

You have to understand here, in layering terms, you need to protect them all.

1. SSH Keys protect against password sniffing.
2. mod_sec rules are important as they close off openings in the scripts themselves.
3. suPHP is important because if mod_sec fails to catch something, suPHP will tie the hacker to a certain namespace.
4. SELinux or GRsec are important because they'll randomize the stack, so exploits for overflows don't go over so easily.

Another important point to think of is the host you use. I've known hackers that have rooted routers that are sniffed and tapped. This isn't paranoia, big brother exists. You don't believe it until you meet him.

If your other boxes aren't rooted, then consider yourself lucky. You may want to start pushing the buck a bit more and installing tripwire and/or AIDE for forensics, combined with a good port knocker to harden SSHd one step further.

Security starts with the security agent. Your practices are not fail-safe, they are fallible, ultimately. If it was a CentOS hole, my network would be crippled. We're all getting the same upstream packages. My net isn't though, so that leads me to believe you're running:

a) Something that's rootable;
b) Some non-conventional software;
c) Your bastion/home-base is tapped.

A good hacker... and they do exist, will NOT leave obvious traces. If they do, they are kiddies. That's for sure. You may be dealing with the former... though I still believe you're dealing with the latter.

Start a remote syslogd as well, for fun, to see what's going on. If that guy gained elevation, then his logs are gone, and your best bet is domlogs to see if they C99'd you. Scan for .txt files, they usually are .txt, generally.

Regards,