Server hacked.. Maybe a new 0day

zigzam

Well-Known Member
May 9, 2005
206
0
166
One of my highly secured servers was hacked today. All software was up to date and the server was running:

- The latest centos 5 kernel
- suPHP on PHP 5
- Apache 2.2
- Mod_security with advanced ruleset
- CSF firewall
- Cpanel latest release version
- Mysql Root password and root password very secure and impossible to guess

This is what I know so far. The hacker created an account out of the sky named: "localhost" and escalated the permissions to superuser (UID 0)

New account [localhost] has been created with uid:[0] gid:[0] login:[/home/localhost] shell:[/bin/bash]

Account [localhost] password has changed

Account [root] password has changed


Once on the server he ran:

PHP:
 root     20745  0.0  0.0  10160  2996 ?        Ss   10:54   0:00  \_ sshd: [email protected]/0,pts/2,pts/3,pts/4
 root     20800  0.0  0.0   5396  1392 pts/0    Ss   10:54   0:00      \_ -bash
 root     23079  4.1  0.1  12056  7112 pts/0    D+   12:26   0:10      |   \_ perl mass.pl -d /home -f index. -n /tmp/index.html
 root      6630  0.0  0.0   6640  1708 ?        Ss   11:16   0:00      \_ /usr/libexec/openssh/sftp-server
 root     20460  0.0  0.0   5392  1396 pts/2    Ss   11:37   0:00      \_ -bash
 root     23842  0.4  0.0   4592   472 pts/2    D+   12:27   0:00      |   \_ rm -rf /var/cpanel
 root     32001  0.0  0.0   5400  1396 pts/3    Ss   11:51   0:00      \_ -bash
 root     24116  5.1  0.0   4596   500 pts/3    D+   12:28   0:07      |   \_ rm -rf daily
 root     14510  0.0  0.0   6492  1592 ?        Ss   12:13   0:00      \_ /usr/libexec/openssh/sftp-server
 root     24536  0.0  0.0   5392  1392 pts/4    Ss   12:29   0:00      \_ -bash

Thoughts?
 

sirotex

Well-Known Member
Jul 10, 2008
121
0
66
lol

Yep...how can you say you're secure when you do not use ssh keys? honestly..

If you disabled logins via passwords in SSHD and used keys then he/she won't have been able to gain access to SSH..

Ahwell it's a harsh reality but it might be a good idea to start using keys :)
 

zigzam

Well-Known Member
May 9, 2005
206
0
166
Yep...how can you say you're secure when you do not use ssh keys? honestly..

If you disabled logins via passwords in SSHD and used keys then he/she won't have been able to gain access to SSH..

Ahwell it's a harsh reality but it might be a good idea to start using keys :)
SSH keys or not he still would have done what he wanted to do. The question is how was he able to gain superuser permissions.
 

zigzam

Well-Known Member
May 9, 2005
206
0
166
With the server running suPHP those scripts should be harmless to the server as long as everything is up to date?

I have disabled root login all together via the public network.
 

sirotex

Well-Known Member
Jul 10, 2008
121
0
66
So basically they had root or sudo axx to the adduser function to add there account anyways..
 

Solokron

Well-Known Member
Aug 8, 2003
852
2
168
Seattle
cPanel Access Level
DataCenter Provider
I have to say, Chirpy's CSF with a modsecurity trigger of five is an excellent way to deter many of these attackers. Stick SSH on a very high port and fend off port scans and invalid logins with CSF as well.

I am interested in how he got in.
 

sirotex

Well-Known Member
Jul 10, 2008
121
0
66
Hense why using SSH keys would fix this as they would not be able to try and bruteforce the password.. IF thats how they got in, you really should think about your passwords.
 

Solokron

Well-Known Member
Aug 8, 2003
852
2
168
Seattle
cPanel Access Level
DataCenter Provider
As I mentioned, CSF blocks brute forces rather quickly.

Hense why using SSH keys would fix this as they would not be able to try and bruteforce the password.. IF thats how they got in, you really should think about your passwords.
 

zigzam

Well-Known Member
May 9, 2005
206
0
166
I have CSF to block IPs after 5 failed login attempts and the server ssh port is on a high port number.

It was not a password guess, it was a hack through some software on the server that was the latest version.
 

weetabix

Well-Known Member
Oct 26, 2006
68
7
158
it's equally likely you are compromised on a personal level. look for trojans and keyloggers.
 

zigzam

Well-Known Member
May 9, 2005
206
0
166
it's equally likely you are compromised on a personal level. look for trojans and keyloggers.
My PC is not compromised. I manage over 100 servers and nothing else was compromised.

As stated already this was not a password login. He created his own account then changed the root password then logged in.
 

LiNUxG0d

Well-Known Member
Jun 25, 2003
206
1
168
Gatineau, Quebec, Canada
With the server running suPHP those scripts should be harmless to the server as long as everything is up to date?

I have disabled root login all together via the public network.
Hey there,

suPHP is not a failsafe at all, it's a container. The great thing about suPHP is it runs your scripts as the user who's directory you're in. The good thing about suPHP is it doesn't allow 7's or 6's across the board, doesn't allow nobody write-access.

The bad thing about suPHP is it doesn't STOP hacks from happening, at all. I'm not sure why you'd believe that either. It just limits the hacks infection to the user account. This still doesn't mean the hacker can't log in via shell or elevate privileges. As soon as a hacker has access to a c99 or r57, it's open territory to get in and hook in.

You have to understand here, in layering terms, you need to protect them all.

1. SSH Keys protect against password sniffing.
2. mod_sec rules are important as they close off openings in the scripts themselves.
3. suPHP is important because if mod_sec fails to catch something, suPHP will tie the hacker to a certain namespace.
4. SELinux or GRsec are important because they'll randomize the stack, so exploits for overflows don't go over so easily.

Another important point to think of is the host you use. I've known hackers that have rooted routers that are sniffed and tapped. This isn't paranoia, big brother exists. You don't believe it until you meet him.

If your other boxes aren't rooted, then consider yourself lucky. You may want to start pushing the buck a bit more and installing tripwire and/or AIDE for forensics, combined with a good port knocker to harden SSHd one step further.

Security starts with the security agent. Your practices are not fail-safe, they are fallible, ultimately. If it was a CentOS hole, my network would be crippled. ;) We're all getting the same upstream packages. My net isn't though, so that leads me to believe you're running:

a) Something that's rootable;
b) Some non-conventional software;
c) Your bastion/home-base is tapped.

A good hacker... and they do exist, will NOT leave obvious traces. If they do, they are kiddies. That's for sure. You may be dealing with the former... though I still believe you're dealing with the latter.

Start a remote syslogd as well, for fun, to see what's going on. If that guy gained elevation, then his logs are gone, and your best bet is domlogs to see if they C99'd you. Scan for .txt files, they usually are .txt, generally. :)

Regards,