Server compromised, concerns...

[4u123]

Had a server compromised earlier today. I'm just in the process of copying the cpanel accounts to a different server as we speak.

What concerns me is that all our servers are configured identically, always kept up to date and looked after properly. I've been running my hosting business for 7 years now and this is the first time we've ever had a root exploit.

An account with a dodgy Joomla module was compromised - a backdoor script copied to /tmp. This was picked up straight away, deleted and the account suspended. Scripts in /tmp cant be executed so they couldnt have run it. So I thought I'd resolved it. Nope. Later on we started getting support tickets from customers - their index pages were being changed - the hacker leaving his little message on any index.* page on the whole server. This can only have been done via a root exploit of some kind.

rkhunter found parts of two rootkits as a result of this incident - but everything else was fine. I have no idea how this happened and I'm worried for our other servers because, as I said, they are all the same.

I admit - I've been a bit slack lately in keping the kernels up to date and have now updated all servers - but if this exploit is something else I'm probably screwed.

Can anyone recommend a good server security service that doesnt cost too much? I'd like to have all our servers checked by an expert.

[serversignature]

Are you using suPHP Apache module (mod_suphp) on your servers ?

You can ask for help in the Server Management and Server Repair Forum

Thanks,

[d_t]

Try Chirpy:
ConfigServer Server Services

[4u123]

Hi guys, yes I'm using suphp on all our servers. I identified the compromised account very quickly and suspended it as mentioned above. The hacker somehow managed to change all index pages in any directory on the server by using some kind of root exploit. Thats what I'm worried about - not the fact that a customers account was used. Normally, if a customers PHP script is hacked, the hacker can only change files that the account's userid is allowed to change - so its not a major issue. But today they managed to access any part of the server they wanted. Any scripts that are copied into /tmp cant be executed so I have no idea how they managed to install a rootkit.

I did actually speak to configserver but they didnt seem to understand my request. The person I spoke to thought I was asking them to identify which customers script was compromised and like you, asked if I was using suphp but I was asking them to take a look at the server and find out how the hacker managed to get higher access. Unfortunately I think I spoke to a junior member of staff who didnt understand my situation. I was told that they didnt provide that kind of service.

[Infopro]

You saw this one then I take it?
ConfigServer Anti-Spammer/Exploit Service

[4u123]

You saw this one then I take it?
ConfigServer Anti-Spammer/Exploit Service

Yes, thats why I contacted them but it seems the service they provide is to copy the accounts onto another disk - which I can do myself. I've already restored all the accounts and moved them to a new server.

The idea behind this service (and is a requirement of it) is that you or your datacenter installs a new OS disk into the server with the OS and cPanel already installed. The old failed OS disk is mounted as a slave drive. We can then come in and migrate all the data from the old OS disk to the new OS disk and thus effectively recover your server from when it was last shutdown and without having to resort to out of date backups.

What I was asking them to do is to look at the compromised server and try and tell me (with their expert server security knowledge) how they were able to gain root level access. But I was told that is not a service they provide. So I'm looking for a security expert to examine the server and tell me where the vulnerability is.

[serversignature]

try to get this info, run this commands as root

/usr/local/cpanel/cpanel -V
cat /etc/redhat-release
uname -r

Mod_security -- Do you have it installed ?
mod_security works for apache only, does not work for whm.

[4u123]

cpanel version is 11.24.5-RELEASE_38506

OS is centos 4.8

Yes mod_security was installed although the rules were a couple of months old.

Kernel was 2.6.9-78.0.22 which was about 4 months out of date but doesnt contain any known security issues.

[ramprage]

In the underground there are many zero day exploits and unpublished exploits. Finding exactly how they gained access can be extremely difficult since most hackers erase all their tracks or try to.

It sounds like they used a script vulnerability to gain shell access then tried hitting your server with rookits and 0 day exploits. My suggestion is to keep your system updated and use layers of security.

Firewall
suphp
mod_security
upload guardian
clamav

Also having directories like /tmp locked and restricted access to binaries like wget can help a ton.

Steve

[sparek-3]

The 2.6.9-78.0.22 kernel was released back in May. I know in August there was a pretty nasty kernel vulnerability that was discovered. I'm not sure if there are any exploits for this vulnerability floating around, but chances are it was a kernel vulnerability or some other vulnerability on the server that allowed the user to gain root on the server.

I would recommend having the server reimaged and restore the accounts from backups.

[Spiral]

What concerns me is that all our servers are configured identically, always kept up to date and looked after properly. I've been running my hosting business for 7 years now and this is the first time we've ever had a root exploit.

The security measures you find in all the tutorials, security threads, books, and articles are certainly a good start in the right direction but in reality don't even cover a small fraction of what all you really need to do with securing your servers.

In fact, the vast majority of "experts" out there are totally and completely unaware of the real potential threats against your server which unfortunately continue to evolve and expand everyday as more exploits are discovered and more creative ways are devised to attack the servers. It's not really surprising that you got a hacking situation. The real question is what you are going to do about it and are you going to take action to prevent the same thing from reoccurring again?

Can anyone recommend a good server security service that doesn't cost too much? I'd like to have all our servers checked by an expert.

Done! Contact me and I WILL help you more than you could possibly imagine!

There is several new exploits in the wild the past week that has kept me psychotically busy the past few days with clients all over the world as I'm right at the top of the call list for most data centers and many hosts particularly where it comes to security but I'll squeeze you into my schedule if you would like me to take a look.

Regarding your side mention of the I-Frame/Index hacks, I can give you the ability to detect those and take action on those kind of attacks instantly even while the hacker is still connected; Plus, got a lot of other security related technologies that could be of great benefit for you.

Oh and to the following comment ....

What I was asking them to do is to look at the compromised server and try and tell me (with their expert server security knowledge) how they were able to gain root level access. But I was told that is not a service they provide. So I'm looking for a security expert to examine the server and tell me where the vulnerability is.

I am not sure exactly if you are talking about your data center or some security service you contacted but what you just described is precisely my specific area of expertise and is exactly what I'm most known for!

Anyway, like I said -- got you covered. You just need to contact me.

Private Message is fine though my schedule doesn't always allow me to watch forums closely.

[SigmaWeb]

You can also check JoneSolutions.Com - Services, Server Management, Security and Stability we are more than 4 years with Jones with more than 35 servers and never had any root exploit.

[BianchiDude]

I would recommend having the server reimaged and restore the accounts from backups.

That is one way to look at it, however a more appropriate way would be to follow these steps:
#1. Do not install, reinstall or delete anything from that drive

#2. List the current open files, lsof, current processes, ps aux, current open ports netstat -anpe

#3. Pull the power cord out from the box (if possible or have the data center do it)

#4. Notify all your users that there has been a compromise, notify your provider if necessary.

#5. Make a forensic image of the drive (or have the data center do it) using the unix dd command, set the original drive in a safe place and ensure you maintain a chain of custody on it.

#6. Go through the logs you have from Chkrootkit / Rootkit Hunter / Aide / Samhain / Snort / Integrit / Osiris or tripwire, if the logs are on the drive itself look at them on the image your made.

#7. Review the image of the compromised drive, was the OS/kernel current? Were all the packages up to date? What was in the world writeable directories like /tmp, /var/tmp, /dev/shm, what services were running on the drive, what was the version of php, perl, etc.

#8. Look at the logs files and logrotated files such as wtmp, secure, messages, firewall logs setuid files, user shell histories, yum logs.

#9. Document any hints, hunches, or gut feeling you have on the the box was hacked.

#10. Only after your investigation and developing a plan to keep the box more secure should you install the OS on the new drive (the compromised drive should still be in a safe place) and only the user home data should be restore, and chowned to the user’s username, prior to the server being live on the internet again.

#11. Contact other parties, such as law enforcement if appropriate.

[BianchiDude]

Can anyone recommend a good server security service that doesnt cost too much? I'd like to have all our servers checked by an expert.

I would help you but I am fully booked right now and currently not taking any new clients.

[jpetersen]

Can anyone recommend a good server security service that doesnt cost too much? I'd like to have all our servers checked by an expert.

Check out ScottMC from admingeekz.com or StevenC from rack911.com. Avoid Spiral ("top of the call list for most datacenters" = pure bs). I've chatted with Scott and Steven over the years which is why I'd recommend them. They've dealt with these situations and have the requisite experience.

Understand, however, that it's not always possible to determine how a box gets popped, so don't expect a completely detailed explanation. Not keeping your kernel updated? Expect to find yourself in this situation again and again.

Any scripts that are copied into /tmp cant be executed

Incorrect. You're using suphp anyway, so if someone is able to gain access through a vulnerable web app, they don't even need to use /tmp, as they can just write to the user's ~.

Good luck. If you figure out how the box was owned, feel free to update this thread.