Root logins from single unknown IP - what next steps, suspicious changes

Re: Root logins from single unknown IP - what next steps, suspicious change

The only part of your message that I really need to see is this:

That statement alone tells me that your computer (or more specifically any that have ever logged in as root before) is very likely infected with a trojan virus designed specifically to give hackers your hosting login information.

This particular type of hacking has been widely in use for roughly three years or so now and is quite difficult to defend because those behind it already know the correct login in advance and just simply login --- no brute force attempts needed!

Changing your password often does no good either because the moment you change your password, the hackers are updated with the new password information which is why it is imperative you do complete full virus and trojan scans of all computers you know of which may legitimately log into your server as the root administrator. Once you are absolutely certain that the server has been cleaned, then go ahead and change your root password again.

Meanwhile, if you have the hacker's current IP from the /usr/local/cpanel/logs/access_log file or other logs in /var/log then I would go ahead and ban that IP either through IP Tables or CSF or if you must /etc/hosts.deny. This will not stop the hacker as they can just simply change their apparent IP but it might slow them down just enough to buy yourself valuable time to resolve the current situation. Ideally, I'd use a different computer that has never before been used to change the password if that is a possible option to you.

After you have your home computer(s) cleared, passwords changed, and known IPs for the hacker banned, then your next task will be going through your whole entire server looking for items that have been changed (activity logs MAY help) and you need to also check all the web sites that you host. With this particular type of attack, it is not uncommon to replace index files on your sites with ones that link to trojan installer locations to infect more home users AND / OR drop in additional scripts to let themselves back in as a backdoor should you cut off their front means of gaining access.

It's kind of a nasty and painful situation all around but if you are methodical in getting everything cleaned up then you should be okay and get over this.

There is no 100% foolproof way of totally protecting yourself from this kind of situation but there is still many different things you can do to protect your server so that this does not happen ever again. Once you get cleaned up a bit, if you want a hand at re-security hardening your server, I would be happy to give you a hand to help make sure you don't miss anything along the way that might leave you vulnerable. Just contact me later on if you need the help.

Right now your priorities would be:

1. Aggressively scan your computers at home/office

2. It is very rare for a hacker to login to root and do nothing other than that so you will
need to find out exactly what it is that they did to you starting with a deep review of the
web sites you host followed by the rest of the server applications and operating system.

Good luck to you. If it makes you feel any better, I've personally seen the reports of hundreds
of different servers attacked this way over the years so you aren't alone in any of this.

 

Re: Root logins from single unknown IP - what next steps, suspicious change

Obviously one urgent thing to do is to contact your team members and find out whether anyone did log in from home. If not, you might want to consider restricting root login via ssh to an IP range or two, then changing the password, as a temporary measure while you work out what happened.

As Netmantis says above, the password could well have been stolen by a trojan / virus keylogger running on your PC. Do a virus scan on any PCs that may have logged in as root, with at least two different scanners.

The real test is whether it logs in again after you've changed the root password. If that happens, look for another PC that could have leaked the password. Silly question I know, but is the password also used in any unsecured protocols such as POP or FTP?

Re your md5sum check - it does look like a lot of stuff has changed. Have you used rpm to verify your binaries? You may also want to compare to another system with same OS version. You mentioned the LF_INTEGRITY setting -- it does look like files have changed. You may want to compare both ways (compare the binaries on your system, then compare them on the other system) as it's possible (probably unlikely) they could have installed a kernel module.

With the number of brute force root attempts these days you really do need to change the SSH port to something other than 22, if you haven't already. If nothing else, this reduces "log noise" so makes future attempts a lot more obvious. The new port number should be high enough to reduce the likelihood that low range port scans will find it, eg: 30122 etc etc (under 65456 from memory).

 

Re: Root logins from single unknown IP - what next steps, suspicious change

On the topic of checksums, I really wouldn't worry too much about the list of files that you posted. You can take a passing look at them to see if they have been altered but I suspect those are just coincidental updates on those files. You can re-download those files from author sources and check for changes.

The type of attack you described has a standard modus operandi they rarely deviate from and in that, I would be more concerned about the content of any websites that you host. Those are usually the first target and it's highly likely that digging through those you'll fine places where pages have been slightly altered or additional scripts have been added to allow for code injection or other access possibilities. They often make the added scripts look like graphics images to make it more difficult for you to find the added code which they simply call as an include from your other programs.

You would be correct. It is never a good idea to transmit a password in any manner where it could be intercepted.

(Generally speaking, it is safer though to send a password by email than it is posting it in a live chat network so if you are faced with one or the other, I'd recommend the password be sent by email but then changed immediately upon receipt)

As an important rule, I would recommend that you consider all passwords emailed to you to be "temporary" and you should change your password first thing you do immediately upon connecting.

 

Re: Root logins from single unknown IP - what next steps, suspicious change

Actually, it's pretty much standard practice - they assume you will be changing it immediately!

To be honest, that's the most likely scenario! Let's hope so!

It's fairly unlikely that the integrity check would be disabled by a hacker but do let us know if it was!