Root logins from single unknown IP - what next steps, suspicious changes

[actived]

Hi,

We have had root logins from an unknown IP a few times in the last few hours. Nothing seems to have been changed or damaged. This one was not a brute force attempt. So it could be someone in the organisation, legitimately logging in, but finding out who takes time in a company - and some damage can be done by then.

The damage could be insidious, hidden - installation of some rogue software / script / etc.

1. First thing I did was to change the root password.

2. Next, I plan to install and run rkhunter (which, yes, wasn't installed, so my bad). I have read that if root is compromised then a worst case scenario is that system binaries and even an installed rkhunter can be compromised/modified. So luckily in that sense, rkhunter is now reliable

3. I have CSF+LFD running which is how I found out in the first place.
But the interesting thing is that in CSF, there is the option LF_INTEGRITY which I *definitely* had set to enabled, and it now shows the red "Warning" message.

I had to put CSF back into "Testing" mode for a while - does this disable LF_INTEGRITY checking automatically?
PT_SKIP_HTTP and PT_ALL_USERS are also not in force ("Warning")

Do these three have to re-enabled every time when setting Testing to 0?

4. Brute force login attempts from rogue IP ranges is a very common daily occurrence with us now. CpHulk seems to discourage those fairly well (30 tries = banned for 15 days)

We have, of course, backed up everything offline.

Any other suggestions?

Thanks in advance,
Dave

 

[actived]

Re: Root logins from single unknown IP - what next steps, suspicious change

Also, here is an email alert from lfd:

Subject: 
lfd on domain.com: System Integrity checking detected a modified system file

Body:
The following list of files have FAILED the md5sum comparison test. This means that the file has been changed in some way. This could be a result of an OS update or application upgrade. If the change is unexpected it should be investigated:

/usr/bin/afs5log: FAILED
/usr/bin/amtu: FAILED
/usr/bin/aulastlog: FAILED
/usr/bin/ausyscall: FAILED
/usr/bin/chage: FAILED
/usr/bin/chfn: FAILED
/usr/bin/chsh: FAILED
/usr/bin/cvs: FAILED
/usr/bin/gpasswd: FAILED
/usr/bin/lchfn: FAILED
/usr/bin/lchsh: FAILED
/usr/bin/newgrp: FAILED
/usr/bin/passwd: FAILED
/usr/bin/php: FAILED
/usr/bin/php5: FAILED
/usr/bin/php5-cgi: FAILED
/usr/bin/php5-cli: FAILED
/usr/bin/php-cgi: FAILED
/usr/bin/php-cli: FAILED
/usr/bin/php-config: FAILED
/usr/bin/pkcs11_eventmgr: FAILED
/usr/bin/pkcs11_inspect: FAILED
/usr/bin/pkcs11_setup: FAILED
/usr/bin/pklogin_finder: FAILED
/usr/bin/scanpci: FAILED
/usr/bin/screen: FAILED
/usr/bin/sg: FAILED
/usr/bin/X: FAILED
/usr/bin/Xnest: FAILED
/usr/bin/Xorg: FAILED
/usr/bin/Xvfb: FAILED
/usr/sbin/adduser: FAILED
/usr/sbin/cc_dump: FAILED
/usr/sbin/cc_test: FAILED
/usr/sbin/dovecotpw: FAILED
/usr/sbin/exim: FAILED
/usr/sbin/exim_dbmbuild: FAILED
/usr/sbin/exim_dumpdb: FAILED
/usr/sbin/exim_fixdb: FAILED
/usr/sbin/exim_lock: FAILED
/usr/sbin/exim_tidydb: FAILED
/usr/sbin/groupadd: FAILED
/usr/sbin/groupdel: FAILED
/usr/sbin/groupmod: FAILED
/usr/sbin/hwclock: FAILED
/usr/sbin/lchage: FAILED
/usr/sbin/lgroupadd: FAILED
/usr/sbin/lgroupdel: FAILED
/usr/sbin/lgroupmod: FAILED
/usr/sbin/lnewusers: FAILED
/usr/sbin/lpasswd: FAILED
/usr/sbin/luseradd: FAILED
/usr/sbin/luserdel: FAILED
/usr/sbin/lusermod: FAILED
/usr/sbin/runq: FAILED
/usr/sbin/saslauthd: FAILED
/usr/sbin/useradd: FAILED
/usr/sbin/userdel: FAILED
/usr/sbin/userhelper: FAILED
/usr/sbin/usermod: FAILED
/bin/login: FAILED
/bin/passwd: FAILED
/sbin/clock: FAILED
/sbin/hwclock: FAILED
/sbin/pam_console_apply: FAILED
/sbin/pam_tally: FAILED
/sbin/pam_tally2: FAILED
/sbin/runuser: FAILED
/usr/local/bin/php: FAILED
/usr/local/bin/php-config: FAILED

Is it confirmed that this is not a false positive of some type?

 

[NetMantis]

Re: Root logins from single unknown IP - what next steps, suspicious change

The only part of your message that I really need to see is this:

We have had root logins from an unknown IP a few times in the last few hours. Nothing seems to have been changed or damaged. This one was not a brute force attempt.

That statement alone tells me that your computer (or more specifically any that have ever logged in as root before) is very likely infected with a trojan virus designed specifically to give hackers your hosting login information.

This particular type of hacking has been widely in use for roughly three years or so now and is quite difficult to defend because those behind it already know the correct login in advance and just simply login --- no brute force attempts needed!

Changing your password often does no good either because the moment you change your password, the hackers are updated with the new password information which is why it is imperative you do complete full virus and trojan scans of all computers you know of which may legitimately log into your server as the root administrator. Once you are absolutely certain that the server has been cleaned, then go ahead and change your root password again.

Meanwhile, if you have the hacker's current IP from the /usr/local/cpanel/logs/access_log file or other logs in /var/log then I would go ahead and ban that IP either through IP Tables or CSF or if you must /etc/hosts.deny. This will not stop the hacker as they can just simply change their apparent IP but it might slow them down just enough to buy yourself valuable time to resolve the current situation. Ideally, I'd use a different computer that has never before been used to change the password if that is a possible option to you.

After you have your home computer(s) cleared, passwords changed, and known IPs for the hacker banned, then your next task will be going through your whole entire server looking for items that have been changed (activity logs MAY help) and you need to also check all the web sites that you host. With this particular type of attack, it is not uncommon to replace index files on your sites with ones that link to trojan installer locations to infect more home users AND / OR drop in additional scripts to let themselves back in as a backdoor should you cut off their front means of gaining access.

It's kind of a nasty and painful situation all around but if you are methodical in getting everything cleaned up then you should be okay and get over this.

There is no 100% foolproof way of totally protecting yourself from this kind of situation but there is still many different things you can do to protect your server so that this does not happen ever again. Once you get cleaned up a bit, if you want a hand at re-security hardening your server, I would be happy to give you a hand to help make sure you don't miss anything along the way that might leave you vulnerable. Just contact me later on if you need the help.

Right now your priorities would be:

1. Aggressively scan your computers at home/office

2. It is very rare for a hacker to login to root and do nothing other than that so you will
need to find out exactly what it is that they did to you starting with a deep review of the
web sites you host followed by the rest of the server applications and operating system.

Good luck to you. If it makes you feel any better, I've personally seen the reports of hundreds
of different servers attacked this way over the years so you aren't alone in any of this.

 

[brianoz]

Re: Root logins from single unknown IP - what next steps, suspicious change

Obviously one urgent thing to do is to contact your team members and find out whether anyone did log in from home. If not, you might want to consider restricting root login via ssh to an IP range or two, then changing the password, as a temporary measure while you work out what happened.

As Netmantis says above, the password could well have been stolen by a trojan / virus keylogger running on your PC. Do a virus scan on any PCs that may have logged in as root, with at least two different scanners.

The real test is whether it logs in again after you've changed the root password. If that happens, look for another PC that could have leaked the password. Silly question I know, but is the password also used in any unsecured protocols such as POP or FTP?

Re your md5sum check - it does look like a lot of stuff has changed. Have you used rpm to verify your binaries? You may also want to compare to another system with same OS version. You mentioned the LF_INTEGRITY setting -- it does look like files have changed. You may want to compare both ways (compare the binaries on your system, then compare them on the other system) as it's possible (probably unlikely) they could have installed a kernel module.

With the number of brute force root attempts these days you really do need to change the SSH port to something other than 22, if you haven't already. If nothing else, this reduces "log noise" so makes future attempts a lot more obvious. The new port number should be high enough to reduce the likelihood that low range port scans will find it, eg: 30122 etc etc (under 65456 from memory).

 

[actived]

Re: Root logins from single unknown IP - what next steps, suspicious change

Thanks a ton, NetMantis and brianoz !

Meanwhile, if you have the hacker's current IP from the /usr/local/cpanel/logs/access_log file or other logs in /var/log then I would go ahead and ban that IP either through IP Tables or CSF or if you must /etc/hosts.deny. This will not stop the hacker as they can just simply change their apparent IP but it might slow them down just enough to buy yourself valuable time to resolve the current situation. Ideally, I'd use a different computer that has never before been used to change the password if that is a possible option to you.

I took all that in and changed the password from a Linux Live CD from my home computer (I dont use Windows )
I've emailed everyone concerned to not use the new password till I say so and not from Windows machines.
Also, I apart from our office IP ranges, I blocked all SSH access from Host Access Control - and tested it - blocking works all right ("Connection closed by remote host")

With the number of brute force root attempts these days you really do need to change the SSH port to something other than 22, if you haven't already. If nothing else, this reduces "log noise" so makes future attempts a lot more obvious. The new port number should be high enough to reduce the likelihood that low range port scans will find it, eg: 30122 etc etc (under 65456 from memory).

Doing this next.

Re your md5sum check - it does look like a lot of stuff has changed. Have you used rpm to verify your binaries? You may also want to compare to another system with same OS version. You mentioned the LF_INTEGRITY setting -- it does look like files have changed. You may want to compare both ways (compare the binaries on your system, then compare them on the other system) as it's possible (probably unlikely) they could have installed a kernel module.

This is the next priority item.

With this particular type of attack, it is not uncommon to replace index files on your sites with ones that link to trojan installer locations to infect more home users AND / OR drop in additional scripts to let themselves back in as a backdoor should you cut off their front means of gaining access.

Checked the website, doesnt seem to show any connections to unknown websites in Firebug Net Console. Planning to run a WebPagetest - Website Performance and Optimization Test test to see if they report any domains which we havent coded into the web app.

1. Aggressively scan your computers at home/office

I'll do this as soon as possible - without stepping on toes

The real test is whether it logs in again after you've changed the root password. If that happens, look for another PC that could have leaked the password. Silly question I know, but is the password also used in any unsecured protocols such as POP or FTP?

As I am learning, in system administration, no question is silly! - Not in FTP or POP. But PuTTY from Windows could be a risk. Looking for another login / login attempt is the thing, agreed.

There is one possibility though:
If a Cpanel update ran at the time that someone from the office logged in as root, then, it is possible that the binaries were changed by the update, the person who logged in is a nutcase for not having told me, and LFD reports this as a big issue.
And all these things happen concidentally and this appears to be an intrusion attack. But even in that case, what I cannot explain is how the LF_INTEGRITY check got disabled - does setting to TESTING disable that - I'll test that and let you know.

Thanks again, folks!
I'll keep you posted.

EDIT:
One more thing I wish to add: Our hosting provider sent us our VPS root password in plaintext email - how bad is that? I think very bad.

 

[NetMantis]

Re: Root logins from single unknown IP - what next steps, suspicious change

On the topic of checksums, I really wouldn't worry too much about the list of files that you posted. You can take a passing look at them to see if they have been altered but I suspect those are just coincidental updates on those files. You can re-download those files from author sources and check for changes.

The type of attack you described has a standard modus operandi they rarely deviate from and in that, I would be more concerned about the content of any websites that you host. Those are usually the first target and it's highly likely that digging through those you'll fine places where pages have been slightly altered or additional scripts have been added to allow for code injection or other access possibilities. They often make the added scripts look like graphics images to make it more difficult for you to find the added code which they simply call as an include from your other programs.

One more thing I wish to add: Our hosting provider sent us our VPS root password in plaintext email - how bad is that? I think very bad.

You would be correct. It is never a good idea to transmit a password in any manner where it could be intercepted.

(Generally speaking, it is safer though to send a password by email than it is posting it in a live chat network so if you are faced with one or the other, I'd recommend the password be sent by email but then changed immediately upon receipt)

As an important rule, I would recommend that you consider all passwords emailed to you to be "temporary" and you should change your password first thing you do immediately upon connecting.

 

[brianoz]

Re: Root logins from single unknown IP - what next steps, suspicious change

One more thing I wish to add: Our hosting provider sent us our VPS root password in plaintext email - how bad is that? I think very bad.

Actually, it's pretty much standard practice - they assume you will be changing it immediately!

If a Cpanel update ran at the time that someone from the office logged in as root, then, it is possible that the binaries were changed by the update, the person who logged in is a nutcase for not having told me, and LFD reports this as a big issue.
And all these things happen concidentally and this appears to be an intrusion attack. But even in that case, what I cannot explain is how the LF_INTEGRITY check got disabled - does setting to TESTING disable that - I'll test that and let you know.

To be honest, that's the most likely scenario! Let's hope so!

It's fairly unlikely that the integrity check would be disabled by a hacker but do let us know if it was!

 

[actived]

Re: Root logins from single unknown IP - what next steps, suspicious change

Actually, it's pretty much standard practice - they assume you will be changing it immediately!

That would be sensible if it were a new account. This was a server move, and we have been using that password for like 1 month now - so it wasnt that they thought we didn't know our own password. Why tell us our password, which we set and were using?

 

[actived]

Re: Root logins from single unknown IP - what next steps, suspicious change

Well, false alarm, after all. The nutcase in question owned up after much investigation. I guess we shouldnt attribute to malice what can be attributed to negligence (or something like that...)

I suspect the LF_INTEGRITY check gets suspended by the system because it is an update or something like that.

Anyway, thanks for the quick and informative replies!
Learnt a few more things!