The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Root logins from single unknown IP - what next steps, suspicious changes

Discussion in 'Security' started by actived, Apr 29, 2012.

  1. actived

    actived Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Website Owner
    Hi,

    We have had root logins from an unknown IP a few times in the last few hours. Nothing seems to have been changed or damaged. This one was not a brute force attempt. So it could be someone in the organisation, legitimately logging in, but finding out who takes time in a company - and some damage can be done by then.

    The damage could be insidious, hidden - installation of some rogue software / script / etc.

    1. First thing I did was to change the root password.

    2. Next, I plan to install and run rkhunter (which, yes, wasn't installed, so my bad). I have read that if root is compromised then a worst case scenario is that system binaries and even an installed rkhunter can be compromised/modified. So luckily in that sense, rkhunter is now reliable ;)

    3. I have CSF+LFD running which is how I found out in the first place.
    But the interesting thing is that in CSF, there is the option LF_INTEGRITY which I *definitely* had set to enabled, and it now shows the red "Warning" message.

    I had to put CSF back into "Testing" mode for a while - does this disable LF_INTEGRITY checking automatically?
    PT_SKIP_HTTP and PT_ALL_USERS are also not in force ("Warning")

    Do these three have to re-enabled every time when setting Testing to 0?

    4. Brute force login attempts from rogue IP ranges is a very common daily occurrence with us now. CpHulk seems to discourage those fairly well (30 tries = banned for 15 days)

    We have, of course, backed up everything offline.

    Any other suggestions?

    Thanks in advance,
    Dave
     
    #1 actived, Apr 29, 2012
  2. actived

    actived Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Website Owner
    Re: Root logins from single unknown IP - what next steps, suspicious change

    Also, here is an email alert from lfd:
    Code:
    Subject: 
lfd on domain.com: System Integrity checking detected a modified system file

Body:
The following list of files have FAILED the md5sum comparison test. This means that the file has been changed in some way. This could be a result of an OS update or application upgrade. If the change is unexpected it should be investigated:

/usr/bin/afs5log: FAILED
/usr/bin/amtu: FAILED
/usr/bin/aulastlog: FAILED
/usr/bin/ausyscall: FAILED
/usr/bin/chage: FAILED
/usr/bin/chfn: FAILED
/usr/bin/chsh: FAILED
/usr/bin/cvs: FAILED
/usr/bin/gpasswd: FAILED
/usr/bin/lchfn: FAILED
/usr/bin/lchsh: FAILED
/usr/bin/newgrp: FAILED
/usr/bin/passwd: FAILED
/usr/bin/php: FAILED
/usr/bin/php5: FAILED
/usr/bin/php5-cgi: FAILED
/usr/bin/php5-cli: FAILED
/usr/bin/php-cgi: FAILED
/usr/bin/php-cli: FAILED
/usr/bin/php-config: FAILED
/usr/bin/pkcs11_eventmgr: FAILED
/usr/bin/pkcs11_inspect: FAILED
/usr/bin/pkcs11_setup: FAILED
/usr/bin/pklogin_finder: FAILED
/usr/bin/scanpci: FAILED
/usr/bin/screen: FAILED
/usr/bin/sg: FAILED
/usr/bin/X: FAILED
/usr/bin/Xnest: FAILED
/usr/bin/Xorg: FAILED
/usr/bin/Xvfb: FAILED
/usr/sbin/adduser: FAILED
/usr/sbin/cc_dump: FAILED
/usr/sbin/cc_test: FAILED
/usr/sbin/dovecotpw: FAILED
/usr/sbin/exim: FAILED
/usr/sbin/exim_dbmbuild: FAILED
/usr/sbin/exim_dumpdb: FAILED
/usr/sbin/exim_fixdb: FAILED
/usr/sbin/exim_lock: FAILED
/usr/sbin/exim_tidydb: FAILED
/usr/sbin/groupadd: FAILED
/usr/sbin/groupdel: FAILED
/usr/sbin/groupmod: FAILED
/usr/sbin/hwclock: FAILED
/usr/sbin/lchage: FAILED
/usr/sbin/lgroupadd: FAILED
/usr/sbin/lgroupdel: FAILED
/usr/sbin/lgroupmod: FAILED
/usr/sbin/lnewusers: FAILED
/usr/sbin/lpasswd: FAILED
/usr/sbin/luseradd: FAILED
/usr/sbin/luserdel: FAILED
/usr/sbin/lusermod: FAILED
/usr/sbin/runq: FAILED
/usr/sbin/saslauthd: FAILED
/usr/sbin/useradd: FAILED
/usr/sbin/userdel: FAILED
/usr/sbin/userhelper: FAILED
/usr/sbin/usermod: FAILED
/bin/login: FAILED
/bin/passwd: FAILED
/sbin/clock: FAILED
/sbin/hwclock: FAILED
/sbin/pam_console_apply: FAILED
/sbin/pam_tally: FAILED
/sbin/pam_tally2: FAILED
/sbin/runuser: FAILED
/usr/local/bin/php: FAILED
/usr/local/bin/php-config: FAILED
    Is it confirmed that this is not a false positive of some type?
     
    #2 actived, Apr 29, 2012
  3. NetMantis

    NetMantis BANNED

    Joined:
    Apr 22, 2012
    Messages:
    116
    Likes Received:
    1
    Trophy Points:
    66
    Location:
    Utah
    cPanel Access Level:
    DataCenter Provider
    Re: Root logins from single unknown IP - what next steps, suspicious change

    The only part of your message that I really need to see is this:
    That statement alone tells me that your computer (or more specifically any that have ever logged in as root before) is very likely infected with a trojan virus designed specifically to give hackers your hosting login information.

    This particular type of hacking has been widely in use for roughly three years or so now and is quite difficult to defend because those behind it already know the correct login in advance and just simply login --- no brute force attempts needed!

    Changing your password often does no good either because the moment you change your password, the hackers are updated with the new password information which is why it is imperative you do complete full virus and trojan scans of all computers you know of which may legitimately log into your server as the root administrator. Once you are absolutely certain that the server has been cleaned, then go ahead and change your root password again.

    Meanwhile, if you have the hacker's current IP from the /usr/local/cpanel/logs/access_log file or other logs in /var/log then I would go ahead and ban that IP either through IP Tables or CSF or if you must /etc/hosts.deny. This will not stop the hacker as they can just simply change their apparent IP but it might slow them down just enough to buy yourself valuable time to resolve the current situation. Ideally, I'd use a different computer that has never before been used to change the password if that is a possible option to you.

    After you have your home computer(s) cleared, passwords changed, and known IPs for the hacker banned, then your next task will be going through your whole entire server looking for items that have been changed (activity logs MAY help) and you need to also check all the web sites that you host. With this particular type of attack, it is not uncommon to replace index files on your sites with ones that link to trojan installer locations to infect more home users AND / OR drop in additional scripts to let themselves back in as a backdoor should you cut off their front means of gaining access.

    It's kind of a nasty and painful situation all around but if you are methodical in getting everything cleaned up then you should be okay and get over this.

    There is no 100% foolproof way of totally protecting yourself from this kind of situation but there is still many different things you can do to protect your server so that this does not happen ever again. Once you get cleaned up a bit, if you want a hand at re-security hardening your server, I would be happy to give you a hand to help make sure you don't miss anything along the way that might leave you vulnerable. Just contact me later on if you need the help.

    Right now your priorities would be:

    1. Aggressively scan your computers at home/office

    2. It is very rare for a hacker to login to root and do nothing other than that so you will
    need to find out exactly what it is that they did to you starting with a deep review of the
    web sites you host followed by the rest of the server applications and operating system.

    Good luck to you. If it makes you feel any better, I've personally seen the reports of hundreds
    of different servers attacked this way over the years so you aren't alone in any of this.
     
    #3 NetMantis, Apr 29, 2012
  4. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    168
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Re: Root logins from single unknown IP - what next steps, suspicious change

    Obviously one urgent thing to do is to contact your team members and find out whether anyone did log in from home. If not, you might want to consider restricting root login via ssh to an IP range or two, then changing the password, as a temporary measure while you work out what happened.

    As Netmantis says above, the password could well have been stolen by a trojan / virus keylogger running on your PC. Do a virus scan on any PCs that may have logged in as root, with at least two different scanners.

    The real test is whether it logs in again after you've changed the root password. If that happens, look for another PC that could have leaked the password. Silly question I know, but is the password also used in any unsecured protocols such as POP or FTP?

    Re your md5sum check - it does look like a lot of stuff has changed. Have you used rpm to verify your binaries? You may also want to compare to another system with same OS version. You mentioned the LF_INTEGRITY setting -- it does look like files have changed. You may want to compare both ways (compare the binaries on your system, then compare them on the other system) as it's possible (probably unlikely) they could have installed a kernel module.

    With the number of brute force root attempts these days you really do need to change the SSH port to something other than 22, if you haven't already. If nothing else, this reduces "log noise" so makes future attempts a lot more obvious. The new port number should be high enough to reduce the likelihood that low range port scans will find it, eg: 30122 etc etc (under 65456 from memory).
     
    #4 brianoz, Apr 29, 2012
  5. actived

    actived Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Website Owner
    Re: Root logins from single unknown IP - what next steps, suspicious change

    Thanks a ton, NetMantis and brianoz !

    I took all that in and changed the password from a Linux Live CD from my home computer (I dont use Windows :) )
    I've emailed everyone concerned to not use the new password till I say so and not from Windows machines.
    Also, I apart from our office IP ranges, I blocked all SSH access from Host Access Control - and tested it - blocking works all right ("Connection closed by remote host")

    Doing this next.

    This is the next priority item.

    Checked the website, doesnt seem to show any connections to unknown websites in Firebug Net Console. Planning to run a WebPagetest - Website Performance and Optimization Test test to see if they report any domains which we havent coded into the web app.

    I'll do this as soon as possible - without stepping on toes ;)

    As I am learning, in system administration, no question is silly! - Not in FTP or POP. But PuTTY from Windows could be a risk. Looking for another login / login attempt is the thing, agreed.

    There is one possibility though:
    If a Cpanel update ran at the time that someone from the office logged in as root, then, it is possible that the binaries were changed by the update, the person who logged in is a nutcase for not having told me, and LFD reports this as a big issue.
    And all these things happen concidentally and this appears to be an intrusion attack. But even in that case, what I cannot explain is how the LF_INTEGRITY check got disabled - does setting to TESTING disable that - I'll test that and let you know.

    Thanks again, folks!
    I'll keep you posted.

    EDIT:
    One more thing I wish to add: Our hosting provider sent us our VPS root password in plaintext email - how bad is that? I think very bad.
     
    #5 actived, Apr 29, 2012
    Last edited: Apr 29, 2012
  6. NetMantis

    NetMantis BANNED

    Joined:
    Apr 22, 2012
    Messages:
    116
    Likes Received:
    1
    Trophy Points:
    66
    Location:
    Utah
    cPanel Access Level:
    DataCenter Provider
    Re: Root logins from single unknown IP - what next steps, suspicious change

    On the topic of checksums, I really wouldn't worry too much about the list of files that you posted. You can take a passing look at them to see if they have been altered but I suspect those are just coincidental updates on those files. You can re-download those files from author sources and check for changes.

    The type of attack you described has a standard modus operandi they rarely deviate from and in that, I would be more concerned about the content of any websites that you host. Those are usually the first target and it's highly likely that digging through those you'll fine places where pages have been slightly altered or additional scripts have been added to allow for code injection or other access possibilities. They often make the added scripts look like graphics images to make it more difficult for you to find the added code which they simply call as an include from your other programs.

    You would be correct. It is never a good idea to transmit a password in any manner where it could be intercepted.

    (Generally speaking, it is safer though to send a password by email than it is posting it in a live chat network so if you are faced with one or the other, I'd recommend the password be sent by email but then changed immediately upon receipt)

    As an important rule, I would recommend that you consider all passwords emailed to you to be "temporary" and you should change your password first thing you do immediately upon connecting.
     
    #6 NetMantis, Apr 29, 2012
  7. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    168
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Re: Root logins from single unknown IP - what next steps, suspicious change

    Actually, it's pretty much standard practice - they assume you will be changing it immediately! :)


    To be honest, that's the most likely scenario! Let's hope so!

    It's fairly unlikely that the integrity check would be disabled by a hacker but do let us know if it was!
     
    #7 brianoz, Apr 29, 2012
    Last edited: Apr 29, 2012
  8. actived

    actived Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Website Owner
    Re: Root logins from single unknown IP - what next steps, suspicious change

    That would be sensible if it were a new account. This was a server move, and we have been using that password for like 1 month now - so it wasnt that they thought we didn't know our own password. Why tell us our password, which we set and were using?
     
    #8 actived, Apr 30, 2012
  9. actived

    actived Well-Known Member

    Joined:
    Mar 30, 2012
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    56
    cPanel Access Level:
    Website Owner
    Re: Root logins from single unknown IP - what next steps, suspicious change

    Well, false alarm, after all. The nutcase in question owned up after much investigation. I guess we shouldnt attribute to malice what can be attributed to negligence (or something like that...)

    I suspect the LF_INTEGRITY check gets suspended by the system because it is an update or something like that.

    Anyway, thanks for the quick and informative replies!
    Learnt a few more things!
     
    #9 actived, May 1, 2012
(You must log in or sign up to post here.)
Loading...
Similar Threads - Root logins single
  1. tryingwebman

    Jailed Shell and Root Logins

    tryingwebman, Sep 28, 2015, in forum: Security
    Replies:
    10
    Views:
    1,631
    cPanelMichael
    Oct 15, 2015
  2. erdeme61

    account "xxx" has user ID 0 (root privileges)

    erdeme61, Jul 28, 2017 at 1:21 AM, in forum: Security
    Replies:
    1
    Views:
    31
    cPanelMichael
    Jul 28, 2017 at 10:34 AM
  3. ayonilari

    Root WHM access disallowed after User WHM access from Cpanel

    ayonilari, Jul 25, 2017 at 4:46 AM, in forum: General Discussion
    Replies:
    6
    Views:
    52
    ayonilari
    Jul 25, 2017 at 10:25 AM
  4. EneTar

    Define default values for /opt/cpanel/ea-php70/root/etc/php-fpm.d/*.conf

    EneTar, Jul 21, 2017, in forum: EasyApache
    Replies:
    3
    Views:
    236
    cPanelMichael
    Jul 26, 2017 at 10:58 AM
  5. gizmo80

    SOLVED Reducing inodes on VolGroup-lv_root?

    gizmo80, Jul 20, 2017, in forum: General Discussion
    Replies:
    4
    Views:
    58
    gizmo80
    Jul 21, 2017

Share This Page