The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

hacked, please help

Discussion in 'General Discussion' started by Secret Agent, Jun 9, 2005.

Thread Status:
Not open for further replies.
  1. Secret Agent

    Secret Agent Guest

    I ran chkrootkit and got this:

    Checking `bindshell'... INFECTED (PORTS: 465)
    Checking `lkm'... You have 1 process hidden for readdir command
    You have 1 process hidden for ps command
    Warning: Possible LKM Trojan installed
    Checking `rexedcs'... not found
    Checking `sniffer'... /proc/11167/fd: No such file or directory
    /proc/12463/fd: No such file or directory


    I ran rkhunter and got this:

    Code:
    Info: prelinked files found
      Performing 'known good' check...
       /usr/bin/find                                              [ OK ]
       /usr/bin/file                                              [ OK ]
       /usr/bin/kill                                              [ BAD ]
       /usr/bin/killall                                           [ OK ]
       /usr/bin/lsattr                                            [ OK ]
       /usr/bin/pstree                                            [ OK ]
       /usr/bin/sha1sum                                           [ OK ]
       /usr/bin/stat                                              [ OK ]
       /usr/bin/users                                             [ OK ]
       /usr/bin/w                                                 [ BAD ]
       /usr/bin/watch                                             [ BAD ]
       /usr/bin/who                                               [ OK ]
       /usr/bin/whoami                                            [ OK ]
       /bin/mount                                                 [ BAD ]
       /bin/netstat                                               [ OK ]
       /bin/egrep                                                 [ OK ]
       /bin/fgrep                                                 [ OK ]
       /bin/grep                                                  [ OK ]
       /bin/cat                                                   [ OK ]
       /bin/chmod                                                 [ OK ]
       /bin/chown                                                 [ OK ]
       /bin/env                                                   [ OK ]
       /bin/ls                                                    [ OK ]
       /bin/su                                                    [ OK ]
       /bin/ps                                                    [ BAD ]
       /bin/dmesg                                                 [ BAD ]
       /bin/kill                                                  [ BAD ]
       /bin/login                                                 [ BAD ]
       /sbin/chkconfig                                            [ OK ]
       /sbin/depmod                                               [ OK ]
       /sbin/ifconfig                                             [ OK ]
       /sbin/insmod                                               [ OK ]
       /sbin/ip                                                   [ OK ]
       /sbin/modinfo                                              [ OK ]
       /sbin/sysctl                                               [ BAD ]
       /sbin/syslogd                                              [ OK ]
       /sbin/init                                                 [ OK ]
       /sbin/runlevel                                             [ OK ]
    
       Checking /dev for suspicious files...                      [ Warning! (unusual files found) ]
    
    MD5
    MD5 compared: 40
    Incorrect MD5 checksums: 9
    
    File scan
    Scanned files: 342
    Possible infected files: 0
    
    Application scan
    Vulnerable applications: 2
    
    
    How can I resolve this? I'll do anything but having to do an OS reload.

    I got the following specs:

    Fedora Core 2
    APF Firewall / BFD
    cPanel 10.2x
    PHP 4.3.11
    Apache 1.33x

    Some security measure:
    APF
    BFD
    Bind masked
    Apache masked
    LES
    mod security
    mod dosevasive
    Sysctl.conf hardened
    Exim Dictionary Attack ACL


    Thank you for your help
     
  2. simplybe

    simplybe Well-Known Member

    Joined:
    Nov 29, 2002
    Messages:
    153
    Likes Received:
    0
    Trophy Points:
    16
    first run rkhunter --update in ssh then run rkhunter again from shell /usr/local/bin/rkhunter -c

    run chkrootkit again

    Checking `bindshell'... INFECTED (PORTS: 465) is usually a false positive and may not be a problem.


    Checking `lkm'... You have 1 process hidden for readdir command
    You have 1 process hidden for ps command

    This is not usual but can be a a false positive, you should investigate further, see what processes are running, check for anything trying to open unusal ports etc...

    If in doubt get/pay an expert to take a look, i have heard good things about http://www.rack911.com but have no experience myself with them.
     
  3. Secret Agent

    Secret Agent Guest

    Ok rkunter shows only this as red

    /usr/bin/kill

    * Filesystem checks
    Checking /dev for suspicious files... [ Warning! (unusual files found) ]

    Whatever has happened is making my server reboot by itself also, happened 3 times last 2 days.


    chkrootkit shows the same
     
  4. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    If unusual files are detected then you should check the rkhunter log file thoroughly (/var/log/rkhunter.log) which will tell you what unexpected files they are (they can sometimes be tmp partition files created there for example which would be a false-positive) but could just as easily be an sshd password logger.

    If you have actually been hacked, then you really have no choice other than to have an OS reload done. If you do go down that route, then you should upgrade to a supported OS since FC2 has been EOL for some time now (preferably completely away from Fedora).

    You really need to hire someone to check your server over for you as determining whether you have definitely been hacked is farily easy to the experienced eye, checking that you have not been hacked can be very difficult indeed.

    Just a final note, if you have suffered a root compromise you should never ever settle for cleaning the OS unless you go the root of taking the disk out of the server and sending it to a qualified security specialist company. You simply cannot guarantee that the server is clean unless you have the disk wiped and reloaded afresh.
     
  5. anup123

    anup123 Well-Known Member

    Joined:
    Mar 29, 2004
    Messages:
    897
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    This Planet
    Here is a script which i found floating somewhere to check for those hidden process alarms:

    #!/usr/bin/perl
    #
    # Tue Nov 4 17:37:18 GMT 2003
    # Copyright Robin * Slomkowski
    #
    # This file is distributed under the GPL
    # Full text can be found at http://www.gnu.org/copyleft/gpl.html
    #

    use strict ;

    my ( @dirs,
    $top_proc,
    $proc_dir,
    $dir,
    %proc_dirs,
    %ps_procs,
    %ps_vprocs,
    $debug,
    $last_pid,
    $virtual_pid,
    ) ;

    # build the ps list

    open PS, "ps -ef |" ;
    while (<PS>) {
    my $pid ;
    $_ =~ /^\S*\s+(\d+)/ ;
    $pid = $1 ;
    if ( $pid == 0 ) {
    $virtual_pid = $last_pid + 1 ;
    $_ =~ /\[(.*)\]$/ ;
    $ps_vprocs{$virtual_pid} = $1;
    $last_pid = $virtual_pid ;
    } else {
    $ps_procs{$pid} = 1 ;
    $last_pid = $pid ;
    }
    }
    close PS ;

    # build the directory listing
    opendir DIR, "/proc" ;
    @dirs = readdir DIR ;
    closedir DIR ;

    @dirs = grep /[0-9]+/, @dirs ;
    @dirs = sort { $a <=> $b } @dirs ;

    foreach $proc_dir ( @dirs ) {
    $proc_dirs{$proc_dir} = 1 ;
    }

    $top_proc = $dirs[$#dirs] ;

    # check the dirs

    $dir = 0 ;
    while ( $dir <= $top_proc ) {
    my $re ;
    $re = chdir "/proc/$dir" ;
    if ( $re ) {
    if ( ! defined ($proc_dirs{$dir}) ) {
    print "ERROR: not listable: /proc/$dir\n" ;
    }
    if ( ! defined ($ps_procs{$dir}) ) {
    print "WARN: not in ps: /proc/$dir" ;
    if ( defined ($ps_vprocs{$dir}) ) {
    print " - probable virtual thread [$ps_vprocs{$dir}]\n" ;
    } else {
    print "\n" ;
    }
    }
    } else {
    print "cannot chrdir to $dir\n" if $debug > 0 ;
    }
    $dir ++ ;
    }

    exit 0 ;

    When run it gives a list of /proc/pid not listable in ps and then each can be checked manually.


    Anup
     
  6. Secret Agent

    Secret Agent Guest

    root@server [~]# sh check

    check: line 10: use: command not found
    check: line 12: syntax error near unexpected token `@dirs,'
    check: line 12: `my ( @dirs,'
     
  7. HelloJeff

    HelloJeff Guest

    It's a perl script not a shell script (chmod 700 check ; ./check)
     
  8. anup123

    anup123 Well-Known Member

    Joined:
    Mar 29, 2004
    Messages:
    897
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    This Planet
    or better still ./check > check.txt

    Then go thru check.txt and then check the listed /proc/pid

    Anup
     
  9. Secret Agent

    Secret Agent Guest

    My mistake sorry

    Results

    root@server [~]# chmod 700 check ; ./check
    ERROR: not listable: /proc/2471
    WARN: not in ps: /proc/2471
    ERROR: not listable: /proc/2472
    WARN: not in ps: /proc/2472
    ERROR: not listable: /proc/2473
    WARN: not in ps: /proc/2473
    ERROR: not listable: /proc/2474
    WARN: not in ps: /proc/2474
    ERROR: not listable: /proc/2475
    WARN: not in ps: /proc/2475
    ERROR: not listable: /proc/2476
    WARN: not in ps: /proc/2476
     
  10. dgbaker

    dgbaker Well-Known Member
    PartnerNOC

    Joined:
    Sep 20, 2002
    Messages:
    2,578
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Toronto, Ontario Canada
    cPanel Access Level:
    DataCenter Provider
    Goto to each of those directories listed and using ls -l see what the cmd or exe is pointing to.

    /usr/sbin/named is one example of a false positive with this script.
     
  11. anup123

    anup123 Well-Known Member

    Joined:
    Mar 29, 2004
    Messages:
    897
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    This Planet
    Depending upon procps version and other factors (like for example if running Mysql-Max, or running SiteStudio, DCC etc) the ps doesn't list just about everything and those are the one's that cause alse alarms. But this lets you check each and every unlisted stuff.

    Anup
     
  12. dgbaker

    dgbaker Well-Known Member
    PartnerNOC

    Joined:
    Sep 20, 2002
    Messages:
    2,578
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Toronto, Ontario Canada
    cPanel Access Level:
    DataCenter Provider
    Here is a nice little wrap, for for the check.pl script

    Code:
    for i in `./check.pl |grep -v ERROR|awk {'print $5'}`;do ls -l $i|grep cwd;done
    
    This will list the cwd link in the found proc directories and show you the commands that are being called.
     
  13. anup123

    anup123 Well-Known Member

    Joined:
    Mar 29, 2004
    Messages:
    897
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    This Planet
    A nice timesaver.
    grep cwd or grep exe?

    Anup
     
    #13 anup123, Jun 9, 2005
    Last edited: Jun 9, 2005
  14. dgbaker

    dgbaker Well-Known Member
    PartnerNOC

    Joined:
    Sep 20, 2002
    Messages:
    2,578
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Toronto, Ontario Canada
    cPanel Access Level:
    DataCenter Provider
    I usually do cwd, but you're right in this case it would be a good idea to grep both exe and cwd.

    You could actually remove the grep and pipe all to file as well.
     
  15. Secret Agent

    Secret Agent Guest

    If it helps anyone, ended up being the buggy kernel on FC2, just what I thought it would be.

    I'd like to point out that Fedora sucks all around :) My experience anyway, even when FC2 first came out - junk.

    centos / enterprise all the way, never had any issues ever and been using both for a long time.

    Maybe I'm wrong, (I know servre is not compromised, which is very nice to know I did a good job so far on security lol)

    2.6.10 (and, to a lesser extent, the 2.6.9 series) for Fedora Core 2.

    The tech at Voxel (very nice people, friendly and truly professional / advanced in administration) told me the kernel is buggy. Not the first time anyway.

    What do you all say?
     
  16. HelloJeff

    HelloJeff Guest

    If it were me, I'd compare the md5sums of the 9 binaries rkhunter flagged against those listed at knowngoods.org. I'd also be curious to know what exactly is in /dev that it threw a warning about. Did you ever determine that whatever is listening on port 465 is a legitimate service? And the 2 vulnerable applications, which are those?
     
  17. AlexSmithMCP

    AlexSmithMCP Well-Known Member

    Joined:
    May 26, 2004
    Messages:
    66
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    fedora normaly shows vulnerable applications as they back port the security fixes. Kinda confusing if you ask me....
     
  18. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Yup, as HelloJeff has said, you still have to track down those md5sum problems and especially the unusual files that were found as neither of those issues have anything to do with your kernel and are the most worrying of all the problems you listed.
     
  19. anup123

    anup123 Well-Known Member

    Joined:
    Mar 29, 2004
    Messages:
    897
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    This Planet
    on RH9 i have identical report on MD5checksum though i am dead sure that i am not hacked.
    i do find a directory /etc/.java dated back Aug 2004 which rkhunter reports as suspicious but there is nothing there. Aug 2004 was the time i remeber playing with tomcat stuff which in any case in not there on my box

    Anup
     
  20. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    That's fine, so long as you're confident in the md5sums you ought to be OK. That said, I've installed and run rkhunter 1.2.7 updated with the latest definitions and it always runs cleaning on RH9 servers that I've worked on. They're likely to mismatch only if you have upgraded the relevant apps from source rather than using the last official rpms.

    Another way to double-check would be to use rpm to verify the md5sums are the same as the original rpm when installed (unless of course you installed from source) by using:

    rpm -V package
     
Loading...
Thread Status:
Not open for further replies.

Share This Page