perl processes hung - serious problem

[Secret Agent]

I am trying to figure out why several perl processes are hanging on the server and causing serious bandwidth output (up to 7mbps).

Results:

root@server2 [~]# lsof -p 23919
COMMAND   PID   USER   FD   TYPE    DEVICE      SIZE      NODE NAME
perl    23919 nobody  cwd    DIR       8,3      4096   1540232 /home/antro/public_html/foro
perl    23919 nobody  rtd    DIR       8,3      4096         2 /
perl    23919 nobody  txt    REG       8,3    969687   9559480 /usr/bin/perl
perl    23919 nobody  mem    REG       8,3     23202   9716305 /usr/lib/perl5/5.8.6/i686-linux/auto/Socket/Socket.so
perl    23919 nobody  mem    REG       8,3    106397   4949112 /lib/ld-2.3.4.so
perl    23919 nobody  mem    REG       8,3   1454462   4949114 /lib/tls/libc-2.3.4.so
perl    23919 nobody  mem    REG       8,3     15324   4949776 /lib/libdl-2.3.4.so
perl    23919 nobody  mem    REG       8,3    178019   4949747 /lib/tls/libm-2.3.4.so
perl    23919 nobody  mem    REG       8,3     27191   4949782 /lib/libcrypt-2.3.4.so
perl    23919 nobody  mem    REG       8,3     95148   4949127 /lib/libnsl-2.3.4.so
perl    23919 nobody  mem    REG       8,3     14542   4948054 /lib/libutil-2.3.4.so
perl    23919 nobody    0r   CHR       1,3                1608 /dev/null
perl    23919 nobody    1w  FIFO       0,7           598245316 pipe
perl    23919 nobody    2w  FIFO       0,7           598245316 pipe
perl    23919 nobody    3u   REG       8,3    449171   9718262 /usr/local/apache/logs/mod_jk.log
perl    23919 nobody    4r  FIFO       0,7           570443939 pipe
perl    23919 nobody    5u  IPv4 598245319                 UDP *:30144
perl    23919 nobody    7u   REG       7,0         0       495 /tmp/ZCUD4cMbxt (deleted)
perl    23919 nobody    8u   REG       8,3     66560   9717025 /usr/local/apache/logs/jk-runtime-status
perl    23919 nobody    9u   REG       8,3         1   9719001 /usr/local/apache/logs/jk-runtime-status.lock
perl    23919 nobody   10r   REG       8,3       152   9718251 /usr/local/apache/logs/mod_throttle.runtime
perl    23919 nobody   12w  FIFO       0,7           590371560 pipe
perl    23919 nobody   13r  FIFO       0,7           590371561 pipe
perl    23919 nobody   15w   REG       8,3         0   9718150 /usr/local/apache/logs/audit_log
perl    23919 nobody   16w   REG       8,3         0   9718261 /usr/local/apache/logs/modsec_debug_log
perl    23919 nobody   17w   REG       8,3  59604711   9719333 /usr/local/apache/logs/error_log
perl    23919 nobody   20w   REG       8,3         0   9737476 /usr/local/apache/domlogs/domain1.com-bytes_log
perl    23919 nobody   21w   REG       8,3         0   9736813 /usr/local/apache/domlogs/domain1.com-bytes_log
perl    23919 nobody   22w   REG       8,3         0   9735852 /usr/local/apache/domlogs/domain1.com-bytes_log
perl    23919 nobody   23w   REG       8,3         0   9735054 /usr/local/apache/domlogs/domain1.com-bytes_log
perl    23919 nobody   24w   REG       8,3         0   9735963 /usr/local/apache/domlogs/domain1.com-bytes_log
perl    23919 nobody   25w   REG       8,3         0   9738276 /usr/local/apache/domlogs/domain1.com-bytes_log

Now the last few lines regarding apache domlogs....the list actually goes on forever, what seems like a list of all domains on the server.

Can someone please explain what is causing these perl processes and how to stop them/prevent them for good?

cPanel 10.8x CURRENT
PHP 4.4.1
Apache 1.33x
Centos 4.2

Thank you.

[randomuser]

What are the names of the perl processes? Do they match up with a legit file on the server? Where is all this bandwidth being directed (tcpdump)? What was the file in /tmp that has been deleted, and why is the process running as "nobody" (got phpsuexec?)? What's in "/home/antro/public_html/foro", any outdated xmlrpc.php's or the like? My first guess is "antro" has some vulns somewhere in his foro directory and his account is being used to DoS via UDP.

[HostMerit]

Add these to your mod security ruleset, as I see you have it running.


SecFilter "perl\x20"
SecFilter "udp.pl"
SecFilter "udp.txt"
SecFilter "wget\x20"
SecFilter "cd\x20/tmp"

Also run ps -u nobody, and if anything except melange, or httpd processes are running there (common are perl psybnc and sh), check their /proc/PID/environ files, where PID is the process ID.

Check your /tmp and rm -rf *sess* to remove the clutter, and look for tools like udp.pl and udp.txt, and similar and remove / try to investigate how they got into the server. As it was /foro, I take it it is an outdated forum, probably phpBB, possibly a very old vBul. version.

[Secret Agent]

What exactly does that ruleset do and where is the ruleset? I have never customized to be honest

[chirpy]

I am trying to figure out why several perl processes are hanging on the server and causing serious bandwidth output (up to 7mbps).

Can someone please explain what is causing these perl processes and how to stop them/prevent them for good?

Yup, that output clearly shows that that account is ether being abused or has been compromised and within the directory /home/antro/public_html/foro is a perl script being used for a either DOS/DDOS attack over UDP, or possibly an IRC bot.

Solution:

Suspend the account immediately, then
Clean up the compromise
Work through that accounts domlogs and find the entry point
Remove any applications that were used as part of the compromise
Warn the user that they've risked the security of the entire server by using vulnerable PHP scripts - up to you if you give them a second chance

Lastly, check very very carefull for a root compromise which is only a single step from having already had the server hacked through a script.