High Server Load - Perl is the culprit

[davecoop]

The load in the past week on one of my servers seems to be a bit high. Here is the output of top:

top - 15:58:09 up 2 days, 20:32,  1 user,  load average: 2.03, 2.20, 2.46
Tasks: 132 total,   3 running, 127 sleeping,   1 stopped,   1 zombie
Cpu(s):  0.2% us, 22.8% sy, 76.7% ni,  0.3% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:    969584k total,   930664k used,    38920k free,    59076k buffers
Swap:  2064376k total,      144k used,  2064232k free,   404764k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3862 nobody    39  19  9188 3648  992 R   99  0.4  10:21.53 perl
 3799 nobody    39  19  9420 3648  992 R   99  0.4  10:55.10 perl

As you can see something is running from perl. Here is the output of the ps command:

nobody    3799 91.2  0.3  9420 3648 ?        RN   15:46  14:05 /usr/sbin
nobody    3859  0.0  0.0     0    0 ?        Z    15:47   0:00 [sh] <defunct>
nobody    3862 93.6  0.3  9188 3648 ?        RN   15:47  13:31 /usr/sbin

Every time I kill these two perl commands right away the processes come back. I feel that this server may have been hacked. However I was wondering if there is any other commands I'm unaware of to find out what exactly the two "perl" processes are exactly running. Any help would be greatly appreciated.

[mtindor]

See if you can get the full path - ps auwwx . and do an 'lsof perl'

If you'd like, I'll take a look - but you've have to PM me if you want me to do that. I can also pass along my phone # via PM if you determine you'd like me to help you look for the culprit.

Mike

The load in the past week on one of my servers seems to be a bit high. Here is the output of top:

top - 15:58:09 up 2 days, 20:32,  1 user,  load average: 2.03, 2.20, 2.46
Tasks: 132 total,   3 running, 127 sleeping,   1 stopped,   1 zombie
Cpu(s):  0.2% us, 22.8% sy, 76.7% ni,  0.3% id,  0.0% wa,  0.0% hi,  0.0% si
Mem:    969584k total,   930664k used,    38920k free,    59076k buffers
Swap:  2064376k total,      144k used,  2064232k free,   404764k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3862 nobody    39  19  9188 3648  992 R   99  0.4  10:21.53 perl
 3799 nobody    39  19  9420 3648  992 R   99  0.4  10:55.10 perl

As you can see something is running from perl. Here is the output of the ps command:

nobody    3799 91.2  0.3  9420 3648 ?        RN   15:46  14:05 /usr/sbin
nobody    3859  0.0  0.0     0    0 ?        Z    15:47   0:00 [sh] <defunct>
nobody    3862 93.6  0.3  9188 3648 ?        RN   15:47  13:31 /usr/sbin

Every time I kill these two perl commands right away the processes come back. I feel that this server may have been hacked. However I was wondering if there is any other commands I'm unaware of to find out what exactly the two "perl" processes are exactly running. Any help would be greatly appreciated.

[heath]

I have this same issue and it's only for the past couple days. My load average gets very high (~10). Here is the output from matching perl processes consuming the CPU (there are many more):

/bin/httpd -k start -DSSL
nobody 13052 0.0 0.8 18924 8836 ? S 19:28 0:03 /usr/local/apache/bin/httpd -k start -DSSL

[brianoz]

Looks like a hacked user account to me. Try "lsof PID" where you get pid from top or "ps aux".

[gbh1935]

I having the same issue too, the only thing that look wierd in my lsof listing is

perl 2884 nobody 4u IPv4 148265203 TCP nameremoved.com:41833->66-249-137-137-btl.blacksun.net:ircd (ESTABLISHED)
p

I don't have IRC enabled on my server, could it be an apache flaw of some kind?

[gbh1935]

I did a restart of apache, and the annoying load and ircd went away, it did not immediately return.

[chirpy]

Restarting apache will simply kill that instance. It still means that you have an IRC script installed on the server, most likely uploaded to a user account through an exploitable PHP script (since it's running under nobody). The rest of the top of the lsof output might indicate where the culrpit is, otherwise you're going to have to sift through your user directories to find it.

[heath]

I've tried to sift thru the lsof logs for the culprit processes and don't see any perl scripts listed. Could the data be piped into perl via the IRC server that my server is taking commands from? That still wouldn't explain how my server is connected to this IRC server in the first place...

[heath]

root@root [/home]# /usr/sbin/lsof | grep 14945 | more
perl 14945 nobody cwd DIR 3,3 4096 2 /
perl 14945 nobody rtd DIR 3,3 4096 2 /
perl 14945 nobody txt REG 3,3 942825 62427 /usr/bin/per
l
perl 14945 nobody mem REG 3,3 1573180 99348 /lib/tls/lib
c-2.3.2.so
perl 14945 nobody mem REG 3,3 76488 99311 /lib/libreso
lv-2.3.2.so
perl 14945 nobody mem REG 3,3 17495 78863 /usr/lib/per
l5/5.8.7/i686-linux/auto/IO/IO.so
perl 14945 nobody mem REG 3,3 12488 99344 /lib/libutil
-2.3.2.so
perl 14945 nobody mem REG 3,3 94540 99252 /lib/libnsl-
2.3.2.so
perl 14945 nobody mem REG 3,3 18576 99267 /lib/libnss_
dns-2.3.2.so
perl 14945 nobody mem REG 3,3 213308 99354 /lib/tls/lib
m-2.3.2.so
perl 14945 nobody mem REG 3,3 106884 98659 /lib/ld-2.3.
2.so
perl 14945 nobody 5r FIFO 0,5 16211 pipe
perl 14945 nobody 6w FIFO 0,5 16211 pipe
perl 14945 nobody mem REG 3,3 24411 79053 /usr/lib/per
l5/5.8.7/i686-linux/auto/Socket/Socket.so
perl 14945 nobody 52u sock 0,0 12321043 can't identi
fy protocol
perl 14945 nobody 53u unix 0xf22c6200 12321044 socket
perl 14945 nobody 54u IPv4 12321064 TCP localhost:55550->83.140.172.212:6665 (ESTABLISHED)
perl 14945 nobody 55u IPv4 12416256 UDP *:57989


That is the lsof output from one of the culprit perl processes. Why is no script listed here? Could the script be piped in rather than executed from a file? Perhaps the data is being piped thru the connection to the IRC server? I have no idea where to go from here... I've checked apache logs for wget, udp, curl, etc. but nothing comes up.

[idealso]

I fixed one of those today. In my case, it was easy to find the culprit. Since we use suphp, I was able to see which account was running it, look at all processes being run by the account, and identify the file. In this case, it was an old version of webcalendar. Fortunately, I have a strict firewall, and irc isn't one of the allowed protocols.

You should be able to see the parent processes with "ps auxf". Also, if you identify the compromised account, you can look at the web logs to identify the vulnerable script.

[heath]

Thanks. I've just setup suphp and suexec so hopefully if this happens again it will be much quicker to narrow it down.