massive high loads???? perl

[Secret Agent]

check this out

root@server [~]# top
top - 00:42:34 up 16 days, 23:42,  2 users,  load average: 101.40, 100.37, 99.07
Tasks: 443 total, 105 running, 333 sleeping,   1 stopped,   4 zombie
Cpu(s): 28.4% us, 14.8% sy,  0.0% ni,  0.0% id,  0.2% wa,  0.9% hi, 55.6% si
Mem:   2075104k total,  1875196k used,   199908k free,   143740k buffers
Swap:  1807304k total,     1620k used,  1805684k free,   787056k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3459 root      18   0 86204  78m 4820 R 41.0  3.9   0:05.70 yum
 4523 nobody    25   0  3920 2076 1460 R  6.9  0.1   9:16.61 perl
 8402 nobody    25   0  4580 2072 1460 R  6.9  0.1   7:00.92 perl
13217 nobody    25   0  4492 2072 1460 R  6.9  0.1   5:06.33 perl
27014 nobody    25   0  6900 3620 1804 R  6.6  0.2  40:23.88 perl
30128 nobody    25   0  4944 2076 1460 R  6.6  0.1  32:30.88 perl
 8367 nobody    25   0  4408 2076 1460 R  6.6  0.1   6:58.91 perl
 8376 nobody    25   0  5632 2072 1460 R  6.6  0.1   6:49.22 perl
15226 nobody    25   0  5436 2072 1460 R  6.6  0.1   4:24.54 perl
18402 nobody    25   0  5328 2076 1460 R  6.6  0.1   3:37.14 perl
18494 nobody    25   0  3700 2072 1460 R  6.6  0.1   3:36.06 perl
23612 nobody    25   0  5736 2076 1460 R  6.6  0.1   2:36.21 perl
23654 nobody    25   0  4920 2076 1460 R  6.6  0.1   2:31.25 perl
23707 nobody    25   0  4988 2072 1460 R  6.6  0.1   2:27.87 perl
 4724 nobody    25   0  5036 2072 1460 R  6.2  0.1   8:39.80 perl
18450 nobody    25   0  4492 2072 1460 R  6.2  0.1   3:37.50 perl
 4512 nobody    25   0  4620 2072 1460 R  5.6  0.1   9:19.19 perl
18425 nobody    25   0  5592 2076 1460 R  4.9  0.1   3:45.63 perl
18436 nobody    25   0  5200 2076 1460 R  4.9  0.1   3:38.49 perl
23645 nobody    25   0  3796 2072 1460 R  4.6  0.1   2:25.50 perl
13291 nobody    25   0  3760 2072 1460 R  4.3  0.1   4:49.66 perl
 4219 nobody    25   0  3788 2072 1460 R  3.6  0.1   9:57.59 perl
 4291 nobody    25   0  4316 2072 1460 R  3.6  0.1   9:40.59 perl
 4475 nobody    25   0  5016 2076 1460 R  3.6  0.1   9:20.49 perl
 4832 nobody    25   0  3976 2076 1460 R  3.6  0.1   8:44.36 perl
 7438 nobody    25   0  5396 2072 1460 R  3.6  0.1   7:15.61 perl
13343 nobody    25   0  4640 2076 1460 R  3.6  0.1   4:50.27 perl
18432 nobody    25   0  5600 2072 1460 R  3.6  0.1   3:46.19 perl
30602 nobody    25   0  4868 2072 1460 R  3.3  0.1  30:23.30 perl
 4211 nobody    25   0  4872 2076 1460 R  3.3  0.1   9:52.69 perl
 4229 nobody    25   0  5196 2072 1460 R  3.3  0.1   9:52.38 perl
 4256 nobody    25   0  4920 2072 1460 R  3.3  0.1   9:44.59 perl
 4272 nobody    25   0  5464 2076 1460 R  3.3  0.1   9:41.65 perl
 4468 nobody    25   0  4504 2076 1460 R  3.3  0.1   9:11.89 perl
 4488 nobody    25   0  4764 2072 1460 R  3.3  0.1   9:11.41 perl
 4500 nobody    25   0  5456 2072 1460 R  3.3  0.1   9:12.20 perl
 4733 nobody    25   0  4644 2072 1460 R  3.3  0.1   8:40.00 perl
 4741 nobody    25   0  4396 2072 1460 R  3.3  0.1   8:47.71 perl
 4750 nobody    25   0  4016 2076 1460 R  3.3  0.1   8:50.39 perl
 4777 nobody    25   0  5552 2076 1460 R  3.3  0.1   8:48.00 perl
 4814 nobody    25   0  4692 2072 1460 R  3.3  0.1   8:42.79 perl
 4841 nobody    25   0  4512 2072 1460 R  3.3  0.1   8:40.60 perl
 4857 nobody    25   0  4772 2072 1460 R  3.3  0.1   8:40.20 perl
 7380 nobody    24   0  5584 2076 1460 R  3.3  0.1   7:14.11 perl
 7430 nobody    25   0  4420 2072 1460 R  3.3  0.1   7:16.81 perl
 7464 nobody    25   0  5040 2076 1460 R  3.3  0.1   7:02.96 perl
 8409 nobody    25   0  4196 2072 1460 R  3.3  0.1   6:49.81 perl
 8420 nobody    25   0  4172 2072 1460 R  3.3  0.1   6:53.33 perl
13168 nobody    25   0  3756 2072 1460 R  3.3  0.1   5:03.33 perl
13247 nobody    25   0  3924 2072 1460 R  3.3  0.1   4:57.03 perl
13268 nobody    25   0  4180 2072 1460 R  3.3  0.1   4:59.13 perl
13280 nobody    25   0  3996 2072 1460 R  3.3  0.1   4:56.33 perl
13292 nobody    25   0  3928 2072 1460 R  3.3  0.1   4:58.76 perl
13308 nobody    25   0  4852 2072 1460 R  3.3  0.1   5:02.54 perl
13354 nobody    25   0  4696 2076 1460 R  3.3  0.1   4:58.63 perl
15242 nobody    25   0  5316 2076 1460 R  3.3  0.1   4:32.83 perl
15291 nobody    25   0  4724 2072 1460 R  3.3  0.1   4:20.53 perl
15329 nobody    25   0  5164 2072 1460 R  3.3  0.1   4:17.84 perl
15410 nobody    25   0  4852 2072 1460 R  3.3  0.1   4:18.34 perl
15420 nobody    25   0  5248 2076 1460 R  3.3  0.1   4:22.13 perl
15444 nobody    25   0  5072 2076 1460 R  3.3  0.1   4:27.04 perl
15491 nobody    25   0  4972 2072 1460 R  3.3  0.1   4:28.03 perl
15517 nobody    25   0  5092 2072 1460 R  3.3  0.1   4:31.33 perl
15534 nobody    25   0  5728 2076 1460 R  3.3  0.1   4:25.74 perl

How could I possibly trace this down and stop this perl nonsene?

 

[HostMerit]

Your server has hackers on it.

I suggest you get a SysAdmin ASAP to help you with this, Email me at [email protected], and let me know your AIM Name, I'll help you trace them.

For now.

Go to /proc/PID/

and ls -al, you may see some shortcuts, check enviroment settings too

IE:


cat /proc/13217/environ
cat /proc/27014/environ
cat /proc/30128/environ


ls -al /proc/13217/*
ls -al /proc/27014/*
ls -al /proc/30128/*

You may see /home/user/file , Suspend this user, then killall -9 perl.

Since those are the longest running, you have the best chance of tracing them.

Judging as they have so many open, alot will be BNCs, shells, possibly DOS malware, etc.

-Kris
Owner
[email protected]
http://www.hostmerit.com/

 

[Secret Agent]

I did a reboot and that fixed everything.

Looks like it was just hung thats all.

My server is very secure and not hacked.

 

[aby]

Did you run any updates or any scripts before this load shoot up.
I see
3459 root 18 0 86204 78m 4820 R 41.0 3.9 0:05.70 yum

That is the one taking too much resources.

Also you may check the /tmp for checking out if there are anything fishy.

I hope you had already sucured the /tmp.

 

[chirpy]

I did a reboot and that fixed everything.

Looks like it was just hung thats all.

My server is very secure and not hacked.

How do you know? Rebooting without tracing exactly what was happening is an extremely bad idea. You could have easily identified which perl script was running and why it was looping if you'd followed HostMerit's advice. Now, you won't know if it's a buggy script or an exploit running.

There's no such thing as a secure server, on the internet.

 

[Secret Agent]

[edit] mistake...will update in a second

 

[Secret Agent]

root@server [/proc/28455]# ls -al
total 0
dr-xr-xr-x    3 nobody nobody 0 Aug  5 20:47 ./
dr-xr-xr-x  298 root   root   0 Aug  5 01:06 ../
dr-xr-xr-x    2 nobody nobody 0 Aug  5 22:57 attr/
-r--------    1 nobody nobody 0 Aug  5 22:56 auxv
-r--r--r--    1 nobody nobody 0 Aug  5 20:47 cmdline
lrwxrwxrwx    1 nobody nobody 0 Aug  5 22:56 cwd -> /tmp/.s.mlock/
-r--------    1 nobody nobody 0 Aug  5 22:56 environ
lrwxrwxrwx    1 nobody nobody 0 Aug  5 20:47 exe -> /usr/bin/perl*
dr-x------    2 nobody nobody 0 Aug  5 22:57 fd/
-r--------    1 nobody nobody 0 Aug  5 22:56 maps
-rw-------    1 nobody nobody 0 Aug  5 22:56 mem
-r--r--r--    1 nobody nobody 0 Aug  5 22:56 mounts
lrwxrwxrwx    1 nobody nobody 0 Aug  5 22:56 root -> //
-r--r--r--    1 nobody nobody 0 Aug  5 20:47 stat
-r--r--r--    1 nobody nobody 0 Aug  5 22:54 statm
-r--r--r--    1 nobody nobody 0 Aug  5 20:47 status
dr-xr-xr-x    3 nobody nobody 0 Aug  5 22:57 task/
-r--r--r--    1 nobody nobody 0 Aug  5 22:56 wchan

root@server [/proc/28455]# cat /proc/28455/environ

CONSOLE=/dev/consoleSELINUX_INIT=YESTERM=linuxINIT_VERSION=sysvinit-2.85PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin_=/usr/bin/perlrunlevel=3RUNLEVEL=3PWD=/tmp/.s.mlockLANG=en_US.UTF-8previous=NPREVLEVEL=Nacpi=htSHLVL=5

When I ran this

ls -al /proc/28455/*

I got so many lines, but was not able to scroll back far enough. How can I do page at a time and ability to scroll down manually?

 

[Secret Agent]

Here's a little more info

root@server [/tmp]# ls -al
total 5644
drwxrwxrwt   8 root     root       61440 Aug  5 23:03 ./
drwxr-xr-x  24 root     root        4096 Aug  5 01:07 ../
-rw-rw-rw-   1 cruster  cruster        5 Aug  5 23:00 .302.494500
-rw-rw-rw-   1 rentals  rentals        5 Aug  5 20:10 .302.c84329
-rw-rw----   1 aspidx   aspidx        13 Aug  5 20:15 aspidx-session-0.92384265137753
-rw-rw----   1 bodybui  bodybui       13 Aug  5 12:29 bodybui-session-0.779935596118779
-rw-rw----   1 brenda   brenda        13 Aug  5 15:44 brenda-session-0.943682348894985
-rw-rw----   1 ccneiva  ccneiva       13 Aug  5 19:56 ccneiva-session-0.0719644705104692
-rw-rw----   1 ccneiva  ccneiva       13 Aug  5 20:34 ccneiva-session-0.998504642005177
-rw-r--r--   1 brians   brians   5595136 Aug  5 14:27 cpanel.TMP.nn318j9qx2Y8JOz8
-rw-r--r--   1 mailnull mail           0 Aug  5 23:01 exim_deny.lock
-rw-r--r--   1 nobody   nobody        84 Aug  5 03:46 fm76d97e.txt
-rw-r--r--   1 cpanel   cpanel      3714 Aug  5 21:27 horde_502.log
drwxr-xr-x   2 root     root        4096 Aug  5 01:07 hsperfdata_root/
drwxr-xr-x   2 tomcat   nobody      4096 Aug  5 01:07 hsperfdata_tomcat/
drwxrwxrwt   2 root     root        4096 Aug  5 01:06 .ICE-unix/
drwx------   2 root     root       16384 Dec 30  2004 lost+found/
-rw-rw----   1 mascotas mascotas      13 Aug  5 10:41 mascotas-session-0.0578936679094149
-rw-rw----   1 morenova morenova      13 Aug  5 23:04 morenova-session-0.0798675150881571
lrwxrwxrwx   1 root     root          30 Aug  5 22:59 mysql.sock -> ../../var/lib/mysql/mysql.sock=
-rw-r--r--   1 netbula  netbula      104 Aug  5 19:24 pd
?---------   ? ?        ?              ?            ? rpm-tmp.10647
-rw-------   1 nobody   nobody       935 Aug  5 23:02 sess_68d14105b078b4adcefef2c45b56fb24
-rw-rw----   1 sitima78 sitima78      13 Aug  5 03:11 sitima78-session-0.920409495317475
drwxrwxrwx   2 nobody   nobody      4096 Aug  5 21:02 .s.mlock/
srwxrwxrwx   1 postgres postgres       0 Aug  5 23:02 .s.PGSQL.5432=
-rw-------   1 postgres postgres      25 Aug  5 23:02 .s.PGSQL.5432.lock
drwxrwxrwt   3 root     root        8192 Feb 16 09:25 tmp/