The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

massive high loads???? perl

Discussion in 'General Discussion' started by Secret Agent, Aug 4, 2005.

  1. Secret Agent

    Secret Agent Guest

    check this out

    Code:
    
    root@server [~]# top
    top - 00:42:34 up 16 days, 23:42,  2 users,  load average: 101.40, 100.37, 99.07
    Tasks: 443 total, 105 running, 333 sleeping,   1 stopped,   4 zombie
    Cpu(s): 28.4% us, 14.8% sy,  0.0% ni,  0.0% id,  0.2% wa,  0.9% hi, 55.6% si
    Mem:   2075104k total,  1875196k used,   199908k free,   143740k buffers
    Swap:  1807304k total,     1620k used,  1805684k free,   787056k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     3459 root      18   0 86204  78m 4820 R 41.0  3.9   0:05.70 yum
     4523 nobody    25   0  3920 2076 1460 R  6.9  0.1   9:16.61 perl
     8402 nobody    25   0  4580 2072 1460 R  6.9  0.1   7:00.92 perl
    13217 nobody    25   0  4492 2072 1460 R  6.9  0.1   5:06.33 perl
    27014 nobody    25   0  6900 3620 1804 R  6.6  0.2  40:23.88 perl
    30128 nobody    25   0  4944 2076 1460 R  6.6  0.1  32:30.88 perl
     8367 nobody    25   0  4408 2076 1460 R  6.6  0.1   6:58.91 perl
     8376 nobody    25   0  5632 2072 1460 R  6.6  0.1   6:49.22 perl
    15226 nobody    25   0  5436 2072 1460 R  6.6  0.1   4:24.54 perl
    18402 nobody    25   0  5328 2076 1460 R  6.6  0.1   3:37.14 perl
    18494 nobody    25   0  3700 2072 1460 R  6.6  0.1   3:36.06 perl
    23612 nobody    25   0  5736 2076 1460 R  6.6  0.1   2:36.21 perl
    23654 nobody    25   0  4920 2076 1460 R  6.6  0.1   2:31.25 perl
    23707 nobody    25   0  4988 2072 1460 R  6.6  0.1   2:27.87 perl
     4724 nobody    25   0  5036 2072 1460 R  6.2  0.1   8:39.80 perl
    18450 nobody    25   0  4492 2072 1460 R  6.2  0.1   3:37.50 perl
     4512 nobody    25   0  4620 2072 1460 R  5.6  0.1   9:19.19 perl
    18425 nobody    25   0  5592 2076 1460 R  4.9  0.1   3:45.63 perl
    18436 nobody    25   0  5200 2076 1460 R  4.9  0.1   3:38.49 perl
    23645 nobody    25   0  3796 2072 1460 R  4.6  0.1   2:25.50 perl
    13291 nobody    25   0  3760 2072 1460 R  4.3  0.1   4:49.66 perl
     4219 nobody    25   0  3788 2072 1460 R  3.6  0.1   9:57.59 perl
     4291 nobody    25   0  4316 2072 1460 R  3.6  0.1   9:40.59 perl
     4475 nobody    25   0  5016 2076 1460 R  3.6  0.1   9:20.49 perl
     4832 nobody    25   0  3976 2076 1460 R  3.6  0.1   8:44.36 perl
     7438 nobody    25   0  5396 2072 1460 R  3.6  0.1   7:15.61 perl
    13343 nobody    25   0  4640 2076 1460 R  3.6  0.1   4:50.27 perl
    18432 nobody    25   0  5600 2072 1460 R  3.6  0.1   3:46.19 perl
    30602 nobody    25   0  4868 2072 1460 R  3.3  0.1  30:23.30 perl
     4211 nobody    25   0  4872 2076 1460 R  3.3  0.1   9:52.69 perl
     4229 nobody    25   0  5196 2072 1460 R  3.3  0.1   9:52.38 perl
     4256 nobody    25   0  4920 2072 1460 R  3.3  0.1   9:44.59 perl
     4272 nobody    25   0  5464 2076 1460 R  3.3  0.1   9:41.65 perl
     4468 nobody    25   0  4504 2076 1460 R  3.3  0.1   9:11.89 perl
     4488 nobody    25   0  4764 2072 1460 R  3.3  0.1   9:11.41 perl
     4500 nobody    25   0  5456 2072 1460 R  3.3  0.1   9:12.20 perl
     4733 nobody    25   0  4644 2072 1460 R  3.3  0.1   8:40.00 perl
     4741 nobody    25   0  4396 2072 1460 R  3.3  0.1   8:47.71 perl
     4750 nobody    25   0  4016 2076 1460 R  3.3  0.1   8:50.39 perl
     4777 nobody    25   0  5552 2076 1460 R  3.3  0.1   8:48.00 perl
     4814 nobody    25   0  4692 2072 1460 R  3.3  0.1   8:42.79 perl
     4841 nobody    25   0  4512 2072 1460 R  3.3  0.1   8:40.60 perl
     4857 nobody    25   0  4772 2072 1460 R  3.3  0.1   8:40.20 perl
     7380 nobody    24   0  5584 2076 1460 R  3.3  0.1   7:14.11 perl
     7430 nobody    25   0  4420 2072 1460 R  3.3  0.1   7:16.81 perl
     7464 nobody    25   0  5040 2076 1460 R  3.3  0.1   7:02.96 perl
     8409 nobody    25   0  4196 2072 1460 R  3.3  0.1   6:49.81 perl
     8420 nobody    25   0  4172 2072 1460 R  3.3  0.1   6:53.33 perl
    13168 nobody    25   0  3756 2072 1460 R  3.3  0.1   5:03.33 perl
    13247 nobody    25   0  3924 2072 1460 R  3.3  0.1   4:57.03 perl
    13268 nobody    25   0  4180 2072 1460 R  3.3  0.1   4:59.13 perl
    13280 nobody    25   0  3996 2072 1460 R  3.3  0.1   4:56.33 perl
    13292 nobody    25   0  3928 2072 1460 R  3.3  0.1   4:58.76 perl
    13308 nobody    25   0  4852 2072 1460 R  3.3  0.1   5:02.54 perl
    13354 nobody    25   0  4696 2076 1460 R  3.3  0.1   4:58.63 perl
    15242 nobody    25   0  5316 2076 1460 R  3.3  0.1   4:32.83 perl
    15291 nobody    25   0  4724 2072 1460 R  3.3  0.1   4:20.53 perl
    15329 nobody    25   0  5164 2072 1460 R  3.3  0.1   4:17.84 perl
    15410 nobody    25   0  4852 2072 1460 R  3.3  0.1   4:18.34 perl
    15420 nobody    25   0  5248 2076 1460 R  3.3  0.1   4:22.13 perl
    15444 nobody    25   0  5072 2076 1460 R  3.3  0.1   4:27.04 perl
    15491 nobody    25   0  4972 2072 1460 R  3.3  0.1   4:28.03 perl
    15517 nobody    25   0  5092 2072 1460 R  3.3  0.1   4:31.33 perl
    15534 nobody    25   0  5728 2076 1460 R  3.3  0.1   4:25.74 perl
    

    How could I possibly trace this down and stop this perl nonsene?
     

    Attached Files:

    • 1.gif
      1.gif
      File size:
      8.2 KB
      Views:
      30
    #1 Secret Agent, Aug 4, 2005
    Last edited by a moderator: Aug 4, 2005
  2. HostMerit

    HostMerit Well-Known Member

    Joined:
    Oct 24, 2004
    Messages:
    160
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New Jersey, USA
    cPanel Access Level:
    DataCenter Provider
    Your server has hackers on it.

    I suggest you get a SysAdmin ASAP to help you with this, Email me at kris@hostmerit.com, and let me know your AIM Name, I'll help you trace them.

    For now.

    Go to /proc/PID/

    and ls -al, you may see some shortcuts, check enviroment settings too

    IE:


    cat /proc/13217/environ
    cat /proc/27014/environ
    cat /proc/30128/environ


    ls -al /proc/13217/*
    ls -al /proc/27014/*
    ls -al /proc/30128/*

    You may see /home/user/file , Suspend this user, then killall -9 perl.

    Since those are the longest running, you have the best chance of tracing them.

    Judging as they have so many open, alot will be BNCs, shells, possibly DOS malware, etc.

    -Kris
    Owner
    kris@hostmerit.com
    http://www.hostmerit.com/
     
  3. Secret Agent

    Secret Agent Guest

    I did a reboot and that fixed everything.

    Looks like it was just hung thats all.

    My server is very secure and not hacked.
     
  4. aby

    aby Well-Known Member

    Joined:
    May 31, 2005
    Messages:
    638
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    India
    Did you run any updates or any scripts before this load shoot up.
    I see
    3459 root 18 0 86204 78m 4820 R 41.0 3.9 0:05.70 yum

    That is the one taking too much resources.

    Also you may check the /tmp for checking out if there are anything fishy.

    I hope you had already sucured the /tmp.
     
  5. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    How do you know? Rebooting without tracing exactly what was happening is an extremely bad idea. You could have easily identified which perl script was running and why it was looping if you'd followed HostMerit's advice. Now, you won't know if it's a buggy script or an exploit running.

    There's no such thing as a secure server, on the internet.
     
  6. Secret Agent

    Secret Agent Guest

    [edit] mistake...will update in a second
     
  7. Secret Agent

    Secret Agent Guest

    Code:
    root@server [/proc/28455]# ls -al
    total 0
    dr-xr-xr-x    3 nobody nobody 0 Aug  5 20:47 ./
    dr-xr-xr-x  298 root   root   0 Aug  5 01:06 ../
    dr-xr-xr-x    2 nobody nobody 0 Aug  5 22:57 attr/
    -r--------    1 nobody nobody 0 Aug  5 22:56 auxv
    -r--r--r--    1 nobody nobody 0 Aug  5 20:47 cmdline
    lrwxrwxrwx    1 nobody nobody 0 Aug  5 22:56 cwd -> /tmp/.s.mlock/
    -r--------    1 nobody nobody 0 Aug  5 22:56 environ
    lrwxrwxrwx    1 nobody nobody 0 Aug  5 20:47 exe -> /usr/bin/perl*
    dr-x------    2 nobody nobody 0 Aug  5 22:57 fd/
    -r--------    1 nobody nobody 0 Aug  5 22:56 maps
    -rw-------    1 nobody nobody 0 Aug  5 22:56 mem
    -r--r--r--    1 nobody nobody 0 Aug  5 22:56 mounts
    lrwxrwxrwx    1 nobody nobody 0 Aug  5 22:56 root -> //
    -r--r--r--    1 nobody nobody 0 Aug  5 20:47 stat
    -r--r--r--    1 nobody nobody 0 Aug  5 22:54 statm
    -r--r--r--    1 nobody nobody 0 Aug  5 20:47 status
    dr-xr-xr-x    3 nobody nobody 0 Aug  5 22:57 task/
    -r--r--r--    1 nobody nobody 0 Aug  5 22:56 wchan
    
    root@server [/proc/28455]# cat /proc/28455/environ

    CONSOLE=/dev/consoleSELINUX_INIT=YESTERM=linuxINIT_VERSION=sysvinit-2.85PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin_=/usr/bin/perlrunlevel=3RUNLEVEL=3PWD=/tmp/.s.mlockLANG=en_US.UTF-8previous=NPREVLEVEL=Nacpi=htSHLVL=5

    When I ran this

    ls -al /proc/28455/*

    I got so many lines, but was not able to scroll back far enough. How can I do page at a time and ability to scroll down manually?
     
  8. Secret Agent

    Secret Agent Guest

    Here's a little more info


    Code:
    root@server [/tmp]# ls -al
    total 5644
    drwxrwxrwt   8 root     root       61440 Aug  5 23:03 ./
    drwxr-xr-x  24 root     root        4096 Aug  5 01:07 ../
    -rw-rw-rw-   1 cruster  cruster        5 Aug  5 23:00 .302.494500
    -rw-rw-rw-   1 rentals  rentals        5 Aug  5 20:10 .302.c84329
    -rw-rw----   1 aspidx   aspidx        13 Aug  5 20:15 aspidx-session-0.92384265137753
    -rw-rw----   1 bodybui  bodybui       13 Aug  5 12:29 bodybui-session-0.779935596118779
    -rw-rw----   1 brenda   brenda        13 Aug  5 15:44 brenda-session-0.943682348894985
    -rw-rw----   1 ccneiva  ccneiva       13 Aug  5 19:56 ccneiva-session-0.0719644705104692
    -rw-rw----   1 ccneiva  ccneiva       13 Aug  5 20:34 ccneiva-session-0.998504642005177
    -rw-r--r--   1 brians   brians   5595136 Aug  5 14:27 cpanel.TMP.nn318j9qx2Y8JOz8
    -rw-r--r--   1 mailnull mail           0 Aug  5 23:01 exim_deny.lock
    -rw-r--r--   1 nobody   nobody        84 Aug  5 03:46 fm76d97e.txt
    -rw-r--r--   1 cpanel   cpanel      3714 Aug  5 21:27 horde_502.log
    drwxr-xr-x   2 root     root        4096 Aug  5 01:07 hsperfdata_root/
    drwxr-xr-x   2 tomcat   nobody      4096 Aug  5 01:07 hsperfdata_tomcat/
    drwxrwxrwt   2 root     root        4096 Aug  5 01:06 .ICE-unix/
    drwx------   2 root     root       16384 Dec 30  2004 lost+found/
    -rw-rw----   1 mascotas mascotas      13 Aug  5 10:41 mascotas-session-0.0578936679094149
    -rw-rw----   1 morenova morenova      13 Aug  5 23:04 morenova-session-0.0798675150881571
    lrwxrwxrwx   1 root     root          30 Aug  5 22:59 mysql.sock -> ../../var/lib/mysql/mysql.sock=
    -rw-r--r--   1 netbula  netbula      104 Aug  5 19:24 pd
    ?---------   ? ?        ?              ?            ? rpm-tmp.10647
    -rw-------   1 nobody   nobody       935 Aug  5 23:02 sess_68d14105b078b4adcefef2c45b56fb24
    -rw-rw----   1 sitima78 sitima78      13 Aug  5 03:11 sitima78-session-0.920409495317475
    drwxrwxrwx   2 nobody   nobody      4096 Aug  5 21:02 .s.mlock/
    srwxrwxrwx   1 postgres postgres       0 Aug  5 23:02 .s.PGSQL.5432=
    -rw-------   1 postgres postgres      25 Aug  5 23:02 .s.PGSQL.5432.lock
    drwxrwxrwt   3 root     root        8192 Feb 16 09:25 tmp/
    
     
Loading...

Share This Page