The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

High Server Load - Perl is the culprit

Discussion in 'General Discussion' started by davecoop, Sep 25, 2007.

  1. davecoop

    davecoop Registered

    Joined:
    Mar 7, 2006
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    The load in the past week on one of my servers seems to be a bit high. Here is the output of top:

    Code:
    top - 15:58:09 up 2 days, 20:32,  1 user,  load average: 2.03, 2.20, 2.46
    Tasks: 132 total,   3 running, 127 sleeping,   1 stopped,   1 zombie
    Cpu(s):  0.2% us, 22.8% sy, 76.7% ni,  0.3% id,  0.0% wa,  0.0% hi,  0.0% si
    Mem:    969584k total,   930664k used,    38920k free,    59076k buffers
    Swap:  2064376k total,      144k used,  2064232k free,   404764k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
     3862 nobody    39  19  9188 3648  992 R   99  0.4  10:21.53 perl
     3799 nobody    39  19  9420 3648  992 R   99  0.4  10:55.10 perl
    
    As you can see something is running from perl. Here is the output of the ps command:

    Code:
    nobody    3799 91.2  0.3  9420 3648 ?        RN   15:46  14:05 /usr/sbin
    nobody    3859  0.0  0.0     0    0 ?        Z    15:47   0:00 [sh] <defunct>
    nobody    3862 93.6  0.3  9188 3648 ?        RN   15:47  13:31 /usr/sbin
    
    Every time I kill these two perl commands right away the processes come back. I feel that this server may have been hacked. However I was wondering if there is any other commands I'm unaware of to find out what exactly the two "perl" processes are exactly running. Any help would be greatly appreciated.
     
  2. mtindor

    mtindor Well-Known Member

    Joined:
    Sep 14, 2004
    Messages:
    1,279
    Likes Received:
    36
    Trophy Points:
    48
    Location:
    inside a catfish
    cPanel Access Level:
    Root Administrator
    See if you can get the full path - ps auwwx . and do an 'lsof perl'

    If you'd like, I'll take a look - but you've have to PM me if you want me to do that. I can also pass along my phone # via PM if you determine you'd like me to help you look for the culprit.

    Mike


     
  3. heath

    heath Member

    Joined:
    Jul 11, 2003
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
    I have this same issue and it's only for the past couple days. My load average gets very high (~10). Here is the output from matching perl processes consuming the CPU (there are many more):

    /bin/httpd -k start -DSSL
    nobody 13052 0.0 0.8 18924 8836 ? S 19:28 0:03 /usr/local/apache/bin/httpd -k start -DSSL
     
  4. brianoz

    brianoz Well-Known Member

    Joined:
    Mar 13, 2004
    Messages:
    1,146
    Likes Received:
    6
    Trophy Points:
    38
    Location:
    Melbourne, Australia
    cPanel Access Level:
    Root Administrator
    Looks like a hacked user account to me. Try "lsof PID" where you get pid from top or "ps aux".
     
  5. gbh1935

    gbh1935 Active Member

    Joined:
    Jun 25, 2005
    Messages:
    26
    Likes Received:
    0
    Trophy Points:
    1
    I having the same issue too, the only thing that look wierd in my lsof listing is

    perl 2884 nobody 4u IPv4 148265203 TCP nameremoved.com:41833->66-249-137-137-btl.blacksun.net:ircd (ESTABLISHED)
    p

    I don't have IRC enabled on my server, could it be an apache flaw of some kind?
     
  6. gbh1935

    gbh1935 Active Member

    Joined:
    Jun 25, 2005
    Messages:
    26
    Likes Received:
    0
    Trophy Points:
    1
    I did a restart of apache, and the annoying load and ircd went away, it did not immediately return.
     
  7. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Restarting apache will simply kill that instance. It still means that you have an IRC script installed on the server, most likely uploaded to a user account through an exploitable PHP script (since it's running under nobody). The rest of the top of the lsof output might indicate where the culrpit is, otherwise you're going to have to sift through your user directories to find it.
     
  8. heath

    heath Member

    Joined:
    Jul 11, 2003
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
    I've tried to sift thru the lsof logs for the culprit processes and don't see any perl scripts listed. Could the data be piped into perl via the IRC server that my server is taking commands from? That still wouldn't explain how my server is connected to this IRC server in the first place...
     
  9. heath

    heath Member

    Joined:
    Jul 11, 2003
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
    root@root [/home]# /usr/sbin/lsof | grep 14945 | more
    perl 14945 nobody cwd DIR 3,3 4096 2 /
    perl 14945 nobody rtd DIR 3,3 4096 2 /
    perl 14945 nobody txt REG 3,3 942825 62427 /usr/bin/per
    l
    perl 14945 nobody mem REG 3,3 1573180 99348 /lib/tls/lib
    c-2.3.2.so
    perl 14945 nobody mem REG 3,3 76488 99311 /lib/libreso
    lv-2.3.2.so
    perl 14945 nobody mem REG 3,3 17495 78863 /usr/lib/per
    l5/5.8.7/i686-linux/auto/IO/IO.so
    perl 14945 nobody mem REG 3,3 12488 99344 /lib/libutil
    -2.3.2.so
    perl 14945 nobody mem REG 3,3 94540 99252 /lib/libnsl-
    2.3.2.so
    perl 14945 nobody mem REG 3,3 18576 99267 /lib/libnss_
    dns-2.3.2.so
    perl 14945 nobody mem REG 3,3 213308 99354 /lib/tls/lib
    m-2.3.2.so
    perl 14945 nobody mem REG 3,3 106884 98659 /lib/ld-2.3.
    2.so
    perl 14945 nobody 5r FIFO 0,5 16211 pipe
    perl 14945 nobody 6w FIFO 0,5 16211 pipe
    perl 14945 nobody mem REG 3,3 24411 79053 /usr/lib/per
    l5/5.8.7/i686-linux/auto/Socket/Socket.so
    perl 14945 nobody 52u sock 0,0 12321043 can't identi
    fy protocol
    perl 14945 nobody 53u unix 0xf22c6200 12321044 socket
    perl 14945 nobody 54u IPv4 12321064 TCP localhost:55550->83.140.172.212:6665 (ESTABLISHED)
    perl 14945 nobody 55u IPv4 12416256 UDP *:57989


    That is the lsof output from one of the culprit perl processes. Why is no script listed here? Could the script be piped in rather than executed from a file? Perhaps the data is being piped thru the connection to the IRC server? I have no idea where to go from here... I've checked apache logs for wget, udp, curl, etc. but nothing comes up.
     
  10. idealso

    idealso Active Member

    Joined:
    Mar 1, 2007
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    1
    I fixed one of those today. In my case, it was easy to find the culprit. Since we use suphp, I was able to see which account was running it, look at all processes being run by the account, and identify the file. In this case, it was an old version of webcalendar. Fortunately, I have a strict firewall, and irc isn't one of the allowed protocols.

    You should be able to see the parent processes with "ps auxf". Also, if you identify the compromised account, you can look at the web logs to identify the vulnerable script.
     
  11. heath

    heath Member

    Joined:
    Jul 11, 2003
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
    Thanks. I've just setup suphp and suexec so hopefully if this happens again it will be much quicker to narrow it down.
     
Loading...

Share This Page