The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server overload

Discussion in 'General Discussion' started by jwhysleep, Oct 10, 2005.

  1. jwhysleep

    jwhysleep Member

    Joined:
    Aug 8, 2003
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    98 processes: 92 sleeping, 3 running, 2 zombie, 1 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 84.4% 0.0% 14.6% 0.0% 0.8% 0.0% 0.0%
    Mem: 1030640k av, 987796k used, 42844k free, 0k shrd, 82292k buff
    754592k actv, 186692k in_d, 4964k in_c
    Swap: 1052248k av, 136152k used, 916096k free 674120k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    7311 nobody 25 0 3000 2976 1432 R 98.9 0.2 2485m 0 perl
    1 root 15 0 456 424 396 S 0.0 0.0 0:04 0 init
    2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
    3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd
    4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    7 root 16 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
    5 root 15 0 0 0 0 SW 0.0 0.0 0:03 0 kswapd
    6 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand
    8 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
    9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    13 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 kjournald
    67 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    2331 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2332 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2653 root 15 0 568 560 488 S 0.0 0.0 0:01 0 syslogd
    2657 root 15 0 448 436 388 S 0.0 0.0 0:00 0 klogd
    7438 root 15 0 948 812 700 S 0.0 0.0 0:01 0 sshd
    7452 root 15 0 776 732 648 S 0.0 0.0 0:00 0 xinetd
    7465 root 25 0 776 536 532 S 0.0 0.0 0:00 0 sshd
    7507 postgres 15 0 1032 784 688 S 0.0 0.0 0:00 0 postmaster
    7509 postgres 25 0 984 648 644 S 0.0 0.0 0:00 0 postmaster
    7510 postgres 25 0 1008 648 644 S 0.0 0.0 0:00 0 postmaster
    7581 mailnull 15 0 1276 1276 944 S 0.0 0.1 0:00 0 exim
    7585 mailnull 15 0 1196 1196 864 S 0.0 0.1 0:00 0 exim
    7591 root 15 0 1064 1012 784 S 0.0 0.0 0:00 0 antirelayd
    7629 root 15 0 21636 2796 1172 S 0.0 0.2 0:01 0 spamd
    7679 root 15 0 608 600 524 S 0.0 0.0 0:00 0 crond
    7771 root 15 0 892 676 600 S 0.0 0.0 0:00 0 pure-ftpd
    7775 root 15 0 572 436 392 S 0.0 0.0 0:00 0 pure-authd
    7834 xfs 15 0 2956 584 556 S 0.0 0.0 0:00 0 xfs
    8164 root 15 0 484 432 428 S 0.0 0.0 0:00 0 rhnsd
    8228 root 25 0 440 376 372 S 0.0 0.0 0:00 0 portsentry
    8242 root 22 0 400 352 348 S 0.0 0.0 0:00 0 mingetty
    8243 root 22 0 396 352 348 S 0.0 0.0 0:00 0 mingetty
    8244 root 22 0 400 352 348 S 0.0 0.0 0:00 0 mingetty
    8245 root 22 0 396 352 348 S 0.0 0.0 0:00 0 mingetty
    8246 root 22 0 396 352 348 S 0.0 0.0 0:00 0 mingetty
    8247 root 22 0 400 352 348 S 0.0 0.0 0:00 0 mingetty
    8646 root 25 0 644 516 512 S 0.0 0.0 0:00 0 mysqld_safe
    8667 mysql 15 0 24708 15M 1792 S 0.0 1.5 0:02 0 mysqld
    8668 mysql 15 0 24708 15M 1792 S 0.0 1.5 0:02 0 mysqld
    8669 mysql 20 0 24708 15M 1792 S 0.0 1.5 0:00 0 mysqld
    8670 mysql 20 0 24708 15M 1792 S 0.0 1.5 0:00 0 mysqld
    8671 mysql 25 0 24708 15M 1792 S 0.0 1.5 0:00 0 mysqld
    8672 mysql 20 0 24708 15M 1792 S 0.0 1.5 0:00 0 mysqld
    8673 mysql 15 0 24708 15M 1792 S 0.0 1.5 0:00 0 mysqld
    8674 mysql 15 0 24708 15M 1792 S 0.0 1.5 0:00 0 mysqld
    8675 mysql 25 0 24708 15M 1792 S 0.0 1.5 0:00 0 mysqld
    8676 mysql 15 0 24708 15M 1792 S 0.0 1.5 0:00 0 mysqld
    9818 root 15 0 3700 1360 996 S 0.0 0.1 0:00 0 httpd
    11810 root 18 0 7528 1900 1092 S 0.0 0.1 0:05 0 chkservd
    11929 nobody 15 0 12932 11M 2676 S 0.0 1.1 2:50 0 httpd
    11930 nobody 15 0 12484 11M 2296 S 0.0 1.1 2:24 0 httpd
    11931 nobody 15 0 12436 11M 2256 S 0.0 1.1 2:30 0 httpd
    11932 nobody 15 0 12660 11M 2308 S 0.0 1.1 2:27 0 httpd
    11933 nobody 15 0 11552 10M 2224 S 0.0 1.0 2:31 0 httpd
    11936 nobody 15 0 10744 7816 2136 S 0.0 0.7 1:55 0 httpd
     
  2. jwhysleep

    jwhysleep Member

    Joined:
    Aug 8, 2003
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    ***bump***
     
  3. domtaj

    domtaj Active Member

    Joined:
    Aug 29, 2005
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    Looks like there is a perl script that is taking almost all your CPU cycles.

    "7311 nobody 25 0 3000 2976 1432 R 98.9 0.2 2485m 0 perl"

    "ps -f 7311" sans quote to see what's up
     
  4. jwhysleep

    jwhysleep Member

    Joined:
    Aug 8, 2003
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    UID PID PPID C STIME TTY STAT TIME CMD
    nobody 7311 1 98 Oct09 ? R 3421:16 /usr/sbin/inetd
     
  5. jwhysleep

    jwhysleep Member

    Joined:
    Aug 8, 2003
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    well im trying to locate the problem so this doesnt happen again i know if i reboot it will go away for the time being but trying to fugure out how to prevent this again!
     
  6. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    That's most likely an exploit masquerading as the init process which your server most likely doesn't use anyway. You'll need to have the server cleaned of exploits, checked that a root kit hasn't been installed and security hardened.
     
  7. jwhysleep

    jwhysleep Member

    Joined:
    Aug 8, 2003
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Security has been checked
    i figured the reason why i got this was because i ran the last script
    /scripts/realperlinstaller DBD::mysql
    /scripts/restartsrv_eximstats
    I forgot the reason something to do with perl and mysql but did this a monthg ago.
     
  8. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Those have nothing to do with that process. It's almost 100% likely to be an exploit as it's using a foged process name and running under the nobody user. My advice stands, it's an exploit and the server needs cleaning and the script that has been hacked needs fixing.
     
  9. jwhysleep

    jwhysleep Member

    Joined:
    Aug 8, 2003
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Ok so as a novice is there any tutorials or anywhere you can point to clean the server out to make these exploits go away
     
  10. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Not really, no. Such experience is gained over a long period of time. You basically need to establish exactly what is running on that PID using tools such as lsof. Once you've identified what is actually being run, you need to kill it off and remove it. Then use the information from what you discovered about when the exploit was made and look back through your server logs to try and identify when and how access was gained. In this case your domlogs and main apache error_log will be the key files in determining that. You should, as a matter of course, ensure that all phpBB and phpNuke installations are all running the latest version of phpBB v2.0.17 otherwise you will be exploited through them.
     
  11. jwhysleep

    jwhysleep Member

    Joined:
    Aug 8, 2003
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    ok ran lsof -p 7311

    COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
    perl 7311 nobody cwd DIR 3,3 4096 311297 /var/tmp
    perl 7311 nobody rtd DIR 3,3 4096 2 /
    perl 7311 nobody txt REG 3,3 1002181 3736193 /usr/bin/perl
    perl 7311 nobody mem REG 3,3 51952 2736221 /lib/libnss_files-2.3.2.so
    perl 7311 nobody mem REG 3,3 1573120 7979557 /lib/tls/libc-2.3.2.so
    perl 7311 nobody mem REG 3,3 12544 2736178 /lib/libutil-2.3.2.so
    perl 7311 nobody mem REG 3,3 23388 2736138 /lib/libcrypt-2.3.2.so
    perl 7311 nobody mem REG 3,3 213508 7979015 /lib/tls/libm-2.3.2.so
    perl 7311 nobody mem REG 3,3 14868 2736177 /lib/libdl-2.3.2.so
    perl 7311 nobody mem REG 3,3 91040 2736182 /lib/libnsl-2.3.2.so
    perl 7311 nobody mem REG 3,3 24582 4702354 /usr/lib/perl5/5.8.1/i686-linux/auto/Socket/Socket.so
    perl 7311 nobody mem REG 3,3 17474 4325491 /usr/lib/perl5/5.8.1/i686-linux/auto/IO/IO.so
    perl 7311 nobody mem REG 3,3 106912 2736131 /lib/ld-2.3.2.so
    perl 7311 nobody 0r CHR 1,3 67071 /dev/null
    perl 7311 nobody 1w FIFO 0,5 3856953 pipe
    perl 7311 nobody 2w REG 3,3 117149871 49511 /usr/local/apache/logs/error_log
    perl 7311 nobody 3u REG 3,3 0 213610 /tmp/ZCUDeirJT2 (deleted)
    perl 7311 nobody 5u unix 0xe115fac0 845791 socket
    perl 7311 nobody 6u unix 0xc7794580 2764262 socket
    perl 7311 nobody 7u unix 0xe4e9d040 3856940 socket
    perl 7311 nobody 15w REG 3,3 117149871 49511 /usr/local/apache/logs/error_log
     
  12. jwhysleep

    jwhysleep Member

    Joined:
    Aug 8, 2003
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    should i run kill 7311 ?
     
  13. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    I think you are missing the point here. Killing this pid won't solve the issue. You'll need to have the server cleaned of exploits, secured and optimized.
     
Loading...

Share This Page