The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server Load Is High

Discussion in 'General Discussion' started by markerpower, Jan 21, 2007.

  1. markerpower

    markerpower Member

    Joined:
    Mar 16, 2005
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    My server load is high, and I need help to lower it. It is usually below 1, but since yesterday it has been above 4. For the past 12 hours I've been getting this email:
    IMPORTANT: Do not ignore this email.
    I've also ran the following command that I found by searching through the forums:

    top c
    Also in WHM under Statistics Software Configuration it gives me the following error:
    The server is having trouble keeping up with your statistics processing schedule. You should increase the time between statistic generation, or upgrade the server. If you have recently decreased the time between statistic generation, you may wish to wait that amount of time to see if the server will catch up before changing back.

    Also "Delete each domain's access logs after stats run" has always been checked.


    If anyone can lead me to the right direction or a topic that has the solution please reply.
     
    #1 markerpower, Jan 21, 2007
    Last edited: Jan 21, 2007
  2. mohit

    mohit Well-Known Member

    Joined:
    Jul 12, 2005
    Messages:
    553
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sticky On Internet
    hi,
    you can always raise the limit for stats programs from tweak settings--->

    Code:
    The load average above the number of cpus at which logs file processing should be suspended (default 0):
    make it 5 or 10.


    Also i think you have some memory consuming process running, you can try to check whats costing you so much cpu and ram on this box.
    check apache status, what is being served currently.
    you can also check WHM--> System Health --> Show Current CPU Usage or use
    Top from ssh and check whats using up CPU.
    (its sunday it might be your weekly backup running, as most people run them on sundays)


    seeya,
    mohit
     
  3. markerpower

    markerpower Member

    Joined:
    Mar 16, 2005
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Ok I've waited a few days to see if the overload would decrease. Now it is over 10, and I know something is wrong. Stats aren't processing. Can someone give me some advice to figure out what is going on?

    Here is what I get for CPU Usage:
    Code:
    Pid Owner Priority Cpu % Mem % Command 
    13465 nobody 0  8.1  0.1 perl udp.txt 84.244.0.15 0 0  
    17617 nobody 0  7.5  0.1 perl udp.txt 66.252.30.37 22 150  
    9396 nobody 0  6.5  0.1 perl udp.txt 69.9.180.186 3522 0  
    9485 nobody 0  6.1  0.1 perl udp.txt 69.9.180.186 3522 0  
    7413 nobody 0  1.9  1.2 /usr/local/apache/bin/httpd -DSSL  
    12117 nobody 0  1.7  2.0 /usr/local/apache/bin/httpd -DSSL  
    12790 nobody 0  1.3  2.1 /usr/local/apache/bin/httpd -DSSL  
    7433 nobody 0  1.1  1.0 /usr/local/apache/bin/httpd -DSSL  
    7499 nobody 0  1.1  1.0 /usr/local/apache/bin/httpd -DSSL  
    7877 root 0  0.3  0.1 0 top -n 2 -b - 
    1 root 0  0.0  0.0 init [3]  
    2 root 0  0.0  0.0 0 keventd 
    3 root 0  0.0  0.0 0 kapmd 
    4 root 19  0.0  0.0 0 ksoftirqd/0 
    7 root 0  0.0  0.0 0 bdflush 
    5 root 0  0.0  0.0 0 kswapd 
    6 root 0  0.0  0.0 0 kscand 
    8 root 0  0.0  0.0 0 kupdated 
    9 root 0  0.0  0.0 0 mdrecoveryd 
    13 root 0  0.0  0.0 0 kjournald 
    68 root 0  0.0  0.0 0 khubd 
    2241 root 0  0.0  0.0 0 kjournald 
    2242 root 0  0.0  0.0 0 kjournald 
    2243 root 0  0.0  0.0 0 kjournald 
    2244 root 0  0.0  0.0 0 kjournald 
    2245 root 0  0.0  0.0 0 kjournald 
    2656 root 0  0.0  0.0 syslogd -m 0  
    2660 root 0  0.0  0.0 klogd -x  
    2794 root 0  0.0  0.0 xinetd -stayalive -pidfile /var/run/xinetd.pid  
    2942 bin 0  0.0  0.0 /usr/sbin/cannaserver -syslog -u bin  
    2980 wnn 0  0.0  0.0 /usr/bin/jserver  
    3067 xfs 0  0.0  0.0 xfs -droppriv -daemon  
    3303 nobody 0  0.0  0.0 entropychat  
    3309 nobody 0  0.0  0.0 /usr/local/cpanel/bin/startmelange  
    3384 root 0  0.0  0.2 cppop - accepting on port 110 
    3397 mailman 0  0.0  0.0 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/mailmanctl -s start  
    3407 mailman 0  0.0  0.1 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=ArchRunner:0:1 -s  
    3408 mailman 0  0.0  0.1 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=BounceRunner:0:1 -s  
    3411 mailman 0  0.0  0.0 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=CommandRunner:0:1 -s  
    3412 mailman 0  0.0  0.1 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s  
    3413 mailman 0  0.0  0.1 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=NewsRunner:0:1 -s  
    3414 mailman 0  0.0  0.1 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s  
    3418 mailman 0  0.0  0.0 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=VirginRunner:0:1 -s  
    3419 mailman 0  0.0  0.0 /usr/bin/python /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=RetryRunner:0:1 -s  
    3434 root 0  0.0  0.0 /usr/sbin/portsentry -tcp  
    3456 root 0  0.0  0.0 /sbin/mingetty tty2  
    3457 root 0  0.0  0.0 /sbin/mingetty tty3  
    3458 root 0  0.0  0.0 /sbin/mingetty tty4  
    3459 root 0  0.0  0.0 /sbin/mingetty tty5  
    3460 root 0  0.0  0.0 /sbin/mingetty tty6  
    3461 root 0  0.0  0.0 /sbin/mingetty ttyS0 CON9600 vt102  
    3980 mailnull 0  0.0  0.1 /usr/bin/perl /usr/local/cpanel/bin/eximstats  
    15112 lctrumpe 0  0.0  0.0 /usr/bin/php index.php  
    29835 root 0  0.0  0.0 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid  
    29855 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29856 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29857 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29858 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29859 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29860 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29861 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29862 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29863 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    29864 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    30223 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    12704 root 0  0.0  0.0 /sbin/mingetty tty1  
    10500 root 0  0.0  0.1 chkservd  
    10679 root 0  0.0  3.1 /usr/sbin/clamd  
    10685 mailnull 0  0.0  0.1 /usr/sbin/exim -bd -q60m  
    10691 mailnull 0  0.0  0.1 /usr/sbin/exim -tls-on-connect -bd -oX 465  
    10772 root 0  0.0  0.0 antirelayd  
    10795 root 0  0.0  0.9 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/spamd.pid --max-children=5  
    421 named 0  0.0  0.1 /usr/sbin/named -u named  
    422 named 0  0.0  0.1 /usr/sbin/named -u named  
    423 named 0  0.0  0.1 /usr/sbin/named -u named  
    424 named 0  0.0  0.1 /usr/sbin/named -u named  
    26303 root 0  0.0  0.0 crond  
    27268 root 0  0.0  0.0 mdadm --monitor --scan -f  
    31606 root 0  0.0  0.0 rhnsd --interval 240  
    14279 root 0  0.0  0.0 /usr/sbin/sshd  
    6807 root 0  0.0  0.4 /usr/local/apache/bin/httpd -DSSL  
    24846 nobody 0  0.0  0.1 proftpd: (accepting connections) 
    31656 root 0  0.0  2.2 spamd child  
    29029 cpanel 0  0.0  0.0 /usr/bin/stunnel-4.04local /usr/local/cpanel/etc/stunnel/default/stunnel.conf.run  
    29059 root 0  0.0  0.2 cpsrvd - waiting for connections 
    7737 root 0  0.0  0.0 crond  
    7738 root 0  0.0  0.2 cPanel Update (upcp) - Master 
    7740 root 0  0.0  0.2 cPanel Update (upcp) - Slave 
    7766 mailnull 0  0.0  0.0 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t  
    7865 clamav 0  0.0  0.0 /usr/bin/freshclam --quiet -l /var/log/clam-update.log  
    26125 nobody 0  0.0  0.5 /usr/local/apache/bin/httpd -DSSL  
    26695 nobody 0  0.0  0.0 0 sh  
    8204 nobody 0  0.0  0.1 /usr/local/apache/bin/httpd -DSSL  
    8215 nobody 0  0.0  0.1 lpd  
    30295 nobody 0  0.0  0.3 /usr/local/apache/bin/httpd -DSSL  
    30341 nobody 0  0.0  0.6 /usr/local/apache/bin/httpd -DSSL  
    31481 nobody 0  0.0  0.8 /usr/local/apache/bin/httpd -DSSL  
    12113 root 0  0.0  0.1 /usr/bin/perl /usr/local/cpanel/bin/leechprotect  
    12114 nobody 0  0.0  2.3 /usr/local/apache/bin/httpd -DSSL  
    12115 nobody 0  0.0  2.2 /usr/local/apache/bin/httpd -DSSL  
    12116 nobody 0  0.0  2.1 /usr/local/apache/bin/httpd -DSSL  
    12131 nobody 0  0.0  1.8 /usr/local/apache/bin/httpd -DSSL  
    12135 nobody 0  0.0  1.8 /usr/local/apache/bin/httpd -DSSL  
    12137 nobody 0  0.0  2.3 /usr/local/apache/bin/httpd -DSSL  
    17184 nobody 0  0.0  0.9 /usr/local/apache/bin/httpd -DSSL  
    23193 nobody 0  0.0  1.5 /usr/local/apache/bin/httpd -DSSL  
    31879 root 0  0.0  0.9 spamd child  
    7779 nobody 0  0.0  1.8 /usr/local/apache/bin/httpd -DSSL  
    12869 root 0  0.0  0.1 cupsd  
    7772 nobody 0  0.0  1.7 /usr/local/apache/bin/httpd -DSSL  
    8862 nobody 0  0.0  0.4 /usr/local/apache/bin/httpd -DSSL  
    9395 nobody 0  0.0  0.0 sh -c cd /tmp ; perl udp.txt 69.9.180.186 3522 0  
    11052 nobody 0  0.0  1.7 /usr/local/apache/bin/httpd -DSSL  
    30501 root 0  0.0  0.0 crond  
    30502 root 0  0.0  0.0 /bin/bash /usr/bin/run-parts /etc/cron.daily  
    30786 mailnull 0  0.0  0.1 /usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t  
    31069 root 0  0.0  0.0 awk -v progname=/etc/cron.daily/slocate.cron progname { 
    7454 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    7591 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    7819 mysql 0  0.0  1.3 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/greenup.servermatrix.com.pid --skip-locking  
    7864 root 0  0.0  0.4 whostmgrd - serving 130.70.13.40 
    7867 root 0  0.0  3.1 /usr/sbin/clamd  
    7874 root 0  0.0  1.9 /usr/local/cpanel/whostmgr/bin/whostmgr2 ./top  
    
    It appears that nobody is using alot of cpu usage, but I can't tell what nobody is.
     
  4. maggy

    maggy Active Member

    Joined:
    Jun 9, 2004
    Messages:
    25
    Likes Received:
    0
    Trophy Points:
    1
    kill those perl processes on the top of the list, thats whats using all your cpu
    then you can search for the file (udp.txt) and see who it belongs to and whats going on with it
     
  5. markerpower

    markerpower Member

    Joined:
    Mar 16, 2005
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Is it ok to kill that process? udp.txt is part of udp.7.gz, which is a User Datagram Protocol or so google says. I got a pm from someone here stating that my server has been hacked. I just can't find out what the problem is. Under CPU/Memory/MySQL Usage it shows nobody using
    123.78 %CPU 1.51%MEM 0.0 Mysql Processes.
     
  6. nwilkens

    nwilkens Well-Known Member

    Joined:
    May 4, 2006
    Messages:
    59
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Monroe MI
    cPanel Access Level:
    DataCenter Provider
    Server compromised

    Your server has likely been compromised.

    udp.txt is probably a denial of service Perl UDP Flood by Odix. This is a < 60 line perl script that sends IP protocol 17 (UDP) packets to either a specific destination or random port between 1 and 65000.

    You need to take immediate action to resolve this issue. I am sure you can find plenty of information on this board and others on the proper next steps to secure / restore a hacked server.
     
  7. mctDarren

    mctDarren Well-Known Member

    Joined:
    Jan 6, 2004
    Messages:
    664
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    New Jersey
    cPanel Access Level:
    Root Administrator
    Yepper - you need to hire a professional admin. But fast. Look up configserver.com, totalserverssolutions.com or tweakservers.com and get one of them to help you out. Good luck.
     
  8. nwilkens

    nwilkens Well-Known Member

    Joined:
    May 4, 2006
    Messages:
    59
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Monroe MI
    cPanel Access Level:
    DataCenter Provider
    Also,

    start with ,

    ps -ef|grep udp.txt|grep -v grep|awk '{print $1}'

    you will get an output with process id numbers of udp.txt.

    go to /proc/PROCESS ID from above/

    and ls -l cwd

    This will show you the current working directory of the program.. go to the directory save this program for further analysis or remove it.

    Check what else is in the directory, and start figuring out how to block the hacker.

    Are you running phpBB, if so which version?

    Sorry for the quick, hopefully somewhat readable reply..
     
  9. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    We have seen these files on many servers in the past and they are either IRC connection or possible BotNet. It is really hard to say that your server has been comprmoised from the output posted in this thread. In addition to what nwilkens said, do the following:
    ls -alh /tmp
    ls -alh /dev/shm/

    Remove the permissions and change the ownership of these suspecious files to root to render them useless. Please remove these files when your investigation is complete.

    Also disable the user's account that is connecting to the IRC covertly.
    /scripts/suspendacct USER

    You cal also use the command: lsof -p PID to see who is running the process.

    Hope this helps!
     
Loading...

Share This Page