The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

high load -- few/normal processes?

Discussion in 'General Discussion' started by Jorel, Sep 1, 2005.

  1. Jorel

    Jorel Well-Known Member

    Joined:
    Aug 15, 2003
    Messages:
    45
    Likes Received:
    0
    Trophy Points:
    6
    I have a load of 30 but nothing I see in "top" seems to indicate why. Everything seems pretty normal except the processor load. Any ideas? Thanks.


    15:37:22 up 30 days, 1:58, 1 user, load average: 36.22, 35.82, 28.30
    385 processes: 381 sleeping, 2 running, 1 zombie, 1 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 2.8% 0.0% 0.4% 0.1% 0.0% 96.3% 0.0%
    cpu00 3.7% 0.0% 0.5% 0.0% 0.0% 95.6% 0.0%
    cpu01 1.9% 0.0% 0.3% 0.3% 0.1% 97.0% 0.0%
    Mem: 1026628k av, 1012568k used, 14060k free, 0k shrd, 18588k buff
    766400k actv, 145364k in_d, 15692k in_c
    Swap: 1052248k av, 1008632k used, 43616k free 148628k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    14021 nobody 15 0 21648 15M 2376 S 0.4 1.5 0:00 1 httpd
    18424 mailnull 19 0 3072 2928 1892 S 0.3 0.2 0:00 0 exim
    17243 nobody 15 0 21548 14M 2360 S 0.2 1.4 0:00 0 httpd
    18327 root 15 0 1436 1436 880 R 0.2 0.1 0:00 0 top
    18426 mailnull 16 0 3820 3768 2580 S 0.2 0.3 0:00 0 exim
    3075 named 15 0 48584 40M 880 D 0.1 4.0 4:34 0 named
    17630 nobody 15 0 19072 12M 2288 D 0.1 1.2 0:00 1 httpd
    18267 spoon1 15 0 4392 3100 1004 D 0.1 0.3 0:00 1 cppop
    1 root 15 0 372 372 308 S 0.0 0.0 0:36 0 init
    2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
    3 root RT 0 0 0 0 SW 0.0 0.0 0:00 1 migration/1
    4 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 keventd
    5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    6 root 34 19 0 0 0 SWN 0.0 0.0 0:00 1 ksoftirqd/1
    9 root 15 0 0 0 0 SW 0.0 0.0 0:06 1 bdflush
    7 root 15 0 0 0 0 SW 0.0 0.0 20:49 1 kswapd
    8 root 15 0 0 0 0 SW 0.0 0.0 47:46 1 kscand
    10 root 15 0 0 0 0 SW 0.0 0.0 2:43 0 kupdated
    11 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    18 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_0
    19 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_1
    20 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_2
    21 root 25 0 0 0 0 SW 0.0 0.0 0:00 1 scsi_eh_3
     
  2. Sheldon

    Sheldon Well-Known Member

    Joined:
    Jun 7, 2004
    Messages:
    378
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    http://www.chkrootkit.org

    go download that see if it detects hidden processes, if so youve been hacked. (well atleast theres a good chance.)

    for now kill off anything that you DONT absolutely need running while you look into it. such as httpd and mysql. if you cant gain control, force a reboot then login as soon as it comes back online and shut off apache and mysql.

    download chkrootkit, compile it and run it.

    -Sheldon
     
  3. Sheldon

    Sheldon Well-Known Member

    Joined:
    Jun 7, 2004
    Messages:
    378
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    oh another thing..

    dont rely on top... its useless..

    #ps auxf

    thats more helpful, that way you can see all the child processes running under each parent.
     
  4. Jorel

    Jorel Well-Known Member

    Joined:
    Aug 15, 2003
    Messages:
    45
    Likes Received:
    0
    Trophy Points:
    6
    I ran chkrootkit (I already had it) and nothing was detected. So I tried a reboot. A few minutes after the reboot, the load went down to about .5 (normal for my 1,000-domain dual opteron). Now it's about at about a 2.0, which I guess is okay. Here is the result of the ps auxf:

    USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
    root 3 0.0 0.0 0 0 ? SW 15:41 0:00 [migration/1]
    root 2 0.0 0.0 0 0 ? SW 15:41 0:00 [migration/0]
    root 1 0.5 0.0 1532 504 ? S 15:41 0:04 init
    root 4 0.0 0.0 0 0 ? SW 15:41 0:00 [keventd]
    root 5 0.0 0.0 0 0 ? SWN 15:41 0:00 [ksoftirqd/0]
    root 6 0.0 0.0 0 0 ? SWN 15:41 0:00 [ksoftirqd/1]
    root 9 0.0 0.0 0 0 ? SW 15:41 0:00 [bdflush]
    root 7 0.0 0.0 0 0 ? SW 15:41 0:00 [kswapd]
    root 8 0.0 0.0 0 0 ? SW 15:41 0:00 [kscand]
    root 10 0.1 0.0 0 0 ? SW 15:41 0:00 [kupdated]
    root 11 0.0 0.0 0 0 ? SW 15:41 0:00 [mdrecoveryd]
    root 18 0.0 0.0 0 0 ? SW 15:41 0:00 [scsi_eh_0]
    root 19 0.0 0.0 0 0 ? SW 15:41 0:00 [scsi_eh_1]
    root 20 0.0 0.0 0 0 ? SW 15:41 0:00 [scsi_eh_2]
    root 21 0.0 0.0 0 0 ? SW 15:41 0:00 [scsi_eh_3]
    root 24 0.1 0.0 0 0 ? DW 15:41 0:01 [kjournald]
    root 79 0.0 0.0 0 0 ? SW 15:41 0:00 [khubd]
    root 532 0.0 0.0 0 0 ? SW 15:41 0:00 [kjournald]
    root 619 0.0 0.0 0 0 ? SW 15:41 0:00 [kjournald]
    root 1356 0.0 0.0 1592 580 ? S 15:42 0:00 syslogd -m 0
    root 1360 0.0 0.0 1532 460 ? S 15:42 0:00 klogd -x
    root 1370 0.0 0.0 1528 440 ? S 15:42 0:00 irqbalance
    root 1382 0.0 0.0 83564 600 ? S 15:42 0:00 /sbin/auditd
    named 1440 0.3 0.8 52560 8240 ? S 15:42 0:02 /usr/sbin/named -u named
    root 1454 0.0 0.1 3668 1548 ? S 15:42 0:00 /usr/sbin/sshd
    root 3161 0.0 0.2 7044 2172 ? S 15:49 0:00 \_ sshd: root@pts/0
    root 3168 0.0 0.1 4256 1356 pts/0 S 15:49 0:00 \_ -bash
    root 5403 0.0 0.0 2744 792 pts/0 R 15:56 0:00 \_ ps auxf
    root 1468 0.0 0.0 2148 832 ? S 15:42 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
    rcm 5392 0.3 0.1 2540 1028 ? D 15:56 0:00 \_ imapd
    root 1486 0.0 0.2 6060 2700 ? S 15:42 0:00 chkservd
    root 1547 0.0 0.8 20232 8776 ? S 15:42 0:00 /usr/sbin/clamd
    mailnull 1551 0.0 0.1 6592 1836 ? S 15:42 0:00 /usr/sbin/exim -bd -oX 26
    mailnull 1557 0.0 0.1 6628 1832 ? S 15:42 0:00 /usr/sbin/exim -bd -q60m
    mailnull 5139 0.1 0.4 7716 4344 ? S 15:55 0:00 \_ /usr/sbin/exim -bd -q60m
    mailnull 5178 0.2 0.4 7716 4344 ? S 15:55 0:00 \_ /usr/sbin/exim -bd -q60m
    mailnull 5366 0.5 0.3 7568 3760 ? D 15:56 0:00 \_ /usr/sbin/exim -bd -q60m
    mailnull 1565 0.0 0.1 6588 1796 ? S 15:42 0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
    root 1572 0.0 0.1 2932 1404 ? S 15:42 0:00 antirelayd
    root 1587 0.0 1.9 21916 20088 ? S 15:42 0:00 /usr/bin/spamd -d --allowed-ips=127.0.0.1 --pidfile=/var/run/s
    root 1728 0.0 2.0 22528 20808 ? S 15:42 0:00 \_ spamd child
    root 1731 0.0 2.0 22408 20628 ? S 15:42 0:00 \_ spamd child
    root 1732 0.0 2.0 22420 20684 ? S 15:42 0:00 \_ spamd child
    root 1733 0.0 2.0 22556 20816 ? S 15:42 0:00 \_ spamd child
    root 1735 0.0 2.0 22280 20552 ? S 15:42 0:00 \_ spamd child
    root 1597 0.2 1.8 26480 19112 ? S 15:42 0:02 /usr/local/apache/bin/httpd -DSSL
    nobody 1712 0.3 2.3 31164 24224 ? S 15:42 0:02 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 1720 0.5 2.3 31312 24368 ? S 15:42 0:04 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 2022 0.6 2.3 31520 24492 ? S 15:43 0:05 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 3218 0.1 2.1 29036 22056 ? S 15:49 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5152 1.7 2.3 30816 23860 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5153 1.4 2.2 30444 23500 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5154 0.7 2.2 30396 23444 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5155 0.8 2.1 29416 22480 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5156 0.6 2.2 29868 22772 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5160 0.7 2.1 29216 22264 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5161 0.3 2.3 32912 23820 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5164 0.5 2.1 29024 22068 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5168 1.2 2.2 30508 23556 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5169 0.4 2.0 28480 21504 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5170 1.4 2.3 30744 23772 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5181 0.5 2.2 30336 23396 ? S 15:55 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5232 0.0 1.8 26484 19196 ? S 15:56 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5297 2.4 2.1 29016 22052 ? S 15:56 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5386 4.0 2.1 29068 22084 ? S 15:56 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    nobody 5401 0.0 1.8 26480 19120 ? S 15:56 0:00 \_ /usr/local/apache/bin/httpd -DSSL
    root 1611 0.0 0.0 1604 648 ? S 15:42 0:00 crond
    root 1740 0.0 0.6 10228 6796 ? S 15:42 0:00 cppop - accepting on port 110
    root 1763 0.0 0.5 7180 5412 ? SN 15:42 0:00 /usr/bin/perl /usr/local/cpanel/cpanellogd
    root 1778 0.0 0.6 8168 6736 ? S 15:42 0:00 cppop - accepting on port 110
    nobody 1782 0.0 0.1 3392 2032 ? S 15:42 0:00 entropychat
    nobody 1786 0.0 0.0 1728 612 ? S 15:42 0:00 /usr/local/cpanel/bin/startmelange
    cpanel 1809 0.0 0.1 3684 1628 ? S 15:42 0:00 /usr/bin/stunnel-4.04local /usr/local/cpanel/etc/stunnel/defau
    root 1831 0.0 0.0 1528 488 ? S 15:42 0:00 /usr/sbin/portsentry -tcp
    root 1850 0.0 0.0 0 0 ? SW 15:42 0:00 [loop0]
    root 1851 0.0 0.7 11596 8156 ? S 15:42 0:00 cpsrvd - waiting for connections
    lemonla 5099 0.0 0.8 11984 8408 ? S 15:55 0:00 \_ cpaneld - serving 83.160.19.223
    lemonla 5100 0.1 1.3 23708 14200 ? S 15:55 0:00 | \_ /usr/local/cpanel/cpanel ./frontend/x/subdomain/doaddd
    root 5104 0.0 0.0 1544 552 ? S 15:55 0:00 | \_ /usr/local/cpanel/bin/domainwrap ADD bbs heart-thr
    root 5105 0.3 0.4 5672 4180 ? S 15:55 0:00 | \_ /usr/local/cpanel/bin/domainadmin
    root 5134 0.5 0.2 3520 2168 ? D 15:55 0:00 | \_ /usr/bin/perl /scripts/convertemails --qui
    cpanel 5387 1.0 0.8 11612 8256 ? S 15:56 0:00 \_ cpaneld - serving 66.169.140.25
    cpanel 5388 7.6 1.0 17504 10400 ? S 15:56 0:00 \_ /usr/local/cpanel/3rdparty/bin/php /usr/local/cpanel/b
    root 1860 0.0 0.0 0 0 ? SW 15:42 0:00 [kjournald]
    root 1866 0.0 0.0 1520 420 tty1 S 15:42 0:00 /sbin/mingetty tty1
    root 1867 0.0 0.0 1524 420 tty2 S 15:42 0:00 /sbin/mingetty tty2
    root 1868 0.0 0.0 1512 420 tty3 S 15:42 0:00 /sbin/mingetty tty3
    root 1869 0.0 0.0 1508 420 tty4 S 15:42 0:00 /sbin/mingetty tty4
    root 1870 0.0 0.0 1512 416 tty5 S 15:42 0:00 /sbin/mingetty tty5
    root 1871 0.0 0.0 1512 420 tty6 S 15:42 0:00 /sbin/mingetty tty6
    root 1921 0.0 0.1 6056 1816 ? S 15:43 0:00 pure-ftpd (SERVER)
    lightup 3369 0.0 0.1 6076 1984 ? S 15:50 0:00 \_ pure-ftpd (IDLE)
    holliday 4170 0.0 0.1 6076 1992 ? S 15:51 0:00 \_ pure-ftpd (IDLE)
    root 1925 0.0 0.0 5588 968 ? S 15:43 0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureau
    root 2054 0.0 0.1 4248 1164 ? S 15:43 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-fi
    mysql 2075 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --us
    mysql 2077 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql
    mysql 2078 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
    mysql 2079 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
    mysql 2080 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
    mysql 2081 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
    mysql 2084 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/my
    mysql 2085 0.0 2.2 36852 22920 ? S 15:43 0:00 \_ /usr/sbin/mysqld --
     
  5. Sheldon

    Sheldon Well-Known Member

    Joined:
    Jun 7, 2004
    Messages:
    378
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Canada
    Try turning off uneccesary services in cpanel such as melange and entropychat. I know they can eat server resources like nothing.

    Next best thing would be to download and install latest version of rkhunter and run that.

    For me to tell you much more id have to sit there and babysit it for awhile to know whats going on. Also make sure you werent running any important high level crons during that time, such as cpanel backups.

    -Sheldon
     
  6. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    There are many variables involved in server high load. To mention a few, you need to track down the resources eater including insecure/bad script and UDP/TCP attaks. Upgrade and then optimize your server services and applications.
     
  7. Myacen

    Myacen Well-Known Member

    Joined:
    Apr 6, 2002
    Messages:
    222
    Likes Received:
    0
    Trophy Points:
    16
    Your using a ton of swap, put some extra ram in the machine and you should be fine.
     
  8. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Indeed. As Myacen ha said, the server is showing the classic symptoms of memory thrashing, where there's too little physical RAM available for the servers needs. It's evident from the huge swap usage and the high IOWAIT %age. It's possible that you may be able to do some tuning to relieve that load (presuming that nothing has a memory leak) with mysql and apache and spamd which are the usual culprits. If not, then only more RAM will help.
     
  9. Jorel

    Jorel Well-Known Member

    Joined:
    Aug 15, 2003
    Messages:
    45
    Likes Received:
    0
    Trophy Points:
    6
    Thanks to everyone for the advice. I will try to track down unnecisary processes and such. I will also try and scrounge up the cash for a RAM upgrade. I mean, what's the point of a Dual Opteron 244 box when you only have 1GB of RAM? Seems like all the pros get at least 4GB and anywhere up to 16GB of RAM for their Dual Opteron machines. I'm leasing a server though, so unfortunately I don't have the benefit of buying cheap RAM for it on my own.
     
Loading...

Share This Page