The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

server load 160, what could be the reason?

Discussion in 'General Discussion' started by shx, Feb 13, 2004.

  1. shx

    shx Member

    Joined:
    Feb 13, 2004
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    Below is what I see in ssh when I enter "top" command to see server load. Could anyone give a suggestion to resolve the problem. I am not good in linux, I have many times tried to find the problem but I couldnt :(

    Thanks for any help and sorry for my bad english


    Server info:
    rh9, cpanel
    p4 2gb, 512 ram


    ### ssh top
    18:15:34 up 4 days, 19:11, 1 user, load average: 159.64, 130.58, 68.19
    815 processes: 810 sleeping, 5 running, 0 zombie, 0 stopped
    CPU states: 10.1% user 2.8% system 0.0% nice 0.0% iowait 86.8% idle
    Mem: 505716k av, 500296k used, 5420k free, 0k shrd, 4768k buff
    316508k actv, 68084k in_d, 10700k in_c
    Swap: 2048276k av, 746692k used, 1301584k free 33204k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    14392 root 16 0 1760 1760 860 R 6.3 0.3 0:00 0 top
    5960 root 15 0 1960 232 172 S 0.9 0.0 5:15 0 httpd
    1 root 15 0 108 80 56 S 0.0 0.0 0:07 0 init
    2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
    3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd
    4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd_CPU
    9 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
    5 root 15 0 0 0 0 RW 0.0 0.0 0:18 0 kswapd
    6 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand/DMA
    7 root 16 0 0 0 0 SW 0.0 0.0 5:49 0 kscand/Normal
    8 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand/HighMe
    10 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
    11 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    15 root 15 0 0 0 0 DW 0.0 0.0 1:27 0 kjournald
    73 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    2797 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
     
  2. TheVoice

    TheVoice Well-Known Member

    Joined:
    Feb 7, 2002
    Messages:
    105
    Likes Received:
    0
    Trophy Points:
    16
    Restart apache
     
  3. shx

    shx Member

    Joined:
    Feb 13, 2004
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    thankx, I had done it
    but I have this problem everyday, so I must find exactly the reason and correct it. Otherwise my servers uptime will be 50%

    .. and I think there is no real difference between 50% uptime and dead server.

    any suggestion or experience?
     
  4. M@jor.P@yne

    M@jor.P@yne Member

    Joined:
    Feb 9, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    have you installed any new applications/ updates of sofware etc.

    i have heard once before of a host that was having very simmlar readings and it was somthing to do with mysql and some user being a n00b maybe restart mysql or something like that, whilst you look at CPU useage, if you see it drop drematicly then you may have found your problem.

    i'm still new at this so dont take as if i have done it ;) just a though i had and had to share it

    edit: also it maybe due to the fact you have: 815 processes: 810 sleeping if they all said howdie yall and wanted CPU time humm
     
    #4 M@jor.P@yne, Feb 14, 2004
    Last edited: Feb 14, 2004
  5. M@jor.P@yne

    M@jor.P@yne Member

    Joined:
    Feb 9, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    14:02:28 up 39 days, 9:01, 1 user, load average: 0.07, 0.25, 0.25
    151 processes: 149 sleeping, 2 running, 0 zombie, 0 stopped
    CPU states: 1.1% user 0.3% system 0.0% nice 0.0% iowait 98.4% idle
    Mem: 902772k av, 865924k used, 36848k free, 0k shrd, 24468k buff
    545440k actv, 89376k in_d, 17676k in_c
    Swap: 2097136k av, 372352k used, 1724784k free 484828k cached
     
  6. shx

    shx Member

    Joined:
    Feb 13, 2004
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    thankx Payne but I think I have virus or trojan...

    I have found more than 2000 .wrk datas in /tmp

    and all of them take nearly 40 GB, I have deleted but more.. I dont know too
     
  7. dgtech

    dgtech Member

    Joined:
    Apr 5, 2002
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    I have the exact same problem. Look at how high the IOWAIT is, that is your problem. I am trying to find the resultion myself.

    Thanks and God Bless!
     
  8. M@jor.P@yne

    M@jor.P@yne Member

    Joined:
    Feb 9, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
  9. techark

    techark Well-Known Member

    Joined:
    May 22, 2002
    Messages:
    280
    Likes Received:
    0
    Trophy Points:
    16
    Geez look at the number of processes running 810 .

    ps -aux and find out what is running so many processes.
    That and you are banging the heck out of swap.
     
  10. dgtech

    dgtech Member

    Joined:
    Apr 5, 2002
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Here is my output on TOP:

    PHP:
     09:41:45  up  8:03,  1 user,  load average570.25322.93204.25
    1902 processes
    1901 sleeping1 running0 zombie0 stopped
    CPU states
    :  cpu    user    nice  system    irq  softirq  iowait    idle
               total    1.9
    %    0.0%    4.5%   0.2%     1.2%   92.1%    0.0%
               
    cpu00    2.6%    0.0%    4.3%   0.0%     0.1%   92.8%    0.0%
               
    cpu01    1.2%    0.0%    4.6%   0.5%     2.2%   91.3%    0.0%
    Mem:   495776k av,  477540k used,   18236k free,       0k shrd,    4552k buff
                        283188k actv
    ,   31008k in_d,    5284k in_c
    Swap
    1052248k av,  526008k used,  526240k free                   74404k cached 
     
  11. techark

    techark Well-Known Member

    Joined:
    May 22, 2002
    Messages:
    280
    Likes Received:
    0
    Trophy Points:
    16
    1902 processes: 1901 sleeping, 1 running, 0 zombie, 0 stopped


    Geez I am surprised your box has not had a meltdown yet.

    Why is there 1902 processes running? ps -aux to find out
     
  12. WCW Fan

    WCW Fan Well-Known Member

    Joined:
    Sep 22, 2003
    Messages:
    68
    Likes Received:
    0
    Trophy Points:
    6
    my goodness that is alot of processes running, :eek:
     
  13. dgtech

    dgtech Member

    Joined:
    Apr 5, 2002
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    Here is the current output, however only one is running out of 106.

    PHP:
    ps -aux
    USER       PID 
    %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
    root         1  0.0  0.0  1524  468 
    ?        S    01:37   0:38 init
    root         2  0.0  0.0     0    0 
    ?        SW   01:37   0:00 [migration/0]
    root         3  0.0  0.0     0    0 ?        SW   01:37   0:00 [migration/1]
    root         4  0.0  0.0     0    0 ?        SW   01:37   0:00 [keventd]
    root         5  0.0  0.0     0    0 ?        SWN  01:37   0:00 [ksoftirqd/0]
    root         6  0.0  0.0     0    0 ?        SWN  01:37   0:00 [ksoftirqd/1]
    root         9  0.0  0.0     0    0 ?        SW   01:37   0:00 [bdflush]
    root         7  0.0  0.0     0    0 ?        SW   01:37   0:34 [kswapd]
    root         8  0.0  0.0     0    0 ?        SW   01:37   0:32 [kscand]
    root        10  0.0  0.0     0    0 ?        SW   01:37   0:01 [kupdated]
    root        11  0.0  0.0     0    0 ?        SW   01:37   0:00 [mdrecoveryd]
    root        18  0.0  0.0     0    0 ?        SW   01:37   0:00 [katad-1]
    root        20  0.0  0.0     0    0 ?        SW   01:37   0:00 [scsi_eh_0]
    root        21  0.0  0.0     0    0 ?        SW   01:37   0:00 [scsi_eh_1]
    root        24  0.7  0.0     0    0 ?        SW   01:37   5:15 [kjournald]
    root        80  0.0  0.0     0    0 ?        SW   01:37   0:00 [khubd]
    root       598  0.0  0.0     0    0 ?        SW   01:37   0:00 [kjournald]
    root      3961  0.0  0.0  1588  460 ?        S    01:38   0:00 syslogd -m 0
    root      3965  0.0  0.0  1508  324 
    ?        S    01:38   0:00 klogd -x
    root      3975  0.0  0.0  1520  412 
    ?        S    01:38   0:02 irqbalance
    root      6111  0.0  0.1  7568  692 
    ?        S    01:38   0:00 cupsd
    root      6148  0.0  0.1  3556  584 
    ?        S    01:38   0:00 /usr/sbin/sshd
    root      6161  0.0  0.1  2144  624 
    ?        S    01:38   0:00 xinetd -stayalive
    root      6170  0.0  0.1  4896  748 
    ?        S    01:38   0:02 antirelayd
    root      6180  0.0  0.3  6516 1512 
    ?        S    01:38   0:01 chkservd
    mailnull  6193  0.0  0.1  4468  588 
    ?        S    01:38   0:00 /usr/sbin/exim -b
    mailnull  6198  0.0  0.0  4468  340 
    ?        S    01:38   0:00 /usr/sbin/exim -t
    root      6202  0.0  0.1  2836  744 
    ?        S    01:38   0:02 antirelayd
    root      6211  0.0  0.0  1572  436 
    ?        S    01:38   0:00 crond
    xfs       6248  0.0  0.0  5092  388 
    ?        S    01:38   0:00 xfs -droppriv -da
    root      6405  0.0  0.4  6872 2360 
    ?        SN   01:38   0:00 cpanellogd slee
    root      6407  0.0  0.0  6080  320 
    ?        S    01:38   0:00 cpaneld listeni
    root      6417  0.0  0.1  8980  516 
    ?        S    01:38   0:00 /usr/local/apache
    root      6433  0.0  0.1  4712  608 
    ?        S    01:38   0:00 webmaild
    nobody    6453  0.0  0.2  8980 1332 
    ?        S    01:38   0:00 /usr/local/apache
    nobody    6456  0.0  0.3  8980 1780 
    ?        S    01:38   0:00 /usr/local/apache
    nobody    6458  0.0  0.3  8980 1740 
    ?        S    01:38   0:00 /usr/local/apache
    nobody    6460  0.0  0.3  8980 1624 
    ?        S    01:38   0:00 /usr/local/apache
    nobody    6462  0.0  0.3  8980 1748 
    ?        S    01:38   0:00 /usr/local/apache
    named     6536  0.2  2.3 55932 11840 
    ?       S    01:38   2:05 /usr/sbin/named -
    cpanel    6549  0.0  0.0 14056  408 ?        S    01:38   0:00 /usr/bin/stunnel-
    root      6568  0.0  0.0  5040  404 ?        S    01:38   0:00 whostmgrd
    root      6582  0.0  0.0  3548  320 
    ?        S    01:38   0:00 rhnsd --interval
    root      6598  0.0  0.3  7048 1692 
    ?        S    01:38   0:00 /usr/bin/perl /us
    root      6622  0.0  0.0  1532  220 
    ?        S    01:38   0:00 /usr/sbin/portsen
    root      6653  0.0  0.0  3452  384 
    ?        S    01:38   0:00 /usr/local/urchin
    nobody    6654  0.0  0.0  3452  324 
    ?        S    01:38   0:00 /usr/local/urchin
    nobody    6655  0.0  0.0  3452  324 
    ?        S    01:38   0:00 /usr/local/urchin
    nobody    6656  0.0  0.0  3452  324 
    ?        S    01:38   0:00 /usr/local/urchin
    nobody    6657  0.0  0.0  3452  324 
    ?        S    01:38   0:00 /usr/local/urchin
    nobody    6658  0.0  0.0  3452  324 
    ?        S    01:38   0:00 /usr/local/urchin
    nobody    6660  0.0  0.0   664  124 
    ?        S    01:38   0:02 /usr/local/urchin
    root      6675  0.0  2.4 12152 12148 
    ?       SL   01:38   0:00 mdmpd
    root      6683  0.0  0.0  1500  200 tty2     S    01
    :38   0:00 /sbin/mingetty tt
    root      6684  0.0  0.0  1504  200 tty3     S    01
    :38   0:00 /sbin/mingetty tt
    root      6685  0.0  0.0  1504  200 tty4     S    01
    :38   0:00 /sbin/mingetty tt
    root      6686  0.0  0.0  1512  200 tty5     S    01
    :38   0:00 /sbin/mingetty tt
    root      6687  0.0  0.0  1500  200 tty6     S    01
    :38   0:00 /sbin/mingetty tt
    root      6895  0.0  0.1  4116  552 
    ?        S    01:38   0:00 pure-ftpd (SERVER
    root      6902  0.0  0.0  3744  232 
    ?        S    01:38   0:00 /usr/sbin/pure-au
    root      6944  0.0  0.0  2192  316 
    ?        S    01:39   0:00 /bin/sh /usr/bin/
    mysql     6963  0.0  2.6 46748 13000 ?       S    01:39   0:01 /usr/sbin/mysqld
    mysql     6964  0.0  2.6 46748 13000 
    ?       S    01:39   0:01 /usr/sbin/mysqld
    mysql     6965  0.0  2.6 46748 13000 
    ?       S    01:39   0:00 /usr/sbin/mysqld
    mysql     6966  0.0  2.6 46748 13000 
    ?       S    01:39   0:00 /usr/sbin/mysqld
    mysql     6967  0.0  2.6 46748 13000 
    ?       S    01:39   0:00 /usr/sbin/mysqld
    mysql     6968  0.0  2.6 46748 13000 
    ?       S    01:39   0:00 /usr/sbin/mysqld
    mysql     6969  0.0  2.6 46748 13000 
    ?       S    01:39   0:00 /usr/sbin/mysqld
    mysql     6970  0.0  2.6 46748 13000 
    ?       S    01:39   0:05 /usr/sbin/mysqld
    mysql     6971  0.0  2.6 46748 13000 
    ?       S    01:39   0:00 /usr/sbin/mysqld
    mysql     6972  0.0  2.6 46748 13000 
    ?       S    01:39   0:00 /usr/sbin/mysqld
    nobody    7146  0.0  0.3  8980 1624 
    ?        S    01:43   0:00 /usr/local/apache
    root      7272  0.1  0.2  7644 1244 
    ?        S    01:47   0:54 /usr/bin/perl /us
    mysql     7273  0.1  2.6 46748 13000 
    ?       S    01:47   1:19 /usr/sbin/mysqld
    root      7498  0.0  0.0  1508  200 tty1     S    01
    :49   0:00 /sbin/mingetty tt
    nobody   23748  0.0  0.3  8980 1808 
    ?        S    08:30   0:00 /usr/local/apache
    nobody     878  0.0  0.2  8980 1252 
    ?        S    08:46   0:00 /usr/local/apache
    nobody   14418  0.0  0.3  8980 1740 
    ?        S    09:24   0:00 /usr/local/apache
    nobody   14419  0.0  0.3  8980 1748 
    ?        S    09:24   0:00 /usr/local/apache
    root      3834  0.1  0.1  4872  824 
    ?        S    12:28   0:05 /usr/sbin/exim -q
    root      6866  0.0  0.2  5004 1472 
    ?        S    12:48   0:01 /usr/sbin/exim -q
    root      5133  0.0  0.3  5132 1644 
    ?        S    13:13   0:00 /usr/sbin/exim -q
    root     17577  0.0  0.3  5132 1732 
    ?        S    13:18   0:00 /usr/sbin/exim -q
    root     24642  0.0  0.4  5132 2080 
    ?        S    13:23   0:00 /usr/sbin/exim -q
    root     26125  0.0  0.5  5288 2668 
    ?        S    13:24   0:00 /usr/sbin/exim -q
    mailnull 26132  0.0  0.5  5292 2760 
    ?        S    13:24   0:00 /usr/sbin/exim -q
    root     30315  0.0  0.4  5140 2072 
    ?        S    13:28   0:00 /usr/sbin/exim -q
    root       304  0.0  0.4  5132 2072 
    ?        S    13:33   0:00 /usr/sbin/exim -q
    root      1112  0.0  0.6  5412 3052 
    ?        S    13:34   0:00 /usr/sbin/exim -q
    mailnull  1114  0.0  0.6  5416 3084 
    ?        S    13:34   0:00 /usr/sbin/exim -q
    root      1279  0.0  0.5  5412 2788 
    ?        S    13:34   0:00 /usr/sbin/exim -q
    mailnull  1281  0.0  0.5  5416 2864 
    ?        S    13:34   0:00 /usr/sbin/exim -q
    root      1943  0.0  0.4  5160 2128 
    ?        S    13:34   0:00 /usr/sbin/exim -q
    mailnull  1944  0.0  0.4  5164 2212 
    ?        S    13:34   0:00 /usr/sbin/exim -q
    root      1964  0.0  0.3  6760 1708 
    ?        S    13:35   0:00 sshdsponder [pr
    sponder   1967  0.0  0.3  6772 1964 
    ?        S    13:35   0:00 sshdsponder@pts
    sponder   1978  0.0  0.2  4272 1340 pts
    /0    S    13:35   0:00 -bash
    root      2012  0.0  0.1  4212  968 pts
    /0    S    13:35   0:00 su -
    root      2013  0.0  0.2  4272 1356 pts/0    S    13:35   0:00 -bash
    root      2157  0.0  0.5  5416 2848 
    ?        S    13:35   0:00 /usr/sbin/exim -q
    mailnull  2158  0.0  0.5  5420 2892 
    ?        S    13:35   0:00 /usr/sbin/exim -q
    root      2189  0.0  0.6  5416 3060 
    ?        S    13:35   0:00 /usr/sbin/exim -q
    mailnull  2190  0.0  0.6  5420 3092 
    ?        S    13:35   0:00 /usr/sbin/exim -q
    root      2193  0.0  0.6  5420 3052 
    ?        S    13:35   0:00 /usr/sbin/exim -q
    mailnull  2194  0.0  0.6  5424 3084 
    ?        S    13:35   0:00 /usr/sbin/exim -q
    root      2201  0.0  0.1  2732  788 pts
    /0    R    13:35   0:00 ps -aux
     
     
  14. shx

    shx Member

    Joined:
    Feb 13, 2004
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    with ps -aux I have found something:

    1 mysql process uses 40% of cpu
    2 httpd process use 38% of cpu

    I have more than 20 sites on server. How can I find which user-script-webpage use most of cpu?
     
  15. techark

    techark Well-Known Member

    Joined:
    May 22, 2002
    Messages:
    280
    Likes Received:
    0
    Trophy Points:
    16
    Where are the other 1800 processes that were showing in the top you posted?

    You need a ps -aux output when top is showing the high load and all the processes running.
     
  16. Andrew1

    Andrew1 Registered

    Joined:
    Jan 11, 2003
    Messages:
    4
    Likes Received:
    0
    Trophy Points:
    1
    I am having the same problem and have been for a long time. My load at random times throughout the day starts rising very fast (i've seen it at 50, 80, 130, 200, 550, etc. on different days). When I catch it, I kill apache. The load goes back down to normal and I start apache again, and everything is fine, until a few hours (or a day) later, when it happens again. Here is what my error_log says every time:

    [Mon Mar 8 14:50:05 2004] [error] child process 12818 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12884 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12829 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12886 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12887 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12889 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12904 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12899 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12912 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12934 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12936 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12937 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12939 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12948 still did not exit, sending a SIGKILL
    [Mon Mar 8 14:50:05 2004] [error] child process 12949 still did not exit, sending a SIGKILL

    (there are also SIGTERM errors above these lines and manymore SIGKILL errors like those above). It does not stop until I manually kill apache. What could be the problem? Any advice would help.
     
  17. joshstein

    joshstein Active Member

    Joined:
    Feb 23, 2003
    Messages:
    40
    Likes Received:
    0
    Trophy Points:
    6
    bump for this
     
  18. DSLWeb

    DSLWeb Member

    Joined:
    Apr 6, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    In my experiance the "Load" doesnt matter as much as "Idle"

    14:02:28 up 39 days, 9:01, 1 user, load average: 0.07, 0.25, 0.25
    151 processes: 149 sleeping, 2 running, 0 zombie, 0 stopped
    CPU states: 1.1% user 0.3% system 0.0% nice 0.0% iowait 98.4% idle <=== leaves alot of room for Processes :D
    Mem: 902772k av, 865924k used, 36848k free, 0k shrd, 24468k buff
    545440k actv, 89376k in_d, 17676k in_c
    Swap: 2097136k av, 372352k used, 1724784k free 484828k cached


    Seems to be a cpanel bug cause you will find most every server that runs a few accounts that rely on MySQL will show a high load this can also be cause be "peaking" as requests come to the server they are scrambling to execute this may be at the same moment you are looking which will give you the 0% idle

    Something to look at tho if you are constantly running as a high load and your server laggs something bad my have a site getting DDoS attacks ...

    You can check top Process's in WHM under the "CPU/Memory/MySQL Usage" this can help you narrow down who is using your resources either by straight site or DDoS attacks

    Hope this helps
    Lee
     
  19. matt621

    matt621 Well-Known Member

    Joined:
    Jun 25, 2003
    Messages:
    175
    Likes Received:
    0
    Trophy Points:
    16
    I'm having something of a similiar problem.

    Early morning the server got a spike of activity and died. When I got up, I rebooted the server and for a few hours it was up. Then got another spike and it died again. (see attached)

    Now, after every reboot, the server shots up to 300 load average and dies.

    This has been a very reliable server until now. Up for about 2 years without a single down time, except for a PS that died and took out a HD but that was a year ago. Since then it's been fine.

    so what would cause a server's load to jump to 300 almost immediately after rebooting?
     
  20. matt621

    matt621 Well-Known Member

    Joined:
    Jun 25, 2003
    Messages:
    175
    Likes Received:
    0
    Trophy Points:
    16
    attachment didn't go first time.
     

    Attached Files:

Loading...

Share This Page