The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

high server load

Discussion in 'General Discussion' started by Michael-MS, Oct 10, 2005.

  1. Michael-MS

    Michael-MS Well-Known Member

    Joined:
    Apr 16, 2003
    Messages:
    144
    Likes Received:
    0
    Trophy Points:
    16
    I can't seem to figure out what's causing the server load to be so high. Just a few days ago it was running at 0.20 to 2.00 on average.

    The server is a Xeon 2.4 Ghz 2048 Megs 2 x 80 GB IDE server. I don't know why it shows two CPU's... it's not a dual xeon. Never noticed that before...


    PHP:
     02:31:20  up  2:32,  1 user,  load average10.1010.0710.00
    132 processes
    130 sleeping1 running1 zombie0 stopped
    CPU states
    :  cpu    user    nice  system    irq  softirq  iowait    idle
               total    0.3
    %    0.0%    0.8%   0.0%     0.2%    0.3%   98.0%
               
    cpu00    0.1%    0.0%    0.7%   0.0%     0.5%    0.3%   98.0%
               
    cpu01    0.5%    0.0%    0.9%   0.0%     0.0%    0.3%   98.0%
    Mem:  2055436k av1306168k used,  749268k free,       0k shrd,  156948k buff
           741284k active
    ,             352400k inactive
    Swap
    2048276k av,       0k used2048276k free                  888588k cached

      PID USER     PRI  NI  SIZE  RSS SHARE STAT 
    %CPU %MEM   TIME CPU COMMAND
    21350 root      15   0  1188 1188   896 R     0.1  0.0   0
    :00   0 top
        1 root      15   0   504  504   444 S     0.0  0.0   0
    :03   1 init
        2 root      RT   0     0    0     0 SW    0.0  0.0   0
    :00   0 migration/0
        3 root      RT   0     0    0     0 SW    0.0  0.0   0
    :00   1 migration/1
        4 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 keventd
        5 root      34  19     0    0     0 SWN   0.0  0.0   0
    :00   0 ksoftirqd/0
        6 root      39  19     0    0     0 SWN   0.0  0.0   0
    :00   1 ksoftirqd/1
        9 root      25   0     0    0     0 SW    0.0  0.0   0
    :00   1 bdflush
        7 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 kswapd
        8 root      15   0     0    0     0 SW    0.0  0.0   0
    :04   1 kscand
       10 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   0 kupdated
       11 root      25   0     0    0     0 SW    0.0  0.0   0
    :00   1 mdrecoveryd
       21 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 kjournald
       76 root      25   0     0    0     0 SW    0.0  0.0   0
    :00   1 khubd
      536 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 kjournald
      537 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   0 kjournald
      538 root      15   0     0    0     0 SW    0.0  0.0   0
    :03   1 kjournald
      539 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 kjournald
      540 root      15   0     0    0     0 SW    0.0  0.0   0
    :01   0 kjournald
      541 root      25   0     0    0     0 SW    0.0  0.0   0
    :00   0 kjournald
     2214 root      15   0   580  580   496 S     0.0  0.0   0
    :01   1 syslogd
     2218 root      15   0   460  460   396 S     0.0  0.0   0
    :00   1 klogd
     2228 root      15   0   440  440   376 S     0.0  0.0   0
    :00   0 irqbalance
     2238 root      15   0   392  392   336 S     0.0  0.0   0
    :00   0 mdadm
     2254 root      RT   0   664  664   528 S     0.0  0.0   0
    :00   1 auditd


    Anything else that I can check to see why it's running at 10.0?

    Michael
     
  2. abubin

    abubin Well-Known Member

    Joined:
    Dec 7, 2004
    Messages:
    393
    Likes Received:
    1
    Trophy Points:
    18
    the 2 cpu is because you have hyperthreading CPU or dual CPU.

    You top doesn't show much CPU usage. How about sorting the list by memory? Your HDD I/O is probably the one that is running high.
     
  3. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    I don't see any reason for the high load from that top display. If disk I/O were an issue you'd expect to see much higher IOWAIT %ages. It might help to know what OS and kernel version you are running on the server.
     
  4. procam

    procam Well-Known Member

    Joined:
    Nov 24, 2003
    Messages:
    123
    Likes Received:
    0
    Trophy Points:
    16
    The game is a foot

    I mean to tell you since early last week - this issue began showing up on my servers as they updated.....
    Now its spread across all my machines - from Fedora core1 units to fc3 units - and now my REL units are doing the same thing--
    Units all appear to have excessive loads at random times that cause them to use swap drive thats what caught my attention is my units typically never or very rarely use swap -
    This morning I checked a unit that has 2 small sites on it - running a P4 3.0 HT 800fsb 2 gigs of ram and its last boot was 3 weeks ago - BUT it has never ever used swap so I rebooted the unit -
    within 1 hour shes back to using 8.0 swap////
    I could see this happening on a busy machine OK -- but how in a week I went from none of my boxes using swap to all of them doing it now this baffles me - and we made no changes aside from updates....

    If you are having the same issue I am Michael-MS - youll need to bird sit this unit for a while to catch it - Ive been chasing it a week now and cant catch it - what I do know is when its happening the cpu will shoot up to 80-100% the unit will slow down use swap and then leave only the swap usage evidence in its wake --
     
  5. Michael-MS

    Michael-MS Well-Known Member

    Joined:
    Apr 16, 2003
    Messages:
    144
    Likes Received:
    0
    Trophy Points:
    16
    I'm locked out of my server right now. :rolleyes: SSH hangs and does not respond to a correct or incorrect password. I'm waiting for my datacenter to look into it. I'll reply to all of the questions once I have access to my server again. Oddly, everything except SSH is still running normally.

    Michael
     
  6. NightStorm

    NightStorm Well-Known Member

    Joined:
    Jul 28, 2003
    Messages:
    286
    Likes Received:
    4
    Trophy Points:
    18
    cPanel Access Level:
    Root Administrator
    Twitter:
    In cases like this, I installed Webmin as a backend system. One of it's modules is a type of watchdog... when certain things happen, the server can do various stuff to rectify or warn.
    Well, I set it to shoot me an email containing the output of 'top' when the system load reaches a certain point.
    Webmin is also helpful when SSH acts screwy. ;)
    Just a thought of a possible way to catch it while it's taking place, instead of trying to parse logs after... it would at least give you a snapshot of what processes were running hot at the time.
     
  7. procam

    procam Well-Known Member

    Joined:
    Nov 24, 2003
    Messages:
    123
    Likes Received:
    0
    Trophy Points:
    16
    All my units are also equipped with webmin - and I assume you mean System and Server Status specifically - can you please give me some further detail about how you set that up to take that shot and exactly how you are monitoring it please.

    Thanks~
     
  8. Michael-MS

    Michael-MS Well-Known Member

    Joined:
    Apr 16, 2003
    Messages:
    144
    Likes Received:
    0
    Trophy Points:
    16
    Thanks very much! I'll be sure to install that when I can. ;)

    The SSH problem looks like a bug:
    http://bugzilla.cpanel.net/show_bug.cgi?id=2802

    Michael
     
  9. Michael-MS

    Michael-MS Well-Known Member

    Joined:
    Apr 16, 2003
    Messages:
    144
    Likes Received:
    0
    Trophy Points:
    16
    NAC was able to fix the SSH problem for me (and charged me to fix it, grr). So I have access to my server again. The load is running normally now... maybe the problems were related?

    PHP:
     23:09:22  up  1:27,  1 user,  load average0.280.671.13
    139 processes
    138 sleeping1 running0 zombie0 stopped
    CPU states
    :  cpu    user    nice  system    irq  softirq  iowait    idle
               total    6.2
    %    0.0%    0.8%   0.0%     0.1%    2.9%   89.5%
               
    cpu00    7.5%    0.0%    0.5%   0.0%     0.3%    2.9%   88.4%
               
    cpu01    4.9%    0.0%    1.1%   0.1%     0.0%    2.9%   90.6%
    Mem:  2055436k av1199244k used,  856192k free,       0k shrd,  134436k buff
           751344k active
    ,             278012k inactive
    Swap
    2048276k av,       0k used2048276k free                  776596k cached

      PID USER     PRI  NI  SIZE  RSS SHARE STAT 
    %CPU %MEM   TIME CPU COMMAND
    20978 nobody    15   0 20664  20M  4636 S     0.7  1.0   0
    :00   1 httpd
    21011 nobody    15   0 20612  20M  4636 S     0.7  1.0   0
    :00   1 httpd
    24045 sshd      20   0  1716 1716  1428 S     0.2  0.0   0
    :00   1 sshd
    24026 root      16   0  1164 1164   888 R     0.1  0.0   0
    :00   1 top
    24030 root      16   0  2008 2008  1644 S     0.1  0.0   0
    :00   1 sshd
    24041 root      19   0  2024 2024  1652 S     0.1  0.0   0
    :00   0 sshd
        1 root      16   0   504  504   444 S     0.0  0.0   0
    :04   1 init
        2 root      RT   0     0    0     0 SW    0.0  0.0   0
    :00   0 migration/0
        3 root      RT   0     0    0     0 SW    0.0  0.0   0
    :00   1 migration/1
        4 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 keventd
        5 root      34  19     0    0     0 SWN   0.0  0.0   0
    :00   0 ksoftirqd/0
        6 root      34  19     0    0     0 SWN   0.0  0.0   0
    :00   1 ksoftirqd/1
        9 root      25   0     0    0     0 SW    0.0  0.0   0
    :00   1 bdflush
        7 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 kswapd
        8 root      15   0     0    0     0 SW    0.0  0.0   0
    :02   1 kscand
       10 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 kupdated
       11 root      25   0     0    0     0 SW    0.0  0.0   0
    :00   0 mdrecoveryd
       21 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   0 kjournald
       76 root      25   0     0    0     0 SW    0.0  0.0   0
    :00   0 khubd
      536 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 kjournald
      537 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   1 kjournald
      538 root      15   0     0    0     0 SW    0.0  0.0   0
    :05   1 kjournald
      539 root      15   0     0    0     0 SW    0.0  0.0   0
    :00   0 kjournald
      540 root      15   0     0    0     0 SW    0.0  0.0   0
    :01   1 kjournald
      541 root      25   0     0    0     0 SW    0.0  0.0   0
    :00   0 kjournald
     1937 root      15   0   640  640   556 S     0.0  0.0   0
    :00   1 crond
     2332 root      15   0   580  580   496 S     0.0  0.0   0
    :05   1 syslogd
     2336 root      15   0   456  456   396 S     0.0  0.0   0
    :00   0 klogd
     2346 root      15   0   440  440   376 S     0.0  0.0   0
    :00   0 irqbalance
     2356 root      15   0   392  392   336 S     0.0  0.0   0
    :00   0 mdadm
     3213 named     25   0  9924 9924  2176 S     0.0  0.4   0
    :00   1 named
     3214 named     15   0  9924 9924  2176 S     0.0  0.4   0
    :03   0 named

    uname 
    -Linux 2.4.21-32.0.1.ELsmp

    I'll update this if the load spikes again without reason.

    Michael
     
Loading...

Share This Page