high server load

[Michael-MS]

I can't seem to figure out what's causing the server load to be so high. Just a few days ago it was running at 0.20 to 2.00 on average.

The server is a Xeon 2.4 Ghz 2048 Megs 2 x 80 GB IDE server. I don't know why it shows two CPU's... it's not a dual xeon. Never noticed that before...

 02:31:20  up  2:32,  1 user,  load average: 10.10, 10.07, 10.00
132 processes: 130 sleeping, 1 running, 1 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.3%    0.0%    0.8%   0.0%     0.2%    0.3%   98.0%
           cpu00    0.1%    0.0%    0.7%   0.0%     0.5%    0.3%   98.0%
           cpu01    0.5%    0.0%    0.9%   0.0%     0.0%    0.3%   98.0%
Mem:  2055436k av, 1306168k used,  749268k free,       0k shrd,  156948k buff
       741284k active,             352400k inactive
Swap: 2048276k av,       0k used, 2048276k free                  888588k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
21350 root      15   0  1188 1188   896 R     0.1  0.0   0:00   0 top
    1 root      15   0   504  504   444 S     0.0  0.0   0:03   1 init
    2 root      RT   0     0    0     0 SW    0.0  0.0   0:00   0 migration/0
    3 root      RT   0     0    0     0 SW    0.0  0.0   0:00   1 migration/1
    4 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 keventd
    5 root      34  19     0    0     0 SWN   0.0  0.0   0:00   0 ksoftirqd/0
    6 root      39  19     0    0     0 SWN   0.0  0.0   0:00   1 ksoftirqd/1
    9 root      25   0     0    0     0 SW    0.0  0.0   0:00   1 bdflush
    7 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kswapd
    8 root      15   0     0    0     0 SW    0.0  0.0   0:04   1 kscand
   10 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kupdated
   11 root      25   0     0    0     0 SW    0.0  0.0   0:00   1 mdrecoveryd
   21 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kjournald
   76 root      25   0     0    0     0 SW    0.0  0.0   0:00   1 khubd
  536 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kjournald
  537 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
  538 root      15   0     0    0     0 SW    0.0  0.0   0:03   1 kjournald
  539 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kjournald
  540 root      15   0     0    0     0 SW    0.0  0.0   0:01   0 kjournald
  541 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
 2214 root      15   0   580  580   496 S     0.0  0.0   0:01   1 syslogd
 2218 root      15   0   460  460   396 S     0.0  0.0   0:00   1 klogd
 2228 root      15   0   440  440   376 S     0.0  0.0   0:00   0 irqbalance
 2238 root      15   0   392  392   336 S     0.0  0.0   0:00   0 mdadm
 2254 root      RT   0   664  664   528 S     0.0  0.0   0:00   1 auditd 

Anything else that I can check to see why it's running at 10.0?

Michael

[abubin]

the 2 cpu is because you have hyperthreading CPU or dual CPU.

You top doesn't show much CPU usage. How about sorting the list by memory? Your HDD I/O is probably the one that is running high.

[chirpy]

I don't see any reason for the high load from that top display. If disk I/O were an issue you'd expect to see much higher IOWAIT %ages. It might help to know what OS and kernel version you are running on the server.

[procam]

I mean to tell you since early last week - this issue began showing up on my servers as they updated.....
Now its spread across all my machines - from Fedora core1 units to fc3 units - and now my REL units are doing the same thing--
Units all appear to have excessive loads at random times that cause them to use swap drive thats what caught my attention is my units typically never or very rarely use swap -
This morning I checked a unit that has 2 small sites on it - running a P4 3.0 HT 800fsb 2 gigs of ram and its last boot was 3 weeks ago - BUT it has never ever used swap so I rebooted the unit -
within 1 hour shes back to using 8.0 swap////
I could see this happening on a busy machine OK -- but how in a week I went from none of my boxes using swap to all of them doing it now this baffles me - and we made no changes aside from updates....

If you are having the same issue I am Michael-MS - youll need to bird sit this unit for a while to catch it - Ive been chasing it a week now and cant catch it - what I do know is when its happening the cpu will shoot up to 80-100% the unit will slow down use swap and then leave only the swap usage evidence in its wake --

[Michael-MS]

I'm locked out of my server right now. SSH hangs and does not respond to a correct or incorrect password. I'm waiting for my datacenter to look into it. I'll reply to all of the questions once I have access to my server again. Oddly, everything except SSH is still running normally.

Michael

[NightStorm]

In cases like this, I installed Webmin as a backend system. One of it's modules is a type of watchdog... when certain things happen, the server can do various stuff to rectify or warn.
Well, I set it to shoot me an email containing the output of 'top' when the system load reaches a certain point.
Webmin is also helpful when SSH acts screwy.
Just a thought of a possible way to catch it while it's taking place, instead of trying to parse logs after... it would at least give you a snapshot of what processes were running hot at the time.

[procam]

In cases like this, I installed Webmin as a backend system. One of it's modules is a type of watchdog... when certain things happen, the server can do various stuff to rectify or warn.
Well, I set it to shoot me an email containing the output of 'top' when the system load reaches a certain point.
Webmin is also helpful when SSH acts screwy.
Just a thought of a possible way to catch it while it's taking place, instead of trying to parse logs after... it would at least give you a snapshot of what processes were running hot at the time.

All my units are also equipped with webmin - and I assume you mean System and Server Status specifically - can you please give me some further detail about how you set that up to take that shot and exactly how you are monitoring it please.

Thanks~

[Michael-MS]

In cases like this, I installed Webmin as a backend system. One of it's modules is a type of watchdog... when certain things happen, the server can do various stuff to rectify or warn.
Well, I set it to shoot me an email containing the output of 'top' when the system load reaches a certain point.
Webmin is also helpful when SSH acts screwy.
Just a thought of a possible way to catch it while it's taking place, instead of trying to parse logs after... it would at least give you a snapshot of what processes were running hot at the time.

Thanks very much! I'll be sure to install that when I can.

The SSH problem looks like a bug:
http://bugzilla.cpanel.net/show_bug.cgi?id=2802

Michael

[Michael-MS]

NAC was able to fix the SSH problem for me (and charged me to fix it, grr). So I have access to my server again. The load is running normally now... maybe the problems were related?

 23:09:22  up  1:27,  1 user,  load average: 0.28, 0.67, 1.13
139 processes: 138 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    6.2%    0.0%    0.8%   0.0%     0.1%    2.9%   89.5%
           cpu00    7.5%    0.0%    0.5%   0.0%     0.3%    2.9%   88.4%
           cpu01    4.9%    0.0%    1.1%   0.1%     0.0%    2.9%   90.6%
Mem:  2055436k av, 1199244k used,  856192k free,       0k shrd,  134436k buff
       751344k active,             278012k inactive
Swap: 2048276k av,       0k used, 2048276k free                  776596k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
20978 nobody    15   0 20664  20M  4636 S     0.7  1.0   0:00   1 httpd
21011 nobody    15   0 20612  20M  4636 S     0.7  1.0   0:00   1 httpd
24045 sshd      20   0  1716 1716  1428 S     0.2  0.0   0:00   1 sshd
24026 root      16   0  1164 1164   888 R     0.1  0.0   0:00   1 top
24030 root      16   0  2008 2008  1644 S     0.1  0.0   0:00   1 sshd
24041 root      19   0  2024 2024  1652 S     0.1  0.0   0:00   0 sshd
    1 root      16   0   504  504   444 S     0.0  0.0   0:04   1 init
    2 root      RT   0     0    0     0 SW    0.0  0.0   0:00   0 migration/0
    3 root      RT   0     0    0     0 SW    0.0  0.0   0:00   1 migration/1
    4 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 keventd
    5 root      34  19     0    0     0 SWN   0.0  0.0   0:00   0 ksoftirqd/0
    6 root      34  19     0    0     0 SWN   0.0  0.0   0:00   1 ksoftirqd/1
    9 root      25   0     0    0     0 SW    0.0  0.0   0:00   1 bdflush
    7 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kswapd
    8 root      15   0     0    0     0 SW    0.0  0.0   0:02   1 kscand
   10 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kupdated
   11 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 mdrecoveryd
   21 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
   76 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 khubd
  536 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kjournald
  537 root      15   0     0    0     0 SW    0.0  0.0   0:00   1 kjournald
  538 root      15   0     0    0     0 SW    0.0  0.0   0:05   1 kjournald
  539 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
  540 root      15   0     0    0     0 SW    0.0  0.0   0:01   1 kjournald
  541 root      25   0     0    0     0 SW    0.0  0.0   0:00   0 kjournald
 1937 root      15   0   640  640   556 S     0.0  0.0   0:00   1 crond
 2332 root      15   0   580  580   496 S     0.0  0.0   0:05   1 syslogd
 2336 root      15   0   456  456   396 S     0.0  0.0   0:00   0 klogd
 2346 root      15   0   440  440   376 S     0.0  0.0   0:00   0 irqbalance
 2356 root      15   0   392  392   336 S     0.0  0.0   0:00   0 mdadm
 3213 named     25   0  9924 9924  2176 S     0.0  0.4   0:00   1 named
 3214 named     15   0  9924 9924  2176 S     0.0  0.4   0:03   0 named

uname -a = Linux 2.4.21-32.0.1.ELsmp 

I'll update this if the load spikes again without reason.

Michael