Need help troubleshooting high load

[lbeachmike]

Hi there -

Recently, my server has began showing a high CPU load. The server is lightly loaded for the most part and I need some help nailing down the cause of the high load.

Here is a snapshot of my top CPU processes -

Pid Owner Priority Cpu % Mem % Command
3854 nobody 0 7.9 0.8 /usr/local/apache/bin/httpd -DSSL
28015 nobody 0 6.9 0.9 /usr/local/apache/bin/httpd -DSSL
9853 nobody 0 3.7 0.6 /usr/local/apache/bin/httpd -DSSL
9918 nobody 0 2.9 0.6 /usr/local/apache/bin/httpd -DSSL
20631 nobody 0 2.1 0.8 /usr/local/apache/bin/httpd -DSSL
9854 nobody 0 1.9 0.4 /usr/local/apache/bin/httpd -DSSL
9857 nobody 0 1.7 0.5 /usr/local/apache/bin/httpd -DSSL
3572 nobody 0 0.3 1.5 /usr/local/apache/bin/httpd -DSSL
9955 root 0 0.3 0.0 0 top -n 2 -b -
3569 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
16437 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
17050 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
22806 nobody 0 0.1 0.7 /usr/local/apache/bin/httpd -DSSL
7147 nobody 0 0.1 0.7 /usr/local/apache/bin/httpd -DSSL
9065 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
9072 nobody 0 0.1 0.5 /usr/local/apache/bin/httpd -DSSL
9872 nobody 0 0.1 0.3 /usr/local/apache/bin/httpd -DSSL
9921 nobody 0 0.1 0.2 /usr/local/apache/bin/httpd -DSSL

I've done a lot of searching, but need some help in where to go from here.

Thanks.

Mike

[jayh38]

It looks like you have a user with a very heavy web page but its hard to tell because the top of top is missing, lol.

info about the server would be helpful too. Oh, you were looking via whm. Use ssh and the top -c command and post back.
Although its not very accurate, in whm you can look at the cpu/memory/mysql usage in whm and see where the calls are coming from. Again its not very accurate but an easy starting point for you to look at.

[lbeachmike]

Hmmm - Top of top missing? I copied/pasted all of it.

Does this help? (top -c from shell instead) -

32381 nobody 15 0 17580 17M 1552 S 2.7 0.8 1:26 0 /usr/local/apache/bin/httpd -DSSL
16437 nobody 15 0 17608 17M 1572 S 2.3 0.8 4:01 0 /usr/local/apache/bin/httpd -DSSL
3678 nobody 15 0 17484 17M 1456 S 0.3 0.8 3:52 0 /usr/local/apache/bin/httpd -DSSL
3572 nobody 15 0 30876 30M 1328 S 0.1 1.5 7:29 0 /usr/local/apache/bin/httpd -DSSL
27173 nobody 15 0 13048 12M 1624 S 0.1 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
29278 root 15 0 1104 1104 776 R 0.1 0.0 0:00 0 top -c
1 root 15 0 112 112 56 S 0.0 0.0 0:04 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
4 root 15 0 0 0 0 SW 0.0 0.0 0:04 0 kapmd
5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
6 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 kswapd
7 root 15 0 0 0 0 SW 0.0 0.0 0:35 0 kscand
9 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kupdated
10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
18 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_0
19 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_1
22 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
77 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
2679 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
2680 root 15 0 0 0 0 SW 0.0 0.0 0:04 0 kjournald
2681 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
2682 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kjournald
2683 root 15 0 0 0 0 SW 0.0 0.0 0:09 0 kjournald
2684 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
3150 root 15 0 256 256 172 S 0.0 0.0 0:03 0 syslogd -m 0
3154 root 15 0 60 60 0 S 0.0 0.0 0:00 0 klogd -x
3190 root 25 0 60 60 0 S 0.0 0.0 0:00 0 /usr/sbin/courierlogger -pid=/var/spool/authda
3191 root 15 0 100 100 4 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond
3201 root 15 0 176 176 116 S 0.0 0.0 0:00 0 mdadm --monitor --scan -f


Server details -

Processors 1
Model Intel(R) Pentium(R) 4 CPU 3.20GHz
CPU Speed 3.19 GHz
Cache Size 512 KB
System Bogomips 6370.09
RAM = 2 GB

Server Version: Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a
Server Built: Aug 16 2006 11:58:54

Kernel Version 2.4.21-40.ELsmp (SMP)
Distro Name Red Hat Enterprise Linux ES release 3 (Taroon Update 8)

WHM 10.8.0 cPanel 10.9.0-C138

Is there any additional info that would be helpful?

Thanks so much.

Mike

[Silver_2000]

top - 19:03:41 up 25 days, 20:08, 1 user, load average: 0.62, 0.71, 0.66
Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie
Cpu0 : 20.9% us, 3.5% sy, 0.0% ni, 75.6% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu1 : 3.0% us, 0.5% sy, 0.0% ni, 96.5% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu2 : 5.0% us, 0.5% sy, 0.0% ni, 94.5% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu3 : 19.9% us, 2.5% sy, 0.0% ni, 77.6% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 2074928k total, 1957908k used, 117020k free, 74240k buffers
Swap: 2096472k total, 144k used, 2096328k free, 1151868k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30468 nobody 16 0 22296 14m 2160 S 39 0.7 0:05.22 /usr/local/apache/bin/httpd -DSSL
30587 nobody 15 0 17380 9740 2032 S 8 0.5 0:02.05 /usr/local/apache/bin/httpd -DSSL
15003 mysql 15 0 540m 168m 3872 S 5 8.3 127:12.94 /usr/sbin/mysqld --basedir=/ --datadir
31301 nobody 16 0 16784 9180 2068 S 1 0.4 0:01.87 /usr/local/apache/bin/httpd -DSSL
24538 nobody 15 0 20956 13m 2444 S 0 0.7 0:21.50 /usr/local/apache/bin/httpd -DSSL
29181 nobody 15 0 21840 13m 2160 S 0 0.7 0:10.89 /usr/local/apache/bin/httpd -DSSL
31127 nobody 15 0 17360 9748 2060 S 0 0.5 0:02.91 /usr/local/apache/bin/httpd -DSSL
1 root 16 0 2124 552 472 S 0 0.0 0:06.94 init [3]
2 root RT 0 0 0 0 S 0 0.0 0:17.20 [migration/0]
3 root 34 19 0 0 0 S 0 0.0 0:02.86 [ksoftirqd/0]
4 root RT 0 0 0 0 S 0 0.0 0:11.66 [migration/1]
5 root 34 19 0 0 0 S 0 0.0 0:00.53 [ksoftirqd/1]
6 root RT 0 0 0 0 S 0 0.0 2:08.15 [migration/2]
7 root 34 19 0 0 0 S 0 0.0 0:01.28 [ksoftirqd/2]
8 root RT 0 0 0 0 S 0 0.0 1:41.13 [migration/3]
9 root 34 19 0 0 0 S 0 0.0 0:01.19 [ksoftirqd/3]
10 root 5 -10 0 0 0 S 0 0.0 0:00.95 [events/0]
11 root 5 -10 0 0 0 S 0 0.0 0:01.02 [events/1]
12 root 5 -10 0 0 0 S 0 0.0 0:01.06 [events/2]
13 root 5 -10 0 0 0 S 0 0.0 0:01.05 [events/3]
14 root 6 -10 0 0 0 S 0 0.0 0:00.00 [khelper]
15 root 15 -10 0 0 0 S 0 0.0 0:00.00 [kacpid]
30 root 5 -10 0 0 0 S 0 0.0 0:00.01 [kblockd/0]
31 root 5 -10 0 0 0 S 0 0.0 0:00.00 [kblockd/1]
32 root 5 -10 0 0 0 S 0 0.0 0:00.00 [kblockd/2]
33 root 5 -10 0 0 0 S 0 0.0 0:00.00 [kblockd/3]
54 root 13 -10 0 0 0 S 0 0.0 0:00.00 [aio/0]
55 root 5 -10 0 0 0 S 0 0.0 0:00.00 [aio/1]
56 root 13 -10 0 0 0 S 0 0.0 0:00.00 [aio/2]
57 root 13 -10 0 0 0 S 0 0.0 0:00.00 [aio/3]
34 root 15 0 0 0 0 S 0 0.0 0:00.00 [khubd]

this is what top shows is you get the whole thing

[lbeachmike]

Thank you for that clarification -

Here's a current snapshot of everything this time

top -c
19:13:06 up 16:51, 2 users, load average: 0.35, 0.31, 0.33
196 processes: 193 sleeping, 2 running, 1 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 7.9% 0.0% 1.9% 0.1% 0.0% 6.5% 83.2%
Mem: 2054948k av, 1975584k used, 79364k free, 0k shrd, 134172k buff
1425108k actv, 285128k in_d, 26348k in_c
Swap: 2040212k av, 19880k used, 2020332k free 849096k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
22802 nobody 15 0 17600 17M 1772 S 2.5 0.8 2:21 0 /usr/local/apache/bin/httpd -DSSL
27759 mailnull 15 0 3120 3120 2044 S 0.7 0.1 0:00 0 /usr/sbin/exim -bd -q60m
20631 nobody 15 0 18508 18M 2132 S 0.3 0.9 3:03 0 /usr/local/apache/bin/httpd -DSSL
29278 root 15 0 1188 1188 776 R 0.3 0.0 0:29 0 top -c
2680 root 15 0 0 0 0 SW 0.1 0.0 0:05 0 kjournald
3570 nobody 15 0 31340 30M 1744 S 0.1 1.5 9:13 0 /usr/local/apache/bin/httpd -DSSL
3572 nobody 15 0 31284 30M 1736 S 0.1 1.5 7:39 0 /usr/local/apache/bin/httpd -DSSL
20847 nobody 15 0 31472 30M 1904 S 0.1 1.5 4:36 0 /usr/local/apache/bin/httpd -DSSL
16437 nobody 15 0 17776 17M 1740 S 0.1 0.8 4:14 0 /usr/local/apache/bin/httpd -DSSL
5110 nobody 15 0 17960 17M 2456 S 0.1 0.8 0:25 0 /usr/local/apache/bin/httpd -DSSL
12194 nobody 15 0 14896 14M 2676 S 0.1 0.7 0:03 0 /usr/local/apache/bin/httpd -DSSL
27679 nobody 15 0 10972 10M 2508 S 0.1 0.5 0:00 0 /usr/local/apache/bin/httpd -DSSL
1 root 15 0 112 112 56 S 0.0 0.0 0:04 0 init
2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
4 root 15 0 0 0 0 SW 0.0 0.0 0:05 0 kapmd
5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
6 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 kswapd
7 root 15 0 0 0 0 SW 0.0 0.0 0:42 0 kscand
9 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kupdated
10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
18 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_0
19 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_1
22 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
77 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
2679 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
2681 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
2682 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kjournald
2683 root 15 0 0 0 0 SW 0.0 0.0 0:11 0 kjournald
2684 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
3150 root 15 0 256 256 172 S 0.0 0.0 0:04 0 syslogd -m 0
3154 root 15 0 60 60 0 S 0.0 0.0 0:00 0 klogd -x
3190 root 25 0 60 60 0 S 0.0 0.0 0:00 0 /usr/sbin/courierlogger -pid=/var/spool/authda
3191 root 15 0 116 116 20 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond
3201 root 15 0 176 176 116 S 0.0 0.0 0:00 0 mdadm --monitor --scan -f
3203 root 15 0 200 200 100 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond

[Silver_2000]

Your load is .3

you are fine...

[lbeachmike]

As of that snapshot, yes. But we're seeing wide fluctuations, spiking up to > 25 at some point in time yesterday - although, generally spiking up between 5 and 10.

I'll grab a snapshot when the load peaks - until then, any other suggestions?

Thanks.

Mike

[Silver_2000]

Keep in mind that load can be disk usage as well .. and it doesnt show up well in TOP

You will see the load go up but not always whats using the disk to cause it ..

doing a cpanel backup is a great example the zipping of the files is both cpu and disk intensive but its the disk usage that drives up the load on my box ..

[jayh38]

Yes, we would definitely need to see your top when you have the loading issue.

Although, its just a snapshot of that moment, you do not seem to have many users to warrant 196 tasks. So I would guess this is shortly after experiencing your trouble?

Here is a command to tally actual connections at the time of your spike. See if its a connection issue to help narrow things down.

Run this constantly and get a feel for the tasks and loading during normal times.
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | wc -l

You do have a bit of a swap file so at one point or several, you ran low on ram which would also cause your loading. Try restarting apache to wipe some of the unused cache away.

also run this utility to monitor statistics. The first column is important and should not exceed your cpu count by much or at all.

vmstat -3

type man vmstat for more details of the report

[lbeachmike]

This is extremely helpful info. Thank you so much for your help.

Mike