Community Forums
Connect with us on LinkedIn
Community Notice
+ Reply to Thread
Results 1 to 10 of 10
  1. #1
    Member
    Join Date
    Dec 2001
    Location
    Long Beach, NY
    Posts
    194

    Default Need help troubleshooting high load

    Hi there -

    Recently, my server has began showing a high CPU load. The server is lightly loaded for the most part and I need some help nailing down the cause of the high load.

    Here is a snapshot of my top CPU processes -

    Pid Owner Priority Cpu % Mem % Command
    3854 nobody 0 7.9 0.8 /usr/local/apache/bin/httpd -DSSL
    28015 nobody 0 6.9 0.9 /usr/local/apache/bin/httpd -DSSL
    9853 nobody 0 3.7 0.6 /usr/local/apache/bin/httpd -DSSL
    9918 nobody 0 2.9 0.6 /usr/local/apache/bin/httpd -DSSL
    20631 nobody 0 2.1 0.8 /usr/local/apache/bin/httpd -DSSL
    9854 nobody 0 1.9 0.4 /usr/local/apache/bin/httpd -DSSL
    9857 nobody 0 1.7 0.5 /usr/local/apache/bin/httpd -DSSL
    3572 nobody 0 0.3 1.5 /usr/local/apache/bin/httpd -DSSL
    9955 root 0 0.3 0.0 0 top -n 2 -b -
    3569 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
    16437 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
    17050 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
    22806 nobody 0 0.1 0.7 /usr/local/apache/bin/httpd -DSSL
    7147 nobody 0 0.1 0.7 /usr/local/apache/bin/httpd -DSSL
    9065 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
    9072 nobody 0 0.1 0.5 /usr/local/apache/bin/httpd -DSSL
    9872 nobody 0 0.1 0.3 /usr/local/apache/bin/httpd -DSSL
    9921 nobody 0 0.1 0.2 /usr/local/apache/bin/httpd -DSSL

    I've done a lot of searching, but need some help in where to go from here.

    Thanks.

    Mike
    Get your hosting business listed on www.HostPremiums.com.
    Visit www.decomodo.com for cool stuff you can't live without.

  2. #2
    Member
    Join Date
    Mar 2006
    Posts
    1,215

    Default

    It looks like you have a user with a very heavy web page but its hard to tell because the top of top is missing, lol.

    info about the server would be helpful too. Oh, you were looking via whm. Use ssh and the top -c command and post back.
    Although its not very accurate, in whm you can look at the cpu/memory/mysql usage in whm and see where the calls are coming from. Again its not very accurate but an easy starting point for you to look at.
    Last edited by jayh38; 02-28-2007 at 05:43 PM.

  3. #3
    Member
    Join Date
    Dec 2001
    Location
    Long Beach, NY
    Posts
    194

    Default

    Hmmm - Top of top missing? I copied/pasted all of it.

    Does this help? (top -c from shell instead) -

    32381 nobody 15 0 17580 17M 1552 S 2.7 0.8 1:26 0 /usr/local/apache/bin/httpd -DSSL
    16437 nobody 15 0 17608 17M 1572 S 2.3 0.8 4:01 0 /usr/local/apache/bin/httpd -DSSL
    3678 nobody 15 0 17484 17M 1456 S 0.3 0.8 3:52 0 /usr/local/apache/bin/httpd -DSSL
    3572 nobody 15 0 30876 30M 1328 S 0.1 1.5 7:29 0 /usr/local/apache/bin/httpd -DSSL
    27173 nobody 15 0 13048 12M 1624 S 0.1 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
    29278 root 15 0 1104 1104 776 R 0.1 0.0 0:00 0 top -c
    1 root 15 0 112 112 56 S 0.0 0.0 0:04 0 init
    2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
    3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
    4 root 15 0 0 0 0 SW 0.0 0.0 0:04 0 kapmd
    5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
    6 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 kswapd
    7 root 15 0 0 0 0 SW 0.0 0.0 0:35 0 kscand
    9 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kupdated
    10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    18 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_0
    19 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_1
    22 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
    77 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    2679 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2680 root 15 0 0 0 0 SW 0.0 0.0 0:04 0 kjournald
    2681 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2682 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kjournald
    2683 root 15 0 0 0 0 SW 0.0 0.0 0:09 0 kjournald
    2684 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    3150 root 15 0 256 256 172 S 0.0 0.0 0:03 0 syslogd -m 0
    3154 root 15 0 60 60 0 S 0.0 0.0 0:00 0 klogd -x
    3190 root 25 0 60 60 0 S 0.0 0.0 0:00 0 /usr/sbin/courierlogger -pid=/var/spool/authda
    3191 root 15 0 100 100 4 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond
    3201 root 15 0 176 176 116 S 0.0 0.0 0:00 0 mdadm --monitor --scan -f


    Server details -

    Processors 1
    Model Intel(R) Pentium(R) 4 CPU 3.20GHz
    CPU Speed 3.19 GHz
    Cache Size 512 KB
    System Bogomips 6370.09
    RAM = 2 GB

    Server Version: Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a
    Server Built: Aug 16 2006 11:58:54

    Kernel Version 2.4.21-40.ELsmp (SMP)
    Distro Name Red Hat Enterprise Linux ES release 3 (Taroon Update 8)

    WHM 10.8.0 cPanel 10.9.0-C138

    Is there any additional info that would be helpful?

    Thanks so much.

    Mike
    Get your hosting business listed on www.HostPremiums.com.
    Visit www.decomodo.com for cool stuff you can't live without.

  4. #4
    Member
    Join Date
    Mar 2002
    Posts
    295

    Default

    top - 19:03:41 up 25 days, 20:08, 1 user, load average: 0.62, 0.71, 0.66
    Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombie
    Cpu0 : 20.9% us, 3.5% sy, 0.0% ni, 75.6% id, 0.0% wa, 0.0% hi, 0.0% si
    Cpu1 : 3.0% us, 0.5% sy, 0.0% ni, 96.5% id, 0.0% wa, 0.0% hi, 0.0% si
    Cpu2 : 5.0% us, 0.5% sy, 0.0% ni, 94.5% id, 0.0% wa, 0.0% hi, 0.0% si
    Cpu3 : 19.9% us, 2.5% sy, 0.0% ni, 77.6% id, 0.0% wa, 0.0% hi, 0.0% si
    Mem: 2074928k total, 1957908k used, 117020k free, 74240k buffers
    Swap: 2096472k total, 144k used, 2096328k free, 1151868k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    30468 nobody 16 0 22296 14m 2160 S 39 0.7 0:05.22 /usr/local/apache/bin/httpd -DSSL
    30587 nobody 15 0 17380 9740 2032 S 8 0.5 0:02.05 /usr/local/apache/bin/httpd -DSSL
    15003 mysql 15 0 540m 168m 3872 S 5 8.3 127:12.94 /usr/sbin/mysqld --basedir=/ --datadir
    31301 nobody 16 0 16784 9180 2068 S 1 0.4 0:01.87 /usr/local/apache/bin/httpd -DSSL
    24538 nobody 15 0 20956 13m 2444 S 0 0.7 0:21.50 /usr/local/apache/bin/httpd -DSSL
    29181 nobody 15 0 21840 13m 2160 S 0 0.7 0:10.89 /usr/local/apache/bin/httpd -DSSL
    31127 nobody 15 0 17360 9748 2060 S 0 0.5 0:02.91 /usr/local/apache/bin/httpd -DSSL
    1 root 16 0 2124 552 472 S 0 0.0 0:06.94 init [3]
    2 root RT 0 0 0 0 S 0 0.0 0:17.20 [migration/0]
    3 root 34 19 0 0 0 S 0 0.0 0:02.86 [ksoftirqd/0]
    4 root RT 0 0 0 0 S 0 0.0 0:11.66 [migration/1]
    5 root 34 19 0 0 0 S 0 0.0 0:00.53 [ksoftirqd/1]
    6 root RT 0 0 0 0 S 0 0.0 2:08.15 [migration/2]
    7 root 34 19 0 0 0 S 0 0.0 0:01.28 [ksoftirqd/2]
    8 root RT 0 0 0 0 S 0 0.0 1:41.13 [migration/3]
    9 root 34 19 0 0 0 S 0 0.0 0:01.19 [ksoftirqd/3]
    10 root 5 -10 0 0 0 S 0 0.0 0:00.95 [events/0]
    11 root 5 -10 0 0 0 S 0 0.0 0:01.02 [events/1]
    12 root 5 -10 0 0 0 S 0 0.0 0:01.06 [events/2]
    13 root 5 -10 0 0 0 S 0 0.0 0:01.05 [events/3]
    14 root 6 -10 0 0 0 S 0 0.0 0:00.00 [khelper]
    15 root 15 -10 0 0 0 S 0 0.0 0:00.00 [kacpid]
    30 root 5 -10 0 0 0 S 0 0.0 0:00.01 [kblockd/0]
    31 root 5 -10 0 0 0 S 0 0.0 0:00.00 [kblockd/1]
    32 root 5 -10 0 0 0 S 0 0.0 0:00.00 [kblockd/2]
    33 root 5 -10 0 0 0 S 0 0.0 0:00.00 [kblockd/3]
    54 root 13 -10 0 0 0 S 0 0.0 0:00.00 [aio/0]
    55 root 5 -10 0 0 0 S 0 0.0 0:00.00 [aio/1]
    56 root 13 -10 0 0 0 S 0 0.0 0:00.00 [aio/2]
    57 root 13 -10 0 0 0 S 0 0.0 0:00.00 [aio/3]
    34 root 15 0 0 0 0 S 0 0.0 0:00.00 [khubd]
    this is what top shows is you get the whole thing

  5. #5
    Member
    Join Date
    Dec 2001
    Location
    Long Beach, NY
    Posts
    194

    Default

    Thank you for that clarification -

    Here's a current snapshot of everything this time

    top -c
    19:13:06 up 16:51, 2 users, load average: 0.35, 0.31, 0.33
    196 processes: 193 sleeping, 2 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 7.9% 0.0% 1.9% 0.1% 0.0% 6.5% 83.2%
    Mem: 2054948k av, 1975584k used, 79364k free, 0k shrd, 134172k buff
    1425108k actv, 285128k in_d, 26348k in_c
    Swap: 2040212k av, 19880k used, 2020332k free 849096k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    22802 nobody 15 0 17600 17M 1772 S 2.5 0.8 2:21 0 /usr/local/apache/bin/httpd -DSSL
    27759 mailnull 15 0 3120 3120 2044 S 0.7 0.1 0:00 0 /usr/sbin/exim -bd -q60m
    20631 nobody 15 0 18508 18M 2132 S 0.3 0.9 3:03 0 /usr/local/apache/bin/httpd -DSSL
    29278 root 15 0 1188 1188 776 R 0.3 0.0 0:29 0 top -c
    2680 root 15 0 0 0 0 SW 0.1 0.0 0:05 0 kjournald
    3570 nobody 15 0 31340 30M 1744 S 0.1 1.5 9:13 0 /usr/local/apache/bin/httpd -DSSL
    3572 nobody 15 0 31284 30M 1736 S 0.1 1.5 7:39 0 /usr/local/apache/bin/httpd -DSSL
    20847 nobody 15 0 31472 30M 1904 S 0.1 1.5 4:36 0 /usr/local/apache/bin/httpd -DSSL
    16437 nobody 15 0 17776 17M 1740 S 0.1 0.8 4:14 0 /usr/local/apache/bin/httpd -DSSL
    5110 nobody 15 0 17960 17M 2456 S 0.1 0.8 0:25 0 /usr/local/apache/bin/httpd -DSSL
    12194 nobody 15 0 14896 14M 2676 S 0.1 0.7 0:03 0 /usr/local/apache/bin/httpd -DSSL
    27679 nobody 15 0 10972 10M 2508 S 0.1 0.5 0:00 0 /usr/local/apache/bin/httpd -DSSL
    1 root 15 0 112 112 56 S 0.0 0.0 0:04 0 init
    2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
    3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
    4 root 15 0 0 0 0 SW 0.0 0.0 0:05 0 kapmd
    5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
    6 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 kswapd
    7 root 15 0 0 0 0 SW 0.0 0.0 0:42 0 kscand
    9 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kupdated
    10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    18 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_0
    19 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_1
    22 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
    77 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    2679 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2681 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2682 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kjournald
    2683 root 15 0 0 0 0 SW 0.0 0.0 0:11 0 kjournald
    2684 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    3150 root 15 0 256 256 172 S 0.0 0.0 0:04 0 syslogd -m 0
    3154 root 15 0 60 60 0 S 0.0 0.0 0:00 0 klogd -x
    3190 root 25 0 60 60 0 S 0.0 0.0 0:00 0 /usr/sbin/courierlogger -pid=/var/spool/authda
    3191 root 15 0 116 116 20 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond
    3201 root 15 0 176 176 116 S 0.0 0.0 0:00 0 mdadm --monitor --scan -f
    3203 root 15 0 200 200 100 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond
    Get your hosting business listed on www.HostPremiums.com.
    Visit www.decomodo.com for cool stuff you can't live without.

  6. #6
    Member
    Join Date
    Mar 2002
    Posts
    295

    Default

    Your load is .3

    you are fine...

  7. #7
    Member
    Join Date
    Dec 2001
    Location
    Long Beach, NY
    Posts
    194

    Default

    As of that snapshot, yes. But we're seeing wide fluctuations, spiking up to > 25 at some point in time yesterday - although, generally spiking up between 5 and 10.

    I'll grab a snapshot when the load peaks - until then, any other suggestions?

    Thanks.

    Mike
    Get your hosting business listed on www.HostPremiums.com.
    Visit www.decomodo.com for cool stuff you can't live without.

  8. #8
    Member
    Join Date
    Mar 2002
    Posts
    295

    Default

    Keep in mind that load can be disk usage as well .. and it doesnt show up well in TOP

    You will see the load go up but not always whats using the disk to cause it ..

    doing a cpanel backup is a great example the zipping of the files is both cpu and disk intensive but its the disk usage that drives up the load on my box ..

  9. #9
    Member
    Join Date
    Mar 2006
    Posts
    1,215

    Default

    Yes, we would definitely need to see your top when you have the loading issue.

    Although, its just a snapshot of that moment, you do not seem to have many users to warrant 196 tasks. So I would guess this is shortly after experiencing your trouble?

    Here is a command to tally actual connections at the time of your spike. See if its a connection issue to help narrow things down.

    Run this constantly and get a feel for the tasks and loading during normal times.
    netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | wc -l

    You do have a bit of a swap file so at one point or several, you ran low on ram which would also cause your loading. Try restarting apache to wipe some of the unused cache away.

    also run this utility to monitor statistics. The first column is important and should not exceed your cpu count by much or at all.

    vmstat -3

    type man vmstat for more details of the report

  10. #10
    Member
    Join Date
    Dec 2001
    Location
    Long Beach, NY
    Posts
    194

    Default

    This is extremely helpful info. Thank you so much for your help.

    Mike
    Get your hosting business listed on www.HostPremiums.com.
    Visit www.decomodo.com for cool stuff you can't live without.

Similar Threads & Tags
Similar threads

  1. high load problem with high traffic on apache at nights
    By LAZer in forum cPanel and WHM Discussions
    Replies: 3
    Last Post: 03-04-2011, 09:12 PM
  2. High Server Load Troubleshooting
    By Major Headache in forum cPanel and WHM Discussions
    Replies: 13
    Last Post: 04-04-2007, 12:07 AM
  3. Extremely high server load - BDflush running very high- possible hack
    By Blink2 in forum cPanel and WHM Discussions
    Replies: 9
    Last Post: 07-30-2005, 07:46 PM
  4. High CPU loading & troubleshooting.
    By SuperBaby in forum cPanel and WHM Discussions
    Replies: 3
    Last Post: 01-24-2005, 05:09 PM
  5. high cpu and time processes... load average high
    By alexd in forum cPanel and WHM Discussions
    Replies: 5
    Last Post: 12-20-2004, 04:50 AM
Linkedin       Facebook       Twitter       RSS       Flickr       YouTube