The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Need help troubleshooting high load

Discussion in 'General Discussion' started by lbeachmike, Feb 28, 2007.

  1. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    Hi there -

    Recently, my server has began showing a high CPU load. The server is lightly loaded for the most part and I need some help nailing down the cause of the high load.

    Here is a snapshot of my top CPU processes -

    Pid Owner Priority Cpu % Mem % Command
    3854 nobody 0 7.9 0.8 /usr/local/apache/bin/httpd -DSSL
    28015 nobody 0 6.9 0.9 /usr/local/apache/bin/httpd -DSSL
    9853 nobody 0 3.7 0.6 /usr/local/apache/bin/httpd -DSSL
    9918 nobody 0 2.9 0.6 /usr/local/apache/bin/httpd -DSSL
    20631 nobody 0 2.1 0.8 /usr/local/apache/bin/httpd -DSSL
    9854 nobody 0 1.9 0.4 /usr/local/apache/bin/httpd -DSSL
    9857 nobody 0 1.7 0.5 /usr/local/apache/bin/httpd -DSSL
    3572 nobody 0 0.3 1.5 /usr/local/apache/bin/httpd -DSSL
    9955 root 0 0.3 0.0 0 top -n 2 -b -
    3569 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
    16437 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
    17050 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
    22806 nobody 0 0.1 0.7 /usr/local/apache/bin/httpd -DSSL
    7147 nobody 0 0.1 0.7 /usr/local/apache/bin/httpd -DSSL
    9065 nobody 0 0.1 0.8 /usr/local/apache/bin/httpd -DSSL
    9072 nobody 0 0.1 0.5 /usr/local/apache/bin/httpd -DSSL
    9872 nobody 0 0.1 0.3 /usr/local/apache/bin/httpd -DSSL
    9921 nobody 0 0.1 0.2 /usr/local/apache/bin/httpd -DSSL

    I've done a lot of searching, but need some help in where to go from here.

    Thanks.

    Mike
     
  2. jayh38

    jayh38 Well-Known Member

    Joined:
    Mar 3, 2006
    Messages:
    1,215
    Likes Received:
    0
    Trophy Points:
    36
    It looks like you have a user with a very heavy web page but its hard to tell because the top of top is missing, lol.

    info about the server would be helpful too. Oh, you were looking via whm. Use ssh and the top -c command and post back.
    Although its not very accurate, in whm you can look at the cpu/memory/mysql usage in whm and see where the calls are coming from. Again its not very accurate but an easy starting point for you to look at.
     
    #2 jayh38, Feb 28, 2007
    Last edited: Feb 28, 2007
  3. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    Hmmm - Top of top missing? I copied/pasted all of it.

    Does this help? (top -c from shell instead) -

    32381 nobody 15 0 17580 17M 1552 S 2.7 0.8 1:26 0 /usr/local/apache/bin/httpd -DSSL
    16437 nobody 15 0 17608 17M 1572 S 2.3 0.8 4:01 0 /usr/local/apache/bin/httpd -DSSL
    3678 nobody 15 0 17484 17M 1456 S 0.3 0.8 3:52 0 /usr/local/apache/bin/httpd -DSSL
    3572 nobody 15 0 30876 30M 1328 S 0.1 1.5 7:29 0 /usr/local/apache/bin/httpd -DSSL
    27173 nobody 15 0 13048 12M 1624 S 0.1 0.6 0:00 0 /usr/local/apache/bin/httpd -DSSL
    29278 root 15 0 1104 1104 776 R 0.1 0.0 0:00 0 top -c
    1 root 15 0 112 112 56 S 0.0 0.0 0:04 0 init
    2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
    3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
    4 root 15 0 0 0 0 SW 0.0 0.0 0:04 0 kapmd
    5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
    6 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 kswapd
    7 root 15 0 0 0 0 SW 0.0 0.0 0:35 0 kscand
    9 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kupdated
    10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    18 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_0
    19 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_1
    22 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
    77 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    2679 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2680 root 15 0 0 0 0 SW 0.0 0.0 0:04 0 kjournald
    2681 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2682 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kjournald
    2683 root 15 0 0 0 0 SW 0.0 0.0 0:09 0 kjournald
    2684 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    3150 root 15 0 256 256 172 S 0.0 0.0 0:03 0 syslogd -m 0
    3154 root 15 0 60 60 0 S 0.0 0.0 0:00 0 klogd -x
    3190 root 25 0 60 60 0 S 0.0 0.0 0:00 0 /usr/sbin/courierlogger -pid=/var/spool/authda
    3191 root 15 0 100 100 4 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond
    3201 root 15 0 176 176 116 S 0.0 0.0 0:00 0 mdadm --monitor --scan -f


    Server details -

    Processors 1
    Model Intel(R) Pentium(R) 4 CPU 3.20GHz
    CPU Speed 3.19 GHz
    Cache Size 512 KB
    System Bogomips 6370.09
    RAM = 2 GB

    Server Version: Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a
    Server Built: Aug 16 2006 11:58:54

    Kernel Version 2.4.21-40.ELsmp (SMP)
    Distro Name Red Hat Enterprise Linux ES release 3 (Taroon Update 8)

    WHM 10.8.0 cPanel 10.9.0-C138

    Is there any additional info that would be helpful?

    Thanks so much.

    Mike
     
  4. Silver_2000

    Silver_2000 Well-Known Member

    Joined:
    Mar 31, 2002
    Messages:
    338
    Likes Received:
    1
    Trophy Points:
    18
    this is what top shows is you get the whole thing
     
  5. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    Thank you for that clarification -

    Here's a current snapshot of everything this time :)

    top -c
    19:13:06 up 16:51, 2 users, load average: 0.35, 0.31, 0.33
    196 processes: 193 sleeping, 2 running, 1 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 7.9% 0.0% 1.9% 0.1% 0.0% 6.5% 83.2%
    Mem: 2054948k av, 1975584k used, 79364k free, 0k shrd, 134172k buff
    1425108k actv, 285128k in_d, 26348k in_c
    Swap: 2040212k av, 19880k used, 2020332k free 849096k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    22802 nobody 15 0 17600 17M 1772 S 2.5 0.8 2:21 0 /usr/local/apache/bin/httpd -DSSL
    27759 mailnull 15 0 3120 3120 2044 S 0.7 0.1 0:00 0 /usr/sbin/exim -bd -q60m
    20631 nobody 15 0 18508 18M 2132 S 0.3 0.9 3:03 0 /usr/local/apache/bin/httpd -DSSL
    29278 root 15 0 1188 1188 776 R 0.3 0.0 0:29 0 top -c
    2680 root 15 0 0 0 0 SW 0.1 0.0 0:05 0 kjournald
    3570 nobody 15 0 31340 30M 1744 S 0.1 1.5 9:13 0 /usr/local/apache/bin/httpd -DSSL
    3572 nobody 15 0 31284 30M 1736 S 0.1 1.5 7:39 0 /usr/local/apache/bin/httpd -DSSL
    20847 nobody 15 0 31472 30M 1904 S 0.1 1.5 4:36 0 /usr/local/apache/bin/httpd -DSSL
    16437 nobody 15 0 17776 17M 1740 S 0.1 0.8 4:14 0 /usr/local/apache/bin/httpd -DSSL
    5110 nobody 15 0 17960 17M 2456 S 0.1 0.8 0:25 0 /usr/local/apache/bin/httpd -DSSL
    12194 nobody 15 0 14896 14M 2676 S 0.1 0.7 0:03 0 /usr/local/apache/bin/httpd -DSSL
    27679 nobody 15 0 10972 10M 2508 S 0.1 0.5 0:00 0 /usr/local/apache/bin/httpd -DSSL
    1 root 15 0 112 112 56 S 0.0 0.0 0:04 0 init
    2 root RT 0 0 0 0 SW 0.0 0.0 0:00 0 migration/0
    3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
    4 root 15 0 0 0 0 SW 0.0 0.0 0:05 0 kapmd
    5 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
    6 root 15 0 0 0 0 SW 0.0 0.0 0:06 0 kswapd
    7 root 15 0 0 0 0 SW 0.0 0.0 0:42 0 kscand
    9 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kupdated
    10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    18 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_0
    19 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 scsi_eh_1
    22 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kjournald
    77 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    2679 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2681 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2682 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 kjournald
    2683 root 15 0 0 0 0 SW 0.0 0.0 0:11 0 kjournald
    2684 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    3150 root 15 0 256 256 172 S 0.0 0.0 0:04 0 syslogd -m 0
    3154 root 15 0 60 60 0 S 0.0 0.0 0:00 0 klogd -x
    3190 root 25 0 60 60 0 S 0.0 0.0 0:00 0 /usr/sbin/courierlogger -pid=/var/spool/authda
    3191 root 15 0 116 116 20 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond
    3201 root 15 0 176 176 116 S 0.0 0.0 0:00 0 mdadm --monitor --scan -f
    3203 root 15 0 200 200 100 S 0.0 0.0 0:00 0 /usr/libexec/courier-authlib/authdaemond
     
  6. Silver_2000

    Silver_2000 Well-Known Member

    Joined:
    Mar 31, 2002
    Messages:
    338
    Likes Received:
    1
    Trophy Points:
    18
    Your load is .3

    you are fine...
     
  7. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    As of that snapshot, yes. But we're seeing wide fluctuations, spiking up to > 25 at some point in time yesterday - although, generally spiking up between 5 and 10.

    I'll grab a snapshot when the load peaks - until then, any other suggestions?

    Thanks.

    Mike
     
  8. Silver_2000

    Silver_2000 Well-Known Member

    Joined:
    Mar 31, 2002
    Messages:
    338
    Likes Received:
    1
    Trophy Points:
    18
    Keep in mind that load can be disk usage as well .. and it doesnt show up well in TOP

    You will see the load go up but not always whats using the disk to cause it ..

    doing a cpanel backup is a great example the zipping of the files is both cpu and disk intensive but its the disk usage that drives up the load on my box ..
     
  9. jayh38

    jayh38 Well-Known Member

    Joined:
    Mar 3, 2006
    Messages:
    1,215
    Likes Received:
    0
    Trophy Points:
    36
    Yes, we would definitely need to see your top when you have the loading issue.

    Although, its just a snapshot of that moment, you do not seem to have many users to warrant 196 tasks. So I would guess this is shortly after experiencing your trouble?

    Here is a command to tally actual connections at the time of your spike. See if its a connection issue to help narrow things down.

    Run this constantly and get a feel for the tasks and loading during normal times.
    netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | wc -l

    You do have a bit of a swap file so at one point or several, you ran low on ram which would also cause your loading. Try restarting apache to wipe some of the unused cache away.

    also run this utility to monitor statistics. The first column is important and should not exceed your cpu count by much or at all.

    vmstat -3

    type man vmstat for more details of the report
     
  10. lbeachmike

    lbeachmike Well-Known Member

    Joined:
    Dec 27, 2001
    Messages:
    313
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Long Beach, NY
    cPanel Access Level:
    Root Administrator
    This is extremely helpful info. Thank you so much for your help.

    Mike
     
Loading...

Share This Page