Results 1 to 11 of 11

Thread: High Loads/High IOWAIT - any ideas?

  1. #1
    Registered Member
    Join Date
    Apr 2005
    Posts
    99

    Default High Loads/High IOWAIT - any ideas?

    For some reason my server occasionally get's very high loads bringing it to a slow halt. Any ideas why this might be? nothing in top stats seems to point to anything apart from the high IOWAIT.

    21:34:07 up 12 days, 6:19, 1 user, load average: 149.20, 117.72, 60.56
    653 processes: 640 sleeping, 3 running, 10 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 1.5% 0.0% 4.3% 0.7% 0.0% 93.3% 0.0%
    Mem: 1011344k av, 999424k used, 11920k free, 0k shrd, 4924k buff
    760808k actv, 141496k in_d, 12104k in_c
    Swap: 1052248k av, 717024k used, 335224k free 85492k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    6 root 15 0 0 0 0 SW 3.0 0.0 2:11 0 kscand
    7273 myusername 16 0 1652 1652 888 R 0.9 0.1 0:01 0 top
    5 root 15 0 0 0 0 SW 0.3 0.0 1:34 0 kswapd
    6148 nobody 15 0 7752 6580 1836 D 0.3 0.6 0:00 0 httpd
    15022 mysql 15 0 202M 167M 2252 D 0.1 16.9 0:30 0 mysqld
    5947 nobody 15 0 9836 7688 1900 D 0.1 0.7 0:00 0 httpd
    5960 nobody 15 0 10720 8812 1900 D 0.1 0.8 0:00 0 httpd
    6099 nobody 15 0 8028 6716 1840 D 0.1 0.6 0:00 0 httpd
    6144 nobody 15 0 8540 7500 1836 S 0.1 0.7 0:00 0 httpd
    6154 nobody 15 0 8304 7196 1840 S 0.1 0.7 0:00 0 httpd
    9048 root 21 0 2832 2832 2332 R 0.1 0.2 0:00 0 exim
    1 root 15 0 188 156 128 S 0.0 0.0 0:08 0 init
    2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
    3 root 15 0 0 0 0 SW 0.0 0.0 0:05 0 kapmd
    4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    7 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
    8 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kupdated
    9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    13 root 15 0 0 0 0 DW 0.0 0.0 2:49 0 kjournald
    301 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    728 root 15 0 324 304 240 D 0.0 0.0 0:17 0 syslogd
    732 root 15 0 192 132 128 S 0.0 0.0 0:00 0 klogd
    753 root RT 0 368 284 220 D 0.0 0.0 3:08 0 auditd
    9677 named 25 0 24456 22M 1092 S 0.0 2.2 0:00 0 named
    9678 named 15 0 24456 22M 1092 S 0.0 2.2 9:39 0 named
    9679 named 15 0 24456 22M 1092 S 0.0 2.2 0:03 0 named
    9680 named 15 0 24456 22M 1092 S 0.0 2.2 1:07 0 named
    9692 root 15 0 424 272 184 S 0.0 0.0 0:02 0 sshd
    9706 root 15 0 340 268 216 S 0.0 0.0 0:01 0 xinetd
    9717 root 25 0 248 100 96 S 0.0 0.0 0:00 0 mysqld_safe
    9738 mysql 15 0 202M 167M 2252 D 0.0 16.9 1:34 0 mysqld
    9743 mysql 15 0 202M 167M 2252 S 0.0 16.9 1:11 0 mysqld
    9760 root 15 0 4408 1980 696 S 0.0 0.1 0:13 0 chkservd
    9825 mailnull 15 0 1540 1280 1196 S 0.0 0.1 0:28 0 exim
    9829 mailnull 15 0 1472 1136 1132 S 0.0 0.1 0:00 0 exim
    9865 root 15 0 4300 2936 1448 S 0.0 0.2 0:07 0 httpd
    9873 root 15 0 648 380 276 S 0.0 0.0 0:01 0 crond
    9914 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:40 0 mysqld
    9916 xfs 15 0 2220 140 112 S 0.0 0.0 0:00 0 xfs
    10001 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:40 0 mysqld
    10014 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:42 0 mysqld
    10035 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:41 0 mysqld
    10038 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:41 0 mysqld
    10041 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:40 0 mysqld
    10042 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:39 0 mysqld
    10049 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:40 0 mysqld
    Then IOWAIT suddenly drops back to zero and then in top I get this:


    20690 nobody 25 0 14192 8004 2248 R 93.3 0.7 10:08 0 httpd
    With the CPU % at 93.3 for command http.

    Then cpanel will automagically restart cpsrvd and loads will eventually come back down again.

    Could this be a dos attack? Any ideas what commands I should use to see what's causing this?

    My forums software sends me a report to say 'MySQL server has gone away' - could that be a hint as to what it is?

    The whole thing lasts about 20 minutes or so and seems to be happening more frequently.

    Would appreciate any help on the matter.

  2. #2
    Registered Member
    Join Date
    Sep 2004
    Location
    inside a catfish
    Posts
    1,157
    cPanel/WHM Access Level

    Root Administrator

    Default

    Quote Originally Posted by Brook View Post
    For some reason my server occasionally get's very high loads bringing it to a slow halt. Any ideas why this might be? nothing in top stats seems to point to anything apart from the high IOWAIT.



    Then IOWAIT suddenly drops back to zero and then in top I get this:




    With the CPU % at 93.3 for command http.

    Then cpanel will automagically restart cpsrvd and loads will eventually come back down again.

    Could this be a dos attack? Any ideas what commands I should use to see what's causing this?

    My forums software sends me a report to say 'MySQL server has gone away' - could that be a hint as to what it is?

    The whole thing lasts about 20 minutes or so and seems to be happening more frequently.

    Would appreciate any help on the matter.
    Look at your Apache Server Status in WHM when this is happening and you might be able to get an idea [if it is Apache] what site is being hit so hard and by whom to cause that.

    SQL going away could mean that MySQL is the problem, or it could just simply mean that because of the load on the server at that time MySQL cannot effectively communicate with PHP/Apache.

    Mike

  3. #3
    Registered Member
    Join Date
    Mar 2006
    Location
    Turkey
    Posts
    151

    Default

    What is your operating system and kernel version?

  4. #4
    Registered Member
    Join Date
    Apr 2005
    Posts
    99

    Default

    Thanks Both...

    CENTOS Enterprise 3.8 i686
    Kernel: 2.4.21-47.0.1.EL

    WHM 11.2.0 cPanel
    WHM X v3.1.0

  5. #5
    BANNED
    Join Date
    Jun 2007
    Posts
    31

    Default

    What kind of mysql version are you running? Have you tried to make sure all the scripts you are hosting are up to date? I dont think it's a dosattack

  6. #6
    Registered Member
    Join Date
    Apr 2005
    Posts
    99

    Default

    I don't think it's a dos attack either, here is some further info:
    Web Server Apache v1.3.37
    PHP 4.4.7
    PHP Max Post Size 55.00 MB
    PHP Maximum Upload Size 2.00 MB
    PHP Memory Limit None
    MySQL Version 5.0.27-standard
    MySQL Packet Size 64.00 MB

  7. #7
    Registered Member
    Join Date
    Sep 2004
    Location
    inside a catfish
    Posts
    1,157
    cPanel/WHM Access Level

    Root Administrator

    Default

    Showing us specs and OS/Cpanel versions really isn't going to help in any way.

    You have to wait til it is happening and learn how to use some other command line tools to determine what is going on at the time it is happening.

    Mike

  8. #8
    Registered Member
    Join Date
    Jun 2005
    Location
    Den Haag
    Posts
    120

    Default

    Swap: 1052248k av, 717024k used, 335224k free 85492k cached

    Look @ your swap usage. Your server is swapping into/from harddrive. Place more memory in your server, IOwait is the time waiting on harddrive.
    So increase your memory (with at least 1 gb, because you are using 700mb in swap)

  9. #9
    Registered Member
    Join Date
    Mar 2006
    Location
    Turkey
    Posts
    151

    Default

    Upgrade your ram and you may try upgrading your kernel.

  10. #10
    BANNED
    Join Date
    Jun 2007
    Posts
    31

    Default

    Yeah it may not have enough ram in your server, hopfully it wont bust your wallet just to upgrade.

  11. #11
    Registered Member
    Join Date
    Mar 2006
    Posts
    1,215

    Default

    You could use more ram and also look into why you are keeping all those processes open. Your httpd.conf may need a tweak with connection limits / keepalive etc etc among endless other possibilities.

    Hire someone to go through your system and give you a proper audit.

Similar Threads

  1. High IOwait white
    By NixTree in forum CloudLinux
    Replies: 24
    Last Post: 01-19-2013, 10:42 PM
  2. Cpu problem: high iowait
    By fchx in forum cPanel & WHM Discussions
    Replies: 1
    Last Post: 06-25-2008, 04:01 AM
  3. Problem with high load, high iowait CPU
    By Berbox in forum cPanel & WHM Discussions
    Replies: 4
    Last Post: 06-20-2007, 05:38 AM
  4. High load time to time high iowait
    By the_bobara in forum cPanel & WHM Discussions
    Replies: 9
    Last Post: 01-03-2007, 01:56 PM
  5. High iowait
    By netlook in forum cPanel & WHM Discussions
    Replies: 7
    Last Post: 11-25-2004, 04:12 AM
bargain