High Loads/High IOWAIT - any ideas?

[Brook]

For some reason my server occasionally get's very high loads bringing it to a slow halt. Any ideas why this might be? nothing in top stats seems to point to anything apart from the high IOWAIT.

21:34:07 up 12 days, 6:19, 1 user, load average: 149.20, 117.72, 60.56
653 processes: 640 sleeping, 3 running, 10 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 1.5% 0.0% 4.3% 0.7% 0.0% 93.3% 0.0%
Mem: 1011344k av, 999424k used, 11920k free, 0k shrd, 4924k buff
760808k actv, 141496k in_d, 12104k in_c
Swap: 1052248k av, 717024k used, 335224k free 85492k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
6 root 15 0 0 0 0 SW 3.0 0.0 2:11 0 kscand
7273 myusername 16 0 1652 1652 888 R 0.9 0.1 0:01 0 top
5 root 15 0 0 0 0 SW 0.3 0.0 1:34 0 kswapd
6148 nobody 15 0 7752 6580 1836 D 0.3 0.6 0:00 0 httpd
15022 mysql 15 0 202M 167M 2252 D 0.1 16.9 0:30 0 mysqld
5947 nobody 15 0 9836 7688 1900 D 0.1 0.7 0:00 0 httpd
5960 nobody 15 0 10720 8812 1900 D 0.1 0.8 0:00 0 httpd
6099 nobody 15 0 8028 6716 1840 D 0.1 0.6 0:00 0 httpd
6144 nobody 15 0 8540 7500 1836 S 0.1 0.7 0:00 0 httpd
6154 nobody 15 0 8304 7196 1840 S 0.1 0.7 0:00 0 httpd
9048 root 21 0 2832 2832 2332 R 0.1 0.2 0:00 0 exim
1 root 15 0 188 156 128 S 0.0 0.0 0:08 0 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 keventd
3 root 15 0 0 0 0 SW 0.0 0.0 0:05 0 kapmd
4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
7 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
8 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 kupdated
9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
13 root 15 0 0 0 0 DW 0.0 0.0 2:49 0 kjournald
301 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
728 root 15 0 324 304 240 D 0.0 0.0 0:17 0 syslogd
732 root 15 0 192 132 128 S 0.0 0.0 0:00 0 klogd
753 root RT 0 368 284 220 D 0.0 0.0 3:08 0 auditd
9677 named 25 0 24456 22M 1092 S 0.0 2.2 0:00 0 named
9678 named 15 0 24456 22M 1092 S 0.0 2.2 9:39 0 named
9679 named 15 0 24456 22M 1092 S 0.0 2.2 0:03 0 named
9680 named 15 0 24456 22M 1092 S 0.0 2.2 1:07 0 named
9692 root 15 0 424 272 184 S 0.0 0.0 0:02 0 sshd
9706 root 15 0 340 268 216 S 0.0 0.0 0:01 0 xinetd
9717 root 25 0 248 100 96 S 0.0 0.0 0:00 0 mysqld_safe
9738 mysql 15 0 202M 167M 2252 D 0.0 16.9 1:34 0 mysqld
9743 mysql 15 0 202M 167M 2252 S 0.0 16.9 1:11 0 mysqld
9760 root 15 0 4408 1980 696 S 0.0 0.1 0:13 0 chkservd
9825 mailnull 15 0 1540 1280 1196 S 0.0 0.1 0:28 0 exim
9829 mailnull 15 0 1472 1136 1132 S 0.0 0.1 0:00 0 exim
9865 root 15 0 4300 2936 1448 S 0.0 0.2 0:07 0 httpd
9873 root 15 0 648 380 276 S 0.0 0.0 0:01 0 crond
9914 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:40 0 mysqld
9916 xfs 15 0 2220 140 112 S 0.0 0.0 0:00 0 xfs
10001 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:40 0 mysqld
10014 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:42 0 mysqld
10035 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:41 0 mysqld
10038 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:41 0 mysqld
10041 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:40 0 mysqld
10042 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:39 0 mysqld
10049 mysql 15 0 202M 167M 2252 S 0.0 16.9 0:40 0 mysqld

Then IOWAIT suddenly drops back to zero and then in top I get this:

20690 nobody 25 0 14192 8004 2248 R 93.3 0.7 10:08 0 httpd

With the CPU % at 93.3 for command http.

Then cpanel will automagically restart cpsrvd and loads will eventually come back down again.

Could this be a dos attack? Any ideas what commands I should use to see what's causing this?

My forums software sends me a report to say 'MySQL server has gone away' - could that be a hint as to what it is?

The whole thing lasts about 20 minutes or so and seems to be happening more frequently.

Would appreciate any help on the matter.

[mtindor]

For some reason my server occasionally get's very high loads bringing it to a slow halt. Any ideas why this might be? nothing in top stats seems to point to anything apart from the high IOWAIT.



Then IOWAIT suddenly drops back to zero and then in top I get this:




With the CPU % at 93.3 for command http.

Then cpanel will automagically restart cpsrvd and loads will eventually come back down again.

Could this be a dos attack? Any ideas what commands I should use to see what's causing this?

My forums software sends me a report to say 'MySQL server has gone away' - could that be a hint as to what it is?

The whole thing lasts about 20 minutes or so and seems to be happening more frequently.

Would appreciate any help on the matter.

Look at your Apache Server Status in WHM when this is happening and you might be able to get an idea [if it is Apache] what site is being hit so hard and by whom to cause that.

SQL going away could mean that MySQL is the problem, or it could just simply mean that because of the load on the server at that time MySQL cannot effectively communicate with PHP/Apache.

Mike

[trhosting.net]

What is your operating system and kernel version?

[Brook]

Thanks Both...

CENTOS Enterprise 3.8 i686
Kernel: 2.4.21-47.0.1.EL

WHM 11.2.0 cPanel
WHM X v3.1.0

[Frimon86]

What kind of mysql version are you running? Have you tried to make sure all the scripts you are hosting are up to date? I dont think it's a dosattack

[Brook]

I don't think it's a dos attack either, here is some further info:

Web Server Apache v1.3.37
PHP 4.4.7
PHP Max Post Size 55.00 MB
PHP Maximum Upload Size 2.00 MB
PHP Memory Limit None
MySQL Version 5.0.27-standard
MySQL Packet Size 64.00 MB

[mtindor]

Showing us specs and OS/Cpanel versions really isn't going to help in any way.

You have to wait til it is happening and learn how to use some other command line tools to determine what is going on at the time it is happening.

Mike

[Freezer]

Swap: 1052248k av, 717024k used, 335224k free 85492k cached

Look @ your swap usage. Your server is swapping into/from harddrive. Place more memory in your server, IOwait is the time waiting on harddrive.
So increase your memory (with at least 1 gb, because you are using 700mb in swap)

[trhosting.net]

Upgrade your ram and you may try upgrading your kernel.

[Frimon86]

Yeah it may not have enough ram in your server, hopfully it wont bust your wallet just to upgrade.

[jayh38]

You could use more ram and also look into why you are keeping all those processes open. Your httpd.conf may need a tweak with connection limits / keepalive etc etc among endless other possibilities.

Hire someone to go through your system and give you a proper audit.