High iowait causing load spikes but for no reason?

Metro2

Well-Known Member
May 24, 2006
574
94
178
USA
cPanel Access Level
Root Administrator
I could really use some advice and/or help, I'm at the end of my rope with this problem after trying to troubleshoot it for weeks along with the help of my data center. I'm getting some crazy load spikes / high load average alerts on this one server and it seem due to iowait, but for no apparent reason.

If you look at the screenshot that I attached to this post (which I just snapped during one of these iowait load spikes) you can see that the iowait is up over 360%.

The thing is, the traffic was normal on the server for this time of day and was fairly light. Typically the average load on this box at this time of day would be .35 to .55 , but now it's spiking up to 16.5 several times a day. Been doing this for about 6 weeks now.

I've had this box for over 2 years and it actually has less accounts on it now than it did 8 months ago. There are about 140 accounts on it, but half of them aren't more than a basic HTML page, and the server only averages about 250gb per month in transfer.

It's running:
Dual-Xeon Dual-Core 2.8ghz CPU's
2gb of RAM
100mbps uplink
WHM 11.1.0 cPanel 11.4.19-R14379
REDHAT Enterprise 3 i686 - WHM X v3.1.0

We've tried narrowing it down by checking cron-job times, making sure mySQL is optimized, making sure the kernel is updated, manually rebooting the server, disabling certain functions, etc... and nothing seems to have any affect.

Sometimes the spikes seem random, but I have noticed that when the chkrootkit cron runs it happens consistently.

For a box with this much power and this little traffic, and no change in customer accounts or the types of things they run in many months, these high iowait load spikes are completely unprecedented. It's happening 3 or 4 times a day at least and when it happens it brings the server to a crawl for 5 to 10 minutes.

My only *guess* at this point is a hardware issue, but I was hoping someone else here might have some ideas.

If any of you could spare a minute to give your opinion I would really appreciate it, thank you very much!
 

Attachments

Last edited:

jayh38

Well-Known Member
Mar 3, 2006
1,212
0
166
Run a few commands in a shell session. This may lead you in a good direction.

dmesg
vmstat
iostat
chkconfig --list
 

Metro2

Well-Known Member
May 24, 2006
574
94
178
USA
cPanel Access Level
Root Administrator
Thanks jay, I will have to read up on what all these results mean that I'm seeing from running those commands. The only thing I really recognize is when I run dmesg I see mostly the typical firewall blocking alerts that I commonly see in my System Check reports. The other command results are a bit greek to me.