High iowait

[launch]

My server has been playing up lately. Some websites on it are taking 20 seconds to load, while others are taking .5ms. The weird thing is its not consistent.

Here is the output from iostat. Is the util column abnormal?

-bash-3.2# iostat -x 1 10

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  2.16  0.62  1.03  14.79  0.85  80.56

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  149.83  64.69 249.07 45.12  7259.20  891.51  27.71  3.20  10.89  2.12  62.25
xvda2  0.00  0.00  0.00  0.00  0.04  0.00  8.56  0.00  1.48  1.48  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  1.60  0.00  1.20  22.60  5.60  69.00

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  56.44  0.00 707.92  0.00 24792.08  0.00  35.02  7.93  11.21  1.38  97.43
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  3.21  0.00  0.20  25.05  2.81  68.74

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  55.00  0.00 570.00  0.00 15200.00  0.00  26.67  7.62  13.34  1.76 100.40
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  0.40  0.00  0.99  20.83  1.79  75.99

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  132.00  0.00 571.00  0.00 18632.00  0.00  32.63  7.00  12.21  1.75 100.00
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  1.99  0.00  0.99  21.27  1.99  73.76

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  70.30  0.00 274.26  0.00  8221.78  0.00  29.98  3.73  14.15  3.57  97.82
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  0.00  0.00  0.00  21.00  0.83  78.17

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  54.00  191.00 514.00 43.00 16640.00  1872.00  33.24  6.14  11.06  1.74  96.80
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  0.00  0.00  0.19  22.12  0.19  77.50

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  20.00  0.00 240.00  0.00  3680.00  0.00  15.33  2.14  8.80  4.10  98.40
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  0.38  0.00  1.34  18.13  1.15  79.01

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  190.29  0.00 525.24  0.00 17188.35  0.00  32.72  4.92  9.44  1.84  96.70
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  0.80  0.00  2.41  20.72  2.41  73.64

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  190.00  0.00 680.00  0.00 19904.00  0.00  29.27  6.53  9.52  1.39  94.80
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  0.40  0.00  0.40  21.69  1.41  76.10

Device:  rrqm/s  wrqm/s  r/s  w/s  rsec/s  wsec/s avgrq-sz avgqu-sz  await  svctm  %util
xvda1  84.16  0.00 535.64  0.00 10225.74  0.00  19.09  6.14  11.45  1.83  98.22
xvda2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

Any help on how to debug this would be appreciated!

 

[cPanelMichael]

Hello,

I do see the iowait values are elevated, but it's difficult to determine why that is. The following thread is a good place to get started when troubleshooting this type of issue:

Troubleshooting High Loads On Linux Systems

Thank you.

 

[launch]

Thanks, I've been reading through that but I'm not sure what values are abnormal. Do these look ok?

sar -p

-          CPU     %user     %nice   %system   %iowait    %steal     %idle
Average:          all      1.78      0.32      0.91     14.12      0.55     82.31

sar -r

-    kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
Average:      1295779   1849949     58.81      3489    570074         0         0      0.00         0

sar -d says "Requested activities not available in file"

 

[launch]

I've managed to solve this. The server had a cron job that was running too often. That link really did help.

Thanks!

 

[cPanelMichael]

I am happy to see you were able to determine the cause of the issue. Thank you for updating us with the outcome.