Hi,
Since a few days ago when I had to restart one of our shared hosting servers (and failed to restart so the DC techs had to intervene - they've probably ran some manual fsck but I am just not sure what they've run) the IO performance on our server dropped dramatically right after reboot.
The HDDs are setup in Raid 1 behind a LSI MegaRAID 8704ELP Version: 1.20.
The HDD themselves are ST3500320NS.
The raid matrix seems to be in a NOT degraded state (status optimal, no media errors).
CPU is E5520 (quad core w. HT, 8MB cache)
The problem is that IO wait is 5 times bigger than it should (probably more) and iostat shows pretty weird data. For comparison I present 2 identical servers:
While r/s and w/s are about the same (I believe this means they share a similar utilization) and avgrq-sz is virtually the same, rrqm/s is much lower in the bad system, avgrq-sz is much higher (it gets to about 75 times under higher load) and the await is also much higher (gets to ~ 50 times larger under load) and also service time (svctm).
Also while on the OK server kjournald is very discreet on the bad server kjournald takes the top through 2 different forks (out of 4) even after setting the ionice class to Idle for those 2 kjournald processes.
So what makes rrqm go down and avgqu-sz, await and svctm go up in a bad system? Is it a HDD, is it the card itself, is it some rogue mount option? What is busting the second server?
Thanks in advance for any suggestion!
Since a few days ago when I had to restart one of our shared hosting servers (and failed to restart so the DC techs had to intervene - they've probably ran some manual fsck but I am just not sure what they've run) the IO performance on our server dropped dramatically right after reboot.
The HDDs are setup in Raid 1 behind a LSI MegaRAID 8704ELP Version: 1.20.
The HDD themselves are ST3500320NS.
The raid matrix seems to be in a NOT degraded state (status optimal, no media errors).
CPU is E5520 (quad core w. HT, 8MB cache)
The problem is that IO wait is 5 times bigger than it should (probably more) and iostat shows pretty weird data. For comparison I present 2 identical servers:
Code:
rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util
OK one: 62.90 67.33 76.98 53.49 2469.66 985.81 26.48 0.10 3.83 0.22 2.84
Bad one: 12.67 78.06 74.31 49.04 2318.72 1017.19 27.04 2.19 17.73 4.38 54.03Also while on the OK server kjournald is very discreet on the bad server kjournald takes the top through 2 different forks (out of 4) even after setting the ionice class to Idle for those 2 kjournald processes.
So what makes rrqm go down and avgqu-sz, await and svctm go up in a bad system? Is it a HDD, is it the card itself, is it some rogue mount option? What is busting the second server?
Thanks in advance for any suggestion!