Excessive Disk Activity Since Last cPanel Update

jman_

Member
Sep 23, 2008
19
0
51
cPanel 11.25.0-R42404 - WHM 11.25.0 - X 3.9
CENTOS 5.4 i686 xen on vps
Linux Kernel: 2.6.18.8

Ever since the 2001-01-28 nightly update (/scripts/upcp on a cron), disk activity has been spiked. No changes had been made to the system other than the auto-update.

It may be Apache, as I am seeing it restart many times, around every 20 minutes. Aside from missing favicons & other trivial listings, this is the only serious-looking error in the logs (8 of them just this morning)

[Sat Jan 30 08:52:51 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)

Unfortunately my kernel is too low a version (you need 2.6.20+ for certain accounting features), so I'm having a hard time showing for sure that Apache is causing the extra disk activity. iotop won't run, for example, and neither atop nor sar won't show anything regarding disk.

Has anyone else out seen this increased activity since the last update, and know how to stop it?

Thanks!
 

jman_

Member
Sep 23, 2008
19
0
51
Ah, should have looked up that error first. Seems Apache has a bug regarding graceful stop/start. Did a hard stop, restarted, error-b-gone, disk IO back down.
 

jman_

Member
Sep 23, 2008
19
0
51
Guess I spoke too soon, the excessive disk activity is still occurring.

Is there any way to track which processes are causing it?
 

jman_

Member
Sep 23, 2008
19
0
51
Anyone? I was averaging around 400 disk IO operations a second before the update. Now it's tripled.
 

jman_

Member
Sep 23, 2008
19
0
51
Have been suspecting Apache. Has anyone else seen a spike in disk IO since the 28th update?
 

cPanelDon

cPanel Quality Assurance Analyst
Staff member
Nov 5, 2008
2,545
12
268
Houston, Texas, U.S.A.
cPanel Access Level
DataCenter Provider
Twitter
Have been suspecting Apache. Has anyone else seen a spike in disk IO since the 28th update?
What is the full Apache version and MPM?
Code:
# /usr/local/apache/bin/httpd -V
What update are you referring to from the 28th (of January)?

How many accounts and domains are on the server?

What is the server load associated with the disk activity?
 

cPanelDon

cPanel Quality Assurance Analyst
Staff member
Nov 5, 2008
2,545
12
268
Houston, Texas, U.S.A.
cPanel Access Level
DataCenter Provider
Twitter

jman_

Member
Sep 23, 2008
19
0
51
Apache 2.2.11 (Unix), MPM is prefork.
Average disk IO was 1,500 ops/sec. CPU & Memory were well within tolerance.
There are some 20 accounts with 80+ domains, but traffic is not appreciably heavy.

The update was /scripts/upcp, it ran 2010-01-28 @ 02:34, and the disk io alerts (which come from my host, and are generated every two hours) commenced the next hour.

Have opened a support case. cPanel's reply indicated stats gathering and associated graceful restarts of Apache were causing the problem. cPanel v11.25 was affected. They were working on a fix.

I was told that in the short term, I could use the "Statistics Software Configuration" section of WHM to adjust how often stats are collected, and also limit when stat runs occur. Changing the frequency from 2 hours to 6, and turning off collection for the next four hours appears to have stopped the excessive IO, and the alerts.
 

ritontor

Member
May 17, 2009
19
0
51
WOAH, I am having basically the exact same issues - inexplicable leap in resource usage, Apache seems to keep on trying to restart (the minute I run an Easyapache build the currently running Apache dies), and all this started after a recent update. I'm going to submit this thread to the guys running my ticket at the moment, there's DEFINITELY something screwy afoot, I'm glad it's not just me!
 

cPanelDon

cPanel Quality Assurance Analyst
Staff member
Nov 5, 2008
2,545
12
268
Houston, Texas, U.S.A.
cPanel Access Level
DataCenter Provider
Twitter
WOAH, I am having basically the exact same issues - inexplicable leap in resource usage, Apache seems to keep on trying to restart (the minute I run an Easyapache build the currently running Apache dies), and all this started after a recent update. I'm going to submit this thread to the guys running my ticket at the moment, there's DEFINITELY something screwy afoot, I'm glad it's not just me!
For internal reference, please let me know the ticket ID number of your support request. Thank you in advance. :)