Excessive Disk Activity Since Last cPanel Update

[jman_]

cPanel 11.25.0-R42404 - WHM 11.25.0 - X 3.9
CENTOS 5.4 i686 xen on vps
Linux Kernel: 2.6.18.8

Ever since the 2001-01-28 nightly update (/scripts/upcp on a cron), disk activity has been spiked. No changes had been made to the system other than the auto-update.

It may be Apache, as I am seeing it restart many times, around every 20 minutes. Aside from missing favicons & other trivial listings, this is the only serious-looking error in the logs (8 of them just this morning)

[Sat Jan 30 08:52:51 2010] [error] (9)Bad file descriptor: apr_socket_accept: (client socket)

Unfortunately my kernel is too low a version (you need 2.6.20+ for certain accounting features), so I'm having a hard time showing for sure that Apache is causing the extra disk activity. iotop won't run, for example, and neither atop nor sar won't show anything regarding disk.

Has anyone else out seen this increased activity since the last update, and know how to stop it?

Thanks!

 

[jman_]

Ah, should have looked up that error first. Seems Apache has a bug regarding graceful stop/start. Did a hard stop, restarted, error-b-gone, disk IO back down.

 

[jman_]

Guess I spoke too soon, the excessive disk activity is still occurring.

Is there any way to track which processes are causing it?

 

[jman_]

Anyone? I was averaging around 400 disk IO operations a second before the update. Now it's tripled.

 

[jman_]

Have been suspecting Apache. Has anyone else seen a spike in disk IO since the 28th update?

 

[cPanelDon]

Have been suspecting Apache. Has anyone else seen a spike in disk IO since the 28th update?

What is the full Apache version and MPM?

# /usr/local/apache/bin/httpd -V

What update are you referring to from the 28th (of January)?

How many accounts and domains are on the server?

What is the server load associated with the disk activity?

 

[cPanelDon]

What update are you referring to from the 28th (of January)?

To clarify the question, there was no new update from January 28th other than in EasyApache that would require a manual run of EasyApache to rebuild Apache for a patch to mono (if using mono/mod_mono):
cPanel Downloads - Latest cPanel/WHM Builds
cPanel Change Log

Easyapache 4990, 2010-01-28 09:11:34
Patch mono for https://bugzilla.novell.com/show_bug.cgi?id=571131

 

[jman_]

Apache 2.2.11 (Unix), MPM is prefork.
Average disk IO was 1,500 ops/sec. CPU & Memory were well within tolerance.
There are some 20 accounts with 80+ domains, but traffic is not appreciably heavy.

The update was /scripts/upcp, it ran 2010-01-28 @ 02:34, and the disk io alerts (which come from my host, and are generated every two hours) commenced the next hour.

Have opened a support case. cPanel's reply indicated stats gathering and associated graceful restarts of Apache were causing the problem. cPanel v11.25 was affected. They were working on a fix.

I was told that in the short term, I could use the "Statistics Software Configuration" section of WHM to adjust how often stats are collected, and also limit when stat runs occur. Changing the frequency from 2 hours to 6, and turning off collection for the next four hours appears to have stopped the excessive IO, and the alerts.

 

[ritontor]

WOAH, I am having basically the exact same issues - inexplicable leap in resource usage, Apache seems to keep on trying to restart (the minute I run an Easyapache build the currently running Apache dies), and all this started after a recent update. I'm going to submit this thread to the guys running my ticket at the moment, there's DEFINITELY something screwy afoot, I'm glad it's not just me!

 

[cPanelDon]

WOAH, I am having basically the exact same issues - inexplicable leap in resource usage, Apache seems to keep on trying to restart (the minute I run an Easyapache build the currently running Apache dies), and all this started after a recent update. I'm going to submit this thread to the guys running my ticket at the moment, there's DEFINITELY something screwy afoot, I'm glad it's not just me!

For internal reference, please let me know the ticket ID number of your support request. Thank you in advance.