Apache's mod_status causes high IOWait.

[virtualreality]

Recently I've read Apache's mod_status causes high IOWait. I've also found Apache's mod_status, among other server software, can raise IOWait in our server.

mod_status.c is listed live in our server, when I check the loaded Apache modules in our server with /usr/local/apache/bin/httpd -l,

and also, status_module (static) is listed live in our server, when I check with /usr/local/apache/bin/httpd -M.

I want to disable and totally remove Apache's mod_status in the Apache loaded modules' lists.

Is there a cPanel server command for us to disable and remove Apache's mod_status?

I've searched the cPanel forum for some cPanel server command to disable Apache's mod_status but haven't found one.

Anyone knows how to totally disable Apache's mod_status on the cPanel server?

Any help is highly expected and appreciated!!!

 

[virtualreality]

Or, what is exactly the Apache loaded modules folder in the cPanel server that I can disable and remove Apache mod_status module?

This morning, I received 4 emails titled as "High 1 minute load average alert":

Time:                    Mon Jan 12 12:04:26 2015 -0600
1 Min Load Avg:          2.67
5 Min Load Avg:          1.51
15 Min Load Avg:         0.71
Running/Total Processes:  4/96

Time:                    Mon Jan 12 11:03:48 2015 -0600
1 Min Load Avg:          4.47
5 Min Load Avg:          1.98
15 Min Load Avg:         0.81
Running/Total Processes:  8/98

Time:                    Mon Jan 12 10:03:48 2015 -0600
1 Min Load Avg:          2.39
5 Min Load Avg:          1.10
15 Min Load Avg:         0.50
Running/Total Processes:  3/96

Time:                    Mon Jan 12 09:03:39 2015 -0600
1 Min Load Avg:          2.75
5 Min Load Avg:          1.17
15 Min Load Avg:         0.49
Running/Total Processes:  4/97

And the vmstat.txt says:

Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id  wa  st
  2  0     0   948280   0    54476   0    0    14    1    0    0  0  0  15  84  0

See wa is 84. It is VERY high!

In one way, the hourly high IOwait is related to Apache's mod_status??

In another way, the hourly high IOwait is also raised by Chkservd, as my previous post Chkservd hangs our website almost every hour spoke about. In this thread, the Chkservd HANG problem is still not really fixed even after I shut down the Chkservd functions in the cPanel server as much as possible. --- I did send a support ticket for this problem and the cPanel technician who received my ticket did try to do some fixing work on the problem, but because I already shut down most of the Chkservd functions to reduce the current Chkservd HANG occurrences, so the ticket had to be canceled with my request. Therefore, simply I didn't know how to answer cPanelMichael's question in the thread: Would you mind sharing the resolution on this thread in-case other users experience a similar issue? Sorry Michael, the Chkservd problem still remains. And now there is another one (Apache's mod_status) that needs to be disabled.

 

[cPanelMichael]

Hello

Mod_Status is a base module in Apache 2.4. To clarify, are you experiencing high disk I/O on your system as a result of mod_status or is this a preventative measure? Please review this thread for information on troubleshooting load issues:

Troubleshooting High Loads

Thank you.

 

[virtualreality]

Thank you Michael for helpful response!

I use Apache 2.2. For the full compatibility of our long-lived website system and the Php version based on the website system we cannot upgrade the Apache version to 2.4 now.

I've checked disk I/O and in most hours the disk I/O looks normal and not high except the First 4-5 Minutes of almost every hour yesterday and this morning today. In one way the high IOwait is raised by Chkservd and/or another cPanel server monitoring system. After I shut down most of the Chkservd functions, I've found the Chkservd Hang occurrences have reduced, and the hourly high IOwait occurrences also have reduced, from almost every hour to about every 4~5 hours. However, yesterday and today, the high IOwait occurrences have resumed.

 

[cPanelMichael]

Do you notice actual reduced performance or are you simply worried about the disk I/O usage numbers? The following thread offers some advice on how to obtain additional information about the usage:

Troubleshooting High Loads

Thank you.

 

[virtualreality]

Thank you Michael for help!

High IOwait and high CPU load can be caused by several things. I'm still looking for some effective ways trying to solve the frequent high server IOwait and CPU load, and still looking for a way to disable Apache mod_status hoping to help reduce the IOwait and CPU load.

 

[cPanelMichael]

You could disable the following entry in /usr/local/apache/conf/httpd.conf:

<Location /whm-server-status>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from 127.0.0.1 ::1
</Location>

Information on modifying the Apache configuration file

However, I'm not sure that's the real cause of your disk I/O.

Thank you.

 

[virtualreality]

Thank you Michael for help!

I've alreaday done with your method in the main Apache Config file httpd as the following:

#ExtendedStatus on
#<Location /whm-server-status>
#SetHandler server-status
#Order deny,allow
#Deny from all
#Allow from 127.0.0.1 ::1
#</Location>

However, the IOwait is still high frequently. The Apache status_module still functions at some extent? I've also checked CSF/lfd. All those IOwait-related CSF/lfd settings (such as System Integrity Checking, processeses sanity check, etc.) were disabled earlier.

I checked /etc/my.cnf and increased tmp_table_size to 256M and max_heap_table_size to 256M, hoping to reduce the disk's in and out and the disk's load. I also largely increased key_buffer_size hoping to help lower some IOwait.

In the Apache server, I reduced ServerLimit and MaxClients, lowered StartServers, MinSpareServers, MaxSpareServers, KeepAliveTimeout and MaxKeepAliveRequests. The MaxRequestsPerChild is 2000 and it's a reasonable value.

Although with all the above done already, the high server IOwait and high CPU load occur at the beginning of almost every hour or every two hours.

Something is confusing.

The high IOwait and CPU load should be caused by several factors, and one of the factors is queueprocd?queueprocd can also raise the high IOwait and CPU load?

Is queueprocd frequently increasing Writes to disk?? I'll keep watching it for some time.

 

[virtualreality]

queueprocd is frequently increasing Writes to the disk. Now this is a major cause of frequent high IOwait on the cPanel server.

Just now, I've received the "High 1 minute load average alert" again:

Time:                    Thu Jan 15 08:02:26 2015 -0600
1 Min Load Avg:          3.83
5 Min Load Avg:          1.42
15 Min Load Avg:         0.62
Running/Total Processes: 1/72

And the vmstat:

Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  1  0      0 930668      0  59352    0    0    13     1    0    0  1  0 15 84  0

cPanel Team, how to fix this?

 

[cPanelMichael]

In "WHM Home » Server Configuration » Tweak Settings", under the "Stats and Logs" tab, I presume the following option is already enabled:

"Use delayed sql inserts in the eximstats process to reduce the number of disk writes"

Also, in "Tweak Settings", under the "System" tab, see if the following option is enabled:

"Conserve memory - Conserve memory at the expense of more CPU usage and disk I/O."

Beyond that, feel free to open a support ticket using the link in my signature so we can take a closer look. It's possible the issue is with your hardware. You can post the ticket number here so we can update this thread with the outcome.

Thank you.

 

[virtualreality]

Thank you Michael for help!

I've sent a support ticket using the link in your signature. The support ticket number is 5973315.

In WHM's Tweak Settings, I select "Use delayed sql inserts in the eximstats process to reduce the number of disk writes." And I never select "Conserve memory". I do consider CPU usage and disk I/O are more important than memory. I feel in a VPS environment, what often makes trouble is high disk I/O and CPU usage.

Our web hosting company uses SSD in our VPS hosting. The hardware is fine, stable and fast, because previously we did not encounter this high IOwait problem in our cPanel server for several years. We've continuously used the same web hosting company hosting for about 5 years until now. Their main node and network are fast, stable and excellent. I must say this after using their hosting service for about 5 years continually. The high IOwait problem does not come from the web hosting company.

I feel since we upgraded our WHM/cPanel to WHM 11.44.1 (build 18) (stable version), there appears the high IOwait problem in our cPanel server.

Now continuously, some new "High 1 minute load average alert" until now today:

Time:                    Thu Jan 15 12:04:31 2015 -0600
1 Min Load Avg:          2.48
5 Min Load Avg:          1.28
15 Min Load Avg:         0.60
Running/Total Processes: 1/73

Time:                    Thu Jan 15 11:04:28 2015 -0600
1 Min Load Avg:          2.42
5 Min Load Avg:          1.50
15 Min Load Avg:         0.75
Running/Total Processes: 1/70

Time:                    Thu Jan 15 10:04:27 2015 -0600
1 Min Load Avg:          3.38
5 Min Load Avg:          1.74
15 Min Load Avg:         0.83
Running/Total Processes: 1/72

Time:                    Thu Jan 15 09:03:54 2015 -0600
1 Min Load Avg:          2.05
5 Min Load Avg:          0.91
15 Min Load Avg:         0.44
Running/Total Processes: 2/69

And the :

Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  0  0      0 840828      0 138636    0    0    13     1    0    0  1  0 15 84  0

Sincerely expect some better customizations and update need to be done for queueprocd and/or some other cPanel system components.

 

[virtualreality]

And during the high IOwait, our WHM, cPanel and website are all stuck. It is really frustrated.

 

[virtualreality]

Some new "High 1 minute load average alert" until now today:

Time:                    Thu Jan 15 15:05:37 2015 -0600
1 Min Load Avg:          4.84
5 Min Load Avg:          2.68
15 Min Load Avg:         1.24
Running/Total Processes: 1/75

Time:                    Thu Jan 15 14:05:14 2015 -0600
1 Min Load Avg:          4.75
5 Min Load Avg:          2.67
15 Min Load Avg:         1.23
Running/Total Processes: 5/80

Time:                    Thu Jan 15 13:04:46 2015 -0600
1 Min Load Avg:          2.61
5 Min Load Avg:          1.39
15 Min Load Avg:         0.69
Running/Total Processes: 1/72

The vmstat:

Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  2  1      0 863248      0 107904    0    0    13     1    0    0  1  0 15 84  0

 

[virtualreality]

Hi cPanel Friends,

Now today, things are clear:

The frequent high IOwait problem DOES NOT come from the cPanel systems.

The frequent high IOwait problem has NOTHING to do with the cPanel systems.


I am really sorry for suspecting something in the cPanel caused the frequent high IOwait, at first. I apologize for my suspicion of some cPanel component causing the frequent high IOwait. The cPanel systems are excellent.

I really don't know what caused the frequent high IOwait problem, but since the VPS was migrated to a new host node last night, I've checked until now there are no more IOwait problems. Now, the VPS performance is smooth, robust, and stable.

 

[virtualreality]

And, the cPanel support team members have given me a lot of professional support.

Thank you cPanel from my heart for your excellent support!

 

[cPanelMichael]

I really don't know what caused the frequent high IOwait problem, but since the VPS was migrated to a new host node last night, I've checked until now there are no more IOwait problems. Now, the VPS performance is smooth, robust, and stable.

I am happy to see the issue has been addressed. Thank you for updating us with the outcome.