Site(s) crash after 20 or so visited clicks on page

[AnthonyG70]

I have been having a issue for the last week or so, or I just noticed it. The server is pretty low usage.

If I am on a site, and say click about 12-15 links it eventually takes down apache. I usually have to wait a few minutes before I can resume. At first I though it was due to CGI scripts on one site, but it's also happening with a PHP site. WHM does not go down, so I can look at apache status, etc.

I have looked into the error logs for the site, with nothing noticeable as to why it goes down. On apache usage I did see a spike of 12 and 17 for two process at same time on site, with maybe another dozen or less processes going on. I noticed this before a system reconfig (the partitions were too low), and thought it would be fixed by a fresh install...but it is still happening.

Any ideas on what to check? I have a bit of output to put here, but am looking to see if it's a configuration problem or a hardware issue. The server is about 3-years old (4g, 2.13 xeon quad, 500g drive, dell 860). Any suggestions on what else to check? The clean install went by the book on setup for security and optimization for best practices.

# php -v
PHP 5.3.20 (cli) (built: Jan 20 2013 12:35:07) 
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2012 Zend Technologies
    with Suhosin v0.9.33, Copyright (c) 2007-2012, by SektionEins GmbH

# httpd -l
Compiled in modules:
  core.c
  mod_authn_file.c
  mod_authn_default.c
  mod_authz_host.c
  mod_authz_groupfile.c
  mod_authz_user.c
  mod_authz_default.c
  mod_auth_basic.c
  mod_include.c
  mod_filter.c
  mod_deflate.c
  mod_log_config.c
  mod_logio.c
  mod_env.c
  mod_expires.c
  mod_headers.c
  mod_unique_id.c
  mod_setenvif.c
  mod_version.c
  mod_proxy.c
  mod_proxy_connect.c
  mod_proxy_ftp.c
  mod_proxy_http.c
  mod_proxy_scgi.c
  mod_proxy_ajp.c
  mod_proxy_balancer.c
  mod_ssl.c
  prefork.c
  http_core.c
  mod_mime.c
  mod_dav.c
  mod_status.c
  mod_autoindex.c
  mod_asis.c
  mod_info.c
  mod_suexec.c
  mod_cgi.c
  mod_dav_fs.c
  mod_negotiation.c
  mod_dir.c
  mod_imagemap.c
  mod_actions.c
  mod_userdir.c
  mod_alias.c
  mod_rewrite.c
  mod_so.c

SuHosin set for 256M (due to WordPress), and mod_qos has been test between 80, 100 and 256 max_clients.

 

[AnthonyG70]

After a bit of digging, I found the following. Just sharing.

Initial look into apache error log did not show any problems, although a look again this morning showed that mod_qos was actually the issue. A limit of 20 requests per IP, and build-up of processes, led to a denial to my IP address. I did have my IP included in the whitelist. I am unsure of why it failed, although I think it may have to do with using MPM Prefork instead of MPM Worker. Error logs show mod_qos does not support MPM Prefork. It was blocking the IP as it is intended to do, but was doing it rather fast.

grep -i mod_qos /usr/local/apache/logs/error_log

[Mon Jan 21 13:02:30 2013] [notice] mod_qos(009): loaded MPM is 'Prefork' but mod_qos should be used with MPM 'Worker' only.
[Mon Jan 21 13:02:30 2013] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/0.9.8e-fips-rhel5 DAV/2 mod_bwlimited/1.4 mod_qos/10.10 configured -- resuming normal operations
[Mon Jan 21 14:17:21 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=20, c=xxx.xxx.xxx.xxx
[Mon Jan 21 14:17:21 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=21, c=xxx.xxx.xxx.xxx
[Mon Jan 21 14:17:21 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=22, c=xxx.xxx.xxx.xxx
[Mon Jan 21 14:41:05 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=20, c=xxx.xxx.xxx.xxx
[Mon Jan 21 14:41:05 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=21, c=xxx.xxx.xxx.xxx
[Mon Jan 21 14:41:05 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=22, c=xxx.xxx.xxx.xxx
[Mon Jan 21 15:15:55 2013] [notice] mod_qos(009): loaded MPM is 'Prefork' but mod_qos should be used with MPM 'Worker' only.

As machine is quad core xeon, I have recompiled using MPM worker and changed to fastcgi. A test led to a higher "click" ratio before failure and identified the page which was denied access (which it did not do before). When mentioning "click" ratio, previously after maybe 22 clicks it failed, although these were rather quick. This time around about 40+ "clicks" were done. Although it is still ignoring my IP on whitelist.

[Mon Jan 21 19:48:20 2013] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/0.9.8e-fips-rhel5 DAV/2 mod_bwlimited/1.4 mod_qos/10.10 mod_fcgid/2.3.6 configured -- resuming normal operations
[Mon Jan 21 19:57:08 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=20, c=xxx.xxx.xxx.xxx
[Mon Jan 21 19:57:08 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=21, c=xxx.xxx.xxx.xxx
[Mon Jan 21 19:57:09 2013] [error] [client xxx.xxx.xxx.xxx] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=22, c=xxx.xxx.xxx.xxx, id=UP3x9UMSt8IAAGl-KrQAAAAQ, referer: http://www.domain.tld/blog/2012/10/10/page_requested_here/
[Mon Jan 21 19:57:09 2013] [error] mod_qos(060): access denied, QS_ClientEventBlockCount rule: max=20, current=22, c=xxx.xxx.xxx.xxx

Hopefully this will help others.