Mysterious Issue - Apache Hangs with Dozens of 127.0.0.1 Connections

[celiac101]

Without you going into my server via a ticket, my hope is for some advice, as this isn't an emergency.

I am having issues where Apache hangs for no apparent reason, and when it happens the only clue I have as to why is when I run:
netstat -anp | grep 443
I see this:


Notice all the 127.0.0.1 connections and the number "71" of them seems to be consistent. I do run memcached like this:
memcached -u memcached -d -m 30 -l 127.0.0.1 -p 11211
so it is possibly associated, however, in my access logs I also see times where there are dozens of entries in a row like this:

127.0.0.1 - - [08/20/2013:18:24:14 -0000] "GET /.__cpanel__service__check__./serviceauth?sendkey=__HIDDEN__&version=1.2 HTTP/1.0" 200 0 "" "" "-"

Is it possible that a hang on this check is causing Apache to hang? Any ideas?

 

[cPanelMichael]

Hello

When Apache hangs, is it not responding or just taking a slower time to respond? Do you notice any error messages in the Apache error log when this happens?

Thank you.

 

[celiac101]

Good question....here is what I saw in /apache/logs/error_log (I changed my domain names of course):

[Tue Aug 20 11:53:23 2013] [notice] Graceful restart requested, doing restart
[Tue Aug 20 11:54:59 2013] [warn] RSA server certificate CommonName (CN) `mydomain1.com' does NOT match server name!?
[Tue Aug 20 11:54:59 2013] [warn] RSA server certificate CommonName (CN) `www.mydomain2.com' does NOT match server name!?
[Tue Aug 20 11:54:59 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity for Apache/2.7.5 (ModSecurity: Open Source Web Application Firewall) configured.
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: LIBXML compiled version="2.9.0"
[Tue Aug 20 11:55:00 2013] [warn] RSA server certificate CommonName (CN) `mydomain1.com' does NOT match server name!?
[Tue Aug 20 11:55:00 2013] [warn] RSA server certificate CommonName (CN) `www.mydomain2.com' does NOT match server name!?
[Tue Aug 20 11:55:00 2013] [notice] Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips DAV/2 mod_bwlimited/1.4 configured -- resuming normal operations

 

[cPanelMichael]

I don't see anything abnormal in the Apache error log output. You could open a support ticket if the issue persists, but I don't see anything related to cPanel that would be causing Apache to hang. You may also want to temporarily disable memcached to rule it out as the cause of the problem.

Thank you.

 

[celiac101]

Well, if you believe that your "graceful" server restarts that bring my server to its knees for a full 2-3 minutes multiple times a day are normal, then I guess you might think nothing is wrong here.

I believe that I solved the issue, and that it was due to my log settings in WHM - Home » Service Configuration » Apache Configuration. I switched my log settings to Piped Log Configuration and so far I haven't been paged today with a down server. Why this isn't the default setting I have no idea.

 

[cPanelMichael]

Graceful restarts for Apache are normal every 2 hours and every 24 hours due to the default statistics processing intervals. You can see this by running a command such as:

grep "Graceful restart" /usr/local/apache/logs/error_log

However, your websites should not stop loading and Apache should not hang after a graceful restart. There is more information on this feature, and using Pipped Logging at:

Apache Restarting Every 2 Hours

Thank you.