Mysterious Issue - Apache Hangs with Dozens of 127.0.0.1 Connections

Without you going into my server via a ticket, my hope is for some advice, as this isn't an emergency.

I am having issues where Apache hangs for no apparent reason, and when it happens the only clue I have as to why is when I run:
netstat -anp | grep 443
I see this:


Notice all the 127.0.0.1 connections and the number "71" of them seems to be consistent. I do run memcached like this:
memcached -u memcached -d -m 30 -l 127.0.0.1 -p 11211
so it is possibly associated, however, in my access logs I also see times where there are dozens of entries in a row like this:

127.0.0.1 - - [08/20/2013:18:24:14 -0000] "GET /.__cpanel__service__check__./serviceauth?sendkey=__HIDDEN__&version=1.2 HTTP/1.0" 200 0 "" "" "-"

Is it possible that a hang on this check is causing Apache to hang? Any ideas?

 

Good question....here is what I saw in /apache/logs/error_log (I changed my domain names of course):

Code:

[Tue Aug 20 11:53:23 2013] [notice] Graceful restart requested, doing restart
[Tue Aug 20 11:54:59 2013] [warn] RSA server certificate CommonName (CN) `mydomain1.com' does NOT match server name!?
[Tue Aug 20 11:54:59 2013] [warn] RSA server certificate CommonName (CN) `www.mydomain2.com' does NOT match server name!?
[Tue Aug 20 11:54:59 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity for Apache/2.7.5 (ModSecurity: Open Source Web Application Firewall) configured.
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: LIBXML compiled version="2.9.0"
[Tue Aug 20 11:55:00 2013] [warn] RSA server certificate CommonName (CN) `mydomain1.com' does NOT match server name!?
[Tue Aug 20 11:55:00 2013] [warn] RSA server certificate CommonName (CN) `www.mydomain2.com' does NOT match server name!?
[Tue Aug 20 11:55:00 2013] [notice] Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips DAV/2 mod_bwlimited/1.4 configured -- resuming normal operations 

 

Well, if you believe that your "graceful" server restarts that bring my server to its knees for a full 2-3 minutes multiple times a day are normal, then I guess you might think nothing is wrong here.

I believe that I solved the issue, and that it was due to my log settings in WHM - Home » Service Configuration » Apache Configuration. I switched my log settings to Piped Log Configuration and so far I haven't been paged today with a down server. Why this isn't the default setting I have no idea.