Mysterious Issue - Apache Hangs with Dozens of 127.0.0.1 Connections

celiac101

Well-Known Member
Dec 19, 2012
85
1
8
cPanel Access Level
Website Owner
Without you going into my server via a ticket, my hope is for some advice, as this isn't an emergency.

I am having issues where Apache hangs for no apparent reason, and when it happens the only clue I have as to why is when I run:
netstat -anp | grep 443
I see this:
screenshot_of_apache_443_issue.jpg

Notice all the 127.0.0.1 connections and the number "71" of them seems to be consistent. I do run memcached like this:
memcached -u memcached -d -m 30 -l 127.0.0.1 -p 11211
so it is possibly associated, however, in my access logs I also see times where there are dozens of entries in a row like this:

127.0.0.1 - - [08/20/2013:18:24:14 -0000] "GET /.__cpanel__service__check__./serviceauth?sendkey=__HIDDEN__&version=1.2 HTTP/1.0" 200 0 "" "" "-"

Is it possible that a hang on this check is causing Apache to hang? Any ideas?
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,911
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Hello :)

When Apache hangs, is it not responding or just taking a slower time to respond? Do you notice any error messages in the Apache error log when this happens?

Thank you.
 

celiac101

Well-Known Member
Dec 19, 2012
85
1
8
cPanel Access Level
Website Owner
Good question....here is what I saw in /apache/logs/error_log (I changed my domain names of course):
Code:
[Tue Aug 20 11:53:23 2013] [notice] Graceful restart requested, doing restart
[Tue Aug 20 11:54:59 2013] [warn] RSA server certificate CommonName (CN) `mydomain1.com' does NOT match server name!?
[Tue Aug 20 11:54:59 2013] [warn] RSA server certificate CommonName (CN) `www.mydomain2.com' does NOT match server name!?
[Tue Aug 20 11:54:59 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity for Apache/2.7.5 (ModSecurity: Open Source Web Application Firewall) configured.
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: APR compiled version="1.4.8"; loaded version="1.4.8"
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: PCRE compiled version="8.21 "; loaded version="8.21 2011-12-12"
[Tue Aug 20 11:54:59 2013] [notice] ModSecurity: LIBXML compiled version="2.9.0"
[Tue Aug 20 11:55:00 2013] [warn] RSA server certificate CommonName (CN) `mydomain1.com' does NOT match server name!?
[Tue Aug 20 11:55:00 2013] [warn] RSA server certificate CommonName (CN) `www.mydomain2.com' does NOT match server name!?
[Tue Aug 20 11:55:00 2013] [notice] Apache/2.2.25 (Unix) mod_ssl/2.2.25 OpenSSL/1.0.0-fips DAV/2 mod_bwlimited/1.4 configured -- resuming normal operations
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,911
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
I don't see anything abnormal in the Apache error log output. You could open a support ticket if the issue persists, but I don't see anything related to cPanel that would be causing Apache to hang. You may also want to temporarily disable memcached to rule it out as the cause of the problem.

Thank you.
 

celiac101

Well-Known Member
Dec 19, 2012
85
1
8
cPanel Access Level
Website Owner
Well, if you believe that your "graceful" server restarts that bring my server to its knees for a full 2-3 minutes multiple times a day are normal, then I guess you might think nothing is wrong here.

I believe that I solved the issue, and that it was due to my log settings in WHM - Home » Service Configuration » Apache Configuration. I switched my log settings to Piped Log Configuration and so far I haven't been paged today with a down server. Why this isn't the default setting I have no idea.
 

cPanelMichael

Technical Support Community Manager
Staff member
Apr 11, 2011
47,911
2,233
363
cPanel Access Level
DataCenter Provider
Twitter
Graceful restarts for Apache are normal every 2 hours and every 24 hours due to the default statistics processing intervals. You can see this by running a command such as:

Code:
grep "Graceful restart" /usr/local/apache/logs/error_log
However, your websites should not stop loading and Apache should not hang after a graceful restart. There is more information on this feature, and using Pipped Logging at:

Apache Restarting Every 2 Hours

Thank you.