Apache down: 'lynx: Can't access startfile http://localhost/whm-server-status'

[David_spm]

I was just on one server and looking at apachetop when suddenly all sites went down and when I tried to restart apache I got this:

Looking up localhost
Making HTTP connection to localhost
Sending HTTP request.
HTTP request sent; waiting for response.
HTTP/1.1 302 Found
Data transfer complete
HTTP/1.1 302 Found
Using https://localhost/whm-server-status
Looking up localhost
Making HTTPS connection to localhost
Retrying connection without TLS.
Looking up localhost
Making HTTPS connection to localhost
Alert!: Unable to make secure connection to remote host.

lynx: Can't access startfile http://localhost/whm-server-status

No chages, updates or configurations have been run at all. I have spent the last hr reading related threads here and have tried the following with no success:

/scripts/upcp --force

httpd -t (syntax ok)

checked for syn flood attacks - all fine

Plenty of resources, RAM & disk space available

Restarting both HTTPD and PHP-FPM via WHm console seems to bring sites back but only briefly

I also tried this (but it also did nothing):

/scripts/rebuildhttpdconf
Sorry, configuration data has not been successfully stored.
Please execute the following commands:

/usr/local/cpanel/bin/apache_conf_distiller --store-data --defaults
touch /var/cpanel/conf/apache/success

Execute the apache_conf_distiller without any flags to see its full usage.
-bash-4.1$ sudo /usr/local/cpanel/bin/apache_conf_distiller --store-data --defaults

Distilled successfully
-bash-4.1$ sudo touch /var/cpanel/conf/apache/success

A swift response would be greatly appreciated as Im out of ideas here and have 30 sites offline right now :|

 

[David_spm]

here's something else:

sudo /scripts/restartsrv_apache
Waiting for “httpd” to start ……waiting for “httpd” to initialize ………finished.

Service Status
httpd (/usr/sbin/httpd -k start) is running as root with PID 6591 (pidfile+/proc check method).

Startup Log
[Thu Dec 06 07:04:48.523852 2018] [so:warn] [pid 6588:tid 47129664890208] AH01574: module status_module is already loaded, skipping

Log Messages
[Thu Dec 06 07:04:48.962164 2018] [mpm_worker:notice] [pid 6591:tid 47129664890208] AH00292: Apache/2.4.37 (cPanel) OpenSSL/1.0.2q mod_bwlimited/1.4 configured -- resuming normal operations
[Thu Dec 06 07:04:48.705601 2018] [:notice] [pid 6588:tid 47129664890208] ModSecurity for Apache/2.9.2 (ModSecurity: Open Source Web Application Firewall) configured.
[Thu Dec 06 06:55:51.615411 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/RESPONSE-980-CORRELATION.conf"] [line "37"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): GET or HEAD Request with Body Content."] [tag "event-correlation"] [hostname "www.example.com"] [uri "/index.php"] [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8"], referer: https://www.example.com/
[Thu Dec 06 06:55:51.211763 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "30"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "www.example.com"] [uri "/"] [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8"], referer: https://www.example.com/
[Thu Dec 06 06:55:51.210719 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Warning. Match of "rx ^0?$" against "REQUEST_HEADERS:Content-Length" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "143"] [id "920170"] [rev "1"] [msg "GET or HEAD Request with Body Content."] [data "247"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"] [tag "CAPEC-272"] [hostname "www.example.com"] [uri "/"] [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8"], referer: https://www.example.com/

Dec 6 07:04:48 server sudo: david : TTY=pts/0 ; PWD=/home/david ; USER=root ; COMMAND=/scripts/restartsrv_apache

httpd started successfully.

but I dont think these modsecurity warnings should be causing an Apache outage?

 

[David_spm]

In apache error_logs Im seeing things like this:

[Thu Dec 06 08:08:05.141717 2018] [mpm_worker:alert] [pid 797:tid 140122596484864] (11)Resource temporarily unavailable: AH03142: apr_thread_create: unable to create worker thread

but thats only when I restart apache and its running (briefly) until it crashes again minutes later.

then Im seeing other errors like this:

[Thu Dec 06 08:15:31.785302 2018] [proxy_fcgi:error] [pid 21009:tid 47421194983168] [client 46.229.168.142:52974] AH01071: Got error 'Primary script unknown\n'

[Thu Dec 06 08:14:51.504584 2018] [ssl:warn] [pid 21001:tid 47421001491808] AH01909: server.myserver.com:443:0 server certificate does NOT include an ID which matches the server name

All my httpd settings are at the defaults though and there has never been any reason to change them:

StartServers 5
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 10
</IfModule>

ServerLimit 256
MaxRequestWorkers 150
MaxConnectionsPerChild 10000
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 768
Timeout 300

Im also using autossl, and for a long time without any issues

 

[David_spm]

been trying some more things, it seems that port 80 keeps closing after httpd is restarted and also its blocked to localhost:

sudo netstat -tulpn | grep 80
tcp        0      0 0.0.0.0:2080                0.0.0.0:*                   LISTEN      19826/cpdavd - acce
udp        0      0 fe80::225:90ff:fe77:c824:123 :::*                                    21553/ntpd         
udp        0      0 fe80::225:90ff:fe77:c825:123 :::*                                    21553/ntpd         

-bash-4.1$ curl -v 127.0.0.1

* Rebuilt URL to: 127.0.0.1/
*   Trying 127.0.0.1...
* connect to 127.0.0.1 port 80 failed: Connection refused
* Failed to connect to 127.0.0.1 port 80: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

-bash-4.1$ telnet localhost 80
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused

-bash-4.1$ sudo nmap -sS 127.0.0.1 -p 80

Starting Nmap 5.51 ( Nmap: the Network Mapper - Free Security Scanner ) at 2018-12-06 10:34 EST
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000070s latency).
PORT   STATE  SERVICE
80/tcp closed http

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds
-bash-4.1$  sudo iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 134K packets, 7860K bytes)
 pkts bytes target     prot opt in     out     source               destination       

Chain POSTROUTING (policy ACCEPT 115K packets, 8260K bytes)
 pkts bytes target     prot opt in     out     source               destination       

Chain OUTPUT (policy ACCEPT 115K packets, 8266K bytes)
 pkts bytes target     prot opt in     out     source               destination       
    0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25,26,465,587 owner GID match 206
  222 11544 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25,26,465,587 owner GID match 12
    0     0 RETURN     tcp  --  *      *       0.0.0.0/0            127.0.0.1            multiport dports 25,26,465,587 owner UID match 201
  166  8632 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25,26,465,587 owner UID match 0
    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25,26,465,587

I cant see anything in iptables that could be causing this though....

Very much open to suggestions right now!

 

[cPanelMichael]

Hello @David_spm,

Can you open a support ticket so we can take a closer look at your system? You can post the ticket number here and we'll link this thread to it.

Thank you.

 

[David_spm]

I went ahead and completed the support ticket even though the prepare server access stage never worked, however I dont have ssh keys on and provided the root pw so you should be able to access the server correct?

 

[David_spm]

I got a pretty swift response from support, Im posting what they said here for future reference in case anyone else needs it:

I think the problem was a custom apache include file that forced all requests to redirect to https

Specifically, in /etc/apache2/conf.d/includes/pre_main_global.conf :

=============
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
=============

Apache looked to be starting up and running perfectly fine -- but cPanel would attempt to connect to it on port 80 and get a particular output. This is done every few minutes as a check to ensure that Apache is up and running properly -- and, if it isn't, cPanel restarts apache automatically.

I believe the forced https redirects were interfering with these service checks (for example, localhost and 127.0.0.1 do not have valid SSL certificates), causing cPanel to detect Apache as being down when it in fact wasn't -- triggering a hard stop and start, which would bring apache down and back up, over and over and over again.

For a little while, I temporarily disabled the monitoring, and things seemed to be alright.

I went ahead and modified pre_main_global.conf to strip that section, and I've re-enabled monitoring, and am watching to see if Apache remains up or not.

After removing those rules, it looks like cPanel was able to properly detect that apache is responsive. Apache hasn't been restarted or gone down since.

##########
Hm ok, is that not a valid rewrite rule?
##########

The rule is perfectly valid, and it does exactly what you would expect -- which is to force a redirect for any incoming request to https.

The problem is that cPanel (and also apache itself) connect to http://localhost / http://127.0.0.1

Since "localhost" and "127.0.0.1" can't possibly have valid SSLs, the rule interfered with those connections.

For example, like this:

==============
[20:51:57 server root@10905649 ~]cPs# curl localhost:80
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://localhost/">here</a>.</p>
</body></html>
============

Or this:

==============
[20:45:58 server root@10905649 ~]cPs# apachectl fullstatus
ELinks: SSL error
==============

Or this:

================
[20:51:48 server root@10905649 ~]cPs# lynx --dump localhost/whm-server-status

Looking up localhost
Making HTTP connection to localhost
Sending HTTP request.
HTTP request sent; waiting for response.
HTTP/1.1 302 Found
Data transfer complete
HTTP/1.1 302 Found
Using https://localhost/whm-server-status
Looking up localhost
Making HTTPS connection to localhost
Retrying connection without TLS.
Looking up localhost
Making HTTPS connection to localhost
Alert!: Unable to make secure connection to remote host.

lynx: Can't access startfile http://localhost/whm-server-status
==================


Apache itself was able to run fine -- but any port 80 connections were forced to reconnect on port 443.


################
Why would this just happen now?
################

When I logged in, I checked the timestamps on the include file that was causing the issue. It was modified this morning:

==================
[20:53:32 server root@10905649 ~]cPs# stat /etc/apache2/conf.d/includes/pre_main_global.conf
File: `/etc/apache2/conf.d/includes/pre_main_global.conf'
Size: 509 Blocks: 8 IO Block: 4096 regular file
Device: 803h/2051d Inode: 14293732 Links: 1
Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-12-06 08:22:44.249566194 -0500
Modify: 2018-12-06 08:22:44.249566194 -0500
Change: 2018-12-06 08:22:44.249566194 -0500
==================

I suspect the rules were only recently added.

So that seems to have fixed it BUT I didnt add those rewrite rules and it looks like the time they were added was when I ran apachetop. Does that add rules itself?

This is solved now anyway.

 

[cPanelMichael]

So that seems to have fixed it BUT I didnt add those rewrite rules and it looks like the time they were added was when I ran apachetop. Does that add rules itself?

=============
RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
=============

Hi David,

I checked, but was unable to find any feature in cPanel & WHM that would automatically add those specific rewrite rules to the /etc/apache2/conf.d/includes/pre_main_global.conf file. It looks like the intent of those rules is to force the use of SSL. Does anyone else have root access to the server? Or, are you using any third-party plugins or addons? Note that in addition to manually editing that file via the command line, it's also possible for someone with root access to WHM to add entries to that file using WHM » Service Configuration » Apache Configuration » Include Editor.

Thank you.

 

[David_spm]

OK thanks, no-one else has any kind of access. I had some problems with autossl not renewing certs on two sites a few weeks ago which I tried to fix and I think I might have added those rewrite rules but like I said that was weeks ago so it doesn't explain why these apache problems happened just now. Thanks for the input though.

 

[cPanelMichael]

I had some problems with autossl not renewing certs on two sites a few weeks ago which I tried to fix and I think I might have added those rewrite rules but like I said that was weeks ago so it doesn't explain why these apache problems happened just now.

Hi @David_spm,

Here are a couple of cases included in the most recent EA4 update that likely lead to the conflict with the custom rewrite rules you enabled:

• ea-apache2-config
  • CPANEL-24111 - Gloval DCV passthrough breaks for hostname certificates
  • EA-8023 - Modify default,vhost template in ea-apache2-config-runtime for HTTPS redirect feature support

In particular, EA-8023 modified the vhost.default configuration and was released in preparation for the HTTPS Redirect feature planned for cPanel & WHM version 78:

# diff -u vhost.default.previous vhost.default
--- vhost.default.previous    2018-11-01 18:17:59.000000000 -0500
+++ vhost.default    2018-12-03 13:48:40.000000000 -0600
@@ -1,6 +1,13 @@
 
 <VirtualHost[% FOREACH ipblock IN vhost.ips %] [% ipblock.ip %]:[% ipblock.port %][% END %]>
   ServerName [% wildcard_safe(vhost.servername) %]
+  [% IF vhost.redirect_to_ssl -%]
+  <IfModule rewrite_module>
+    RewriteEngine On
+    RewriteCond %{HTTPS} !=on
+    RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
+  </IfModule>
+  [% END -%]
 [% IF vhost.serveralias_array.size -%]
 [% FOREACH alias IN vhost.serveralias_array -%]
   ServerAlias [% alias %]

Thank you.