Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Apache down: 'lynx: Can't access startfile http://localhost/whm-server-status'

Discussion in 'EasyApache' started by David_spm, Dec 6, 2018 at 6:01 AM.

  1. David_spm

    David_spm Well-Known Member

    Joined:
    May 28, 2017
    Messages:
    49
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    I was just on one server and looking at apachetop when suddenly all sites went down and when I tried to restart apache I got this:

    Looking up localhost
    Making HTTP connection to localhost
    Sending HTTP request.
    HTTP request sent; waiting for response.
    HTTP/1.1 302 Found
    Data transfer complete
    HTTP/1.1 302 Found
    Using https://localhost/whm-server-status
    Looking up localhost
    Making HTTPS connection to localhost
    Retrying connection without TLS.
    Looking up localhost
    Making HTTPS connection to localhost
    Alert!: Unable to make secure connection to remote host.

    lynx: Can't access startfile http://localhost/whm-server-status

    No chages, updates or configurations have been run at all. I have spent the last hr reading related threads here and have tried the following with no success:

    /scripts/upcp --force

    httpd -t (syntax ok)

    checked for syn flood attacks - all fine

    Plenty of resources, RAM & disk space available

    Restarting both HTTPD and PHP-FPM via WHm console seems to bring sites back but only briefly

    I also tried this (but it also did nothing):

    /scripts/rebuildhttpdconf
    Sorry, configuration data has not been successfully stored.
    Please execute the following commands:

    /usr/local/cpanel/bin/apache_conf_distiller --store-data --defaults
    touch /var/cpanel/conf/apache/success

    Execute the apache_conf_distiller without any flags to see its full usage.
    -bash-4.1$ sudo /usr/local/cpanel/bin/apache_conf_distiller --store-data --defaults

    Distilled successfully
    -bash-4.1$ sudo touch /var/cpanel/conf/apache/success

    A swift response would be greatly appreciated as Im out of ideas here and have 30 sites offline right now :|
     
  2. David_spm

    David_spm Well-Known Member

    Joined:
    May 28, 2017
    Messages:
    49
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    here's something else:

    Code:
    sudo /scripts/restartsrv_apache
    Waiting for “httpd” to start ……waiting for “httpd” to initialize ………finished.
    
    Service Status
    httpd (/usr/sbin/httpd -k start) is running as root with PID 6591 (pidfile+/proc check method).
    
    Startup Log
    [Thu Dec 06 07:04:48.523852 2018] [so:warn] [pid 6588:tid 47129664890208] AH01574: module status_module is already loaded, skipping
    
    Log Messages
    [Thu Dec 06 07:04:48.962164 2018] [mpm_worker:notice] [pid 6591:tid 47129664890208] AH00292: Apache/2.4.37 (cPanel) OpenSSL/1.0.2q mod_bwlimited/1.4 configured -- resuming normal operations
    [Thu Dec 06 07:04:48.705601 2018] [:notice] [pid 6588:tid 47129664890208] ModSecurity for Apache/2.9.2 (ModSecurity: Open Source Web Application Firewall) configured.
    [Thu Dec 06 06:55:51.615411 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/RESPONSE-980-CORRELATION.conf"] [line "37"] [id "980130"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5 - SQLI=0,XSS=0,RFI=0,LFI=0,RCE=0,PHPI=0,HTTP=0,SESS=0): GET or HEAD Request with Body Content."] [tag "event-correlation"] [hostname "www.example.com"] [uri "/index.php"] [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8"], referer: https://www.example.com/
    [Thu Dec 06 06:55:51.211763 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "30"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "www.example.com"] [uri "/"] [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8"], referer: https://www.example.com/
    [Thu Dec 06 06:55:51.210719 2018] [:error] [pid 10971:tid 140380729206528] [client 162.158.xx.xxx:52054] [client 162.158.xx.xxx] ModSecurity: Warning. Match of "rx ^0?$" against "REQUEST_HEADERS:Content-Length" required. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "143"] [id "920170"] [rev "1"] [msg "GET or HEAD Request with Body Content."] [data "247"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_HREQ"] [tag "CAPEC-272"] [hostname "www.example.com"] [uri "/"] [unique_id "XAkORwb4GghU0J1Ps7uNZQAAAA8"], referer: https://www.example.com/
    
    Dec 6 07:04:48 server sudo: david : TTY=pts/0 ; PWD=/home/david ; USER=root ; COMMAND=/scripts/restartsrv_apache
    
    httpd started successfully.
    
    but I dont think these modsecurity warnings should be causing an Apache outage?
     
    #2 David_spm, Dec 6, 2018 at 6:11 AM
    Last edited by a moderator: Dec 6, 2018 at 6:29 AM
  3. David_spm

    David_spm Well-Known Member

    Joined:
    May 28, 2017
    Messages:
    49
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    In apache error_logs Im seeing things like this:

    [Thu Dec 06 08:08:05.141717 2018] [mpm_worker:alert] [pid 797:tid 140122596484864] (11)Resource temporarily unavailable: AH03142: apr_thread_create: unable to create worker thread

    but thats only when I restart apache and its running (briefly) until it crashes again minutes later.

    then Im seeing other errors like this:

    [Thu Dec 06 08:15:31.785302 2018] [proxy_fcgi:error] [pid 21009:tid 47421194983168] [client 46.229.168.142:52974] AH01071: Got error 'Primary script unknown\n'

    [Thu Dec 06 08:14:51.504584 2018] [ssl:warn] [pid 21001:tid 47421001491808] AH01909: server.myserver.com:443:0 server certificate does NOT include an ID which matches the server name

    All my httpd settings are at the defaults though and there has never been any reason to change them:

    StartServers 5
    <IfModule prefork.c>
    MinSpareServers 5
    MaxSpareServers 10
    </IfModule>

    ServerLimit 256
    MaxRequestWorkers 150
    MaxConnectionsPerChild 10000
    KeepAlive On
    KeepAliveTimeout 5
    MaxKeepAliveRequests 768
    Timeout 300

    Im also using autossl, and for a long time without any issues
     
  4. David_spm

    David_spm Well-Known Member

    Joined:
    May 28, 2017
    Messages:
    49
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    been trying some more things, it seems that port 80 keeps closing after httpd is restarted and also its blocked to localhost:
    Code:
    sudo netstat -tulpn | grep 80
    tcp        0      0 0.0.0.0:2080                0.0.0.0:*                   LISTEN      19826/cpdavd - acce
    udp        0      0 fe80::225:90ff:fe77:c824:123 :::*                                    21553/ntpd         
    udp        0      0 fe80::225:90ff:fe77:c825:123 :::*                                    21553/ntpd         
    
    -bash-4.1$ curl -v 127.0.0.1
    
    * Rebuilt URL to: 127.0.0.1/
    *   Trying 127.0.0.1...
    * connect to 127.0.0.1 port 80 failed: Connection refused
    * Failed to connect to 127.0.0.1 port 80: Connection refused
    * Closing connection 0
    curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused
    
    -bash-4.1$ telnet localhost 80
    Trying ::1...
    telnet: connect to address ::1: Connection refused
    Trying 127.0.0.1...
    telnet: connect to address 127.0.0.1: Connection refused
    
    -bash-4.1$ sudo nmap -sS 127.0.0.1 -p 80
    
    Starting Nmap 5.51 ( Nmap: the Network Mapper - Free Security Scanner ) at 2018-12-06 10:34 EST
    Nmap scan report for localhost (127.0.0.1)
    Host is up (0.000070s latency).
    PORT   STATE  SERVICE
    80/tcp closed http
    
    Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds
    -bash-4.1$  sudo iptables -t nat -nvL
    Chain PREROUTING (policy ACCEPT 134K packets, 7860K bytes)
     pkts bytes target     prot opt in     out     source               destination       
    
    Chain POSTROUTING (policy ACCEPT 115K packets, 8260K bytes)
     pkts bytes target     prot opt in     out     source               destination       
    
    Chain OUTPUT (policy ACCEPT 115K packets, 8266K bytes)
     pkts bytes target     prot opt in     out     source               destination       
        0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25,26,465,587 owner GID match 206
      222 11544 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25,26,465,587 owner GID match 12
        0     0 RETURN     tcp  --  *      *       0.0.0.0/0            127.0.0.1            multiport dports 25,26,465,587 owner UID match 201
      166  8632 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25,26,465,587 owner UID match 0
        0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 25,26,465,587
    
    I cant see anything in iptables that could be causing this though....

    Very much open to suggestions right now!
     
  5. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,411
    Likes Received:
    1,956
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @David_spm,

    Can you open a support ticket so we can take a closer look at your system? You can post the ticket number here and we'll link this thread to it.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. David_spm

    David_spm Well-Known Member

    Joined:
    May 28, 2017
    Messages:
    49
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    I went ahead and completed the support ticket even though the prepare server access stage never worked, however I dont have ssh keys on and provided the root pw so you should be able to access the server correct?
     
  7. David_spm

    David_spm Well-Known Member

    Joined:
    May 28, 2017
    Messages:
    49
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    I got a pretty swift response from support, Im posting what they said here for future reference in case anyone else needs it:

    I think the problem was a custom apache include file that forced all requests to redirect to https

    Specifically, in /etc/apache2/conf.d/includes/pre_main_global.conf :

    =============
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
    =============

    Apache looked to be starting up and running perfectly fine -- but cPanel would attempt to connect to it on port 80 and get a particular output. This is done every few minutes as a check to ensure that Apache is up and running properly -- and, if it isn't, cPanel restarts apache automatically.

    I believe the forced https redirects were interfering with these service checks (for example, localhost and 127.0.0.1 do not have valid SSL certificates), causing cPanel to detect Apache as being down when it in fact wasn't -- triggering a hard stop and start, which would bring apache down and back up, over and over and over again.

    For a little while, I temporarily disabled the monitoring, and things seemed to be alright.

    I went ahead and modified pre_main_global.conf to strip that section, and I've re-enabled monitoring, and am watching to see if Apache remains up or not.

    After removing those rules, it looks like cPanel was able to properly detect that apache is responsive. Apache hasn't been restarted or gone down since.

    ##########
    Hm ok, is that not a valid rewrite rule?
    ##########

    The rule is perfectly valid, and it does exactly what you would expect -- which is to force a redirect for any incoming request to https.

    The problem is that cPanel (and also apache itself) connect to http://localhost / http://127.0.0.1

    Since "localhost" and "127.0.0.1" can't possibly have valid SSLs, the rule interfered with those connections.

    For example, like this:

    ==============
    [20:51:57 server root@10905649 ~]cPs# curl localhost:80
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html><head>
    <title>302 Found</title>
    </head><body>
    <h1>Found</h1>
    <p>The document has moved <a href="https://localhost/">here</a>.</p>
    </body></html>
    ============

    Or this:

    ==============
    [20:45:58 server root@10905649 ~]cPs# apachectl fullstatus
    ELinks: SSL error
    ==============

    Or this:

    ================
    [20:51:48 server root@10905649 ~]cPs# lynx --dump localhost/whm-server-status

    Looking up localhost
    Making HTTP connection to localhost
    Sending HTTP request.
    HTTP request sent; waiting for response.
    HTTP/1.1 302 Found
    Data transfer complete
    HTTP/1.1 302 Found
    Using https://localhost/whm-server-status
    Looking up localhost
    Making HTTPS connection to localhost
    Retrying connection without TLS.
    Looking up localhost
    Making HTTPS connection to localhost
    Alert!: Unable to make secure connection to remote host.

    lynx: Can't access startfile http://localhost/whm-server-status
    ==================


    Apache itself was able to run fine -- but any port 80 connections were forced to reconnect on port 443.


    ################
    Why would this just happen now?
    ################

    When I logged in, I checked the timestamps on the include file that was causing the issue. It was modified this morning:

    ==================
    [20:53:32 server root@10905649 ~]cPs# stat /etc/apache2/conf.d/includes/pre_main_global.conf
    File: `/etc/apache2/conf.d/includes/pre_main_global.conf'
    Size: 509 Blocks: 8 IO Block: 4096 regular file
    Device: 803h/2051d Inode: 14293732 Links: 1
    Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root)
    Access: 2018-12-06 08:22:44.249566194 -0500
    Modify: 2018-12-06 08:22:44.249566194 -0500
    Change: 2018-12-06 08:22:44.249566194 -0500
    ==================

    I suspect the rules were only recently added.

    So that seems to have fixed it BUT I didnt add those rewrite rules and it looks like the time they were added was when I ran apachetop. Does that add rules itself?

    This is solved now anyway.
     
  8. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,411
    Likes Received:
    1,956
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hi David,

    I checked, but was unable to find any feature in cPanel & WHM that would automatically add those specific rewrite rules to the /etc/apache2/conf.d/includes/pre_main_global.conf file. It looks like the intent of those rules is to force the use of SSL. Does anyone else have root access to the server? Or, are you using any third-party plugins or addons? Note that in addition to manually editing that file via the command line, it's also possible for someone with root access to WHM to add entries to that file using WHM » Service Configuration » Apache Configuration » Include Editor.

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. David_spm

    David_spm Well-Known Member

    Joined:
    May 28, 2017
    Messages:
    49
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Thailand
    cPanel Access Level:
    Root Administrator
    OK thanks, no-one else has any kind of access. I had some problems with autossl not renewing certs on two sites a few weeks ago which I tried to fix and I think I might have added those rewrite rules but like I said that was weeks ago so it doesn't explain why these apache problems happened just now. Thanks for the input though.
     
  10. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,411
    Likes Received:
    1,956
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hi @David_spm,

    Here are a couple of cases included in the most recent EA4 update that likely lead to the conflict with the custom rewrite rules you enabled:

    In particular, EA-8023 modified the vhost.default configuration and was released in preparation for the HTTPS Redirect feature planned for cPanel & WHM version 78:

    Code:
    # diff -u vhost.default.previous vhost.default
    --- vhost.default.previous    2018-11-01 18:17:59.000000000 -0500
    +++ vhost.default    2018-12-03 13:48:40.000000000 -0600
    @@ -1,6 +1,13 @@
     
     <VirtualHost[% FOREACH ipblock IN vhost.ips %] [% ipblock.ip %]:[% ipblock.port %][% END %]>
       ServerName [% wildcard_safe(vhost.servername) %]
    +  [% IF vhost.redirect_to_ssl -%]
    +  <IfModule rewrite_module>
    +    RewriteEngine On
    +    RewriteCond %{HTTPS} !=on
    +    RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
    +  </IfModule>
    +  [% END -%]
     [% IF vhost.serveralias_array.size -%]
     [% FOREACH alias IN vhost.serveralias_array -%]
       ServerAlias [% alias %]
    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    #10 cPanelMichael, Dec 10, 2018 at 9:00 AM
    Last edited: Dec 10, 2018 at 9:20 AM
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice