SOLVED [EA-7980] Rewrite rules broken after latest EA4 update

[kdean]

After updating to the EasyApache4 October 31 update. Wordpress Multisite using PHP-FPM no longer can load it's subfolder subsite admin dashboards without getting a bunch of 404s for it's resources. Trying to load other admin pages results in 404s. Turning off PHP-FPM for the domain loads the pages as expected.

It's like the htaccess is no longer being interpreted properly.

 

[kdean]

wwwdomaincom/wp-admin/network & wwwdomaincom/wp-admin/ loads as expected.

wwwdomaincom/subsite/wp-admin/ loads the page content but referenced css, js, etc. does not load with 404s.

wwwdomaincom/subsite/wp-admin/edit.php and other variants return 404s

Disabling PHP-FPM returns to normal functionality with all of the above link formats working.

 

[batesy87]

Also having the same issue and can confirm disabling FPM for the site fixes the issue

 

[AoM_Scott]

I was having a problem with my PHP-FPM status page after last night's update and it looks like WHM have at some point added new RewriteRules to the generated virtualhost config for each website:

    <IfModule proxy_fcgi_module>
        <FilesMatch \.(phtml|php[0-9]*)$>
            SetHandler proxy:unix:/opt/cpanel/ea-php56/.......
        </FilesMatch>

        RewriteCond %{REQUEST_FILENAME} \.php$
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} !-f
        RewriteRule (.*) - [H=text/html]
    </IfModule>

It looks like the idea is to redirect requests for PHP files that do not exist to the standard text/html handler. This might be beneficial from a security point of view but has broken my status page as I named it as a non-existent file (fpm.php) but I will probably have to be more creative with the naming to fix this issue.

Anyway this could also be interfering with the Wordpress plugin or could be completely unrelated. It certainly shouldn't affect resources such as JS, CSS and IMGs as these shouldn't be processed by PHP-FPM unless they are dynamic generated.

Which rewrite rules are you using for the plugin?

 

[cPanelMichael]

Hello Everyone,

Thanks for the reports. We're working to address this issue as soon as possible as part of internal case EA-7980. I'll update this thread with more information as soon as it's available. In the meantime, the temporary workaround is to disable PHP-FPM for the affected domains.

Thank you.

 

[cPanelMichael]

Hello Everyone,

To update, it looks like new rewrite rules added to the virtual host template as part of the <IfModule proxy_fcgi_module> statement for accounts assigned PHP-FPM caused the issue. We should have an update published to address this issue within the next few hours. I'll update this thread again once the update is published.

Thanks!

 

[kdean]

Anyway this could also be interfering with the Wordpress plugin or could be completely unrelated. It certainly shouldn't affect resources such as JS, CSS and IMGs as these shouldn't be processed by PHP-FPM unless they are dynamic generated.

Which rewrite rules are you using for the plugin?

To clarify some. Multisite is a special mode for Wordpress that let's one run multiple wordpress sites off of one set of wordpress files. So, when accessing any subsite in a subfolder of the domain (especially the admin pages), almost everything goes through redirects. Changes to what is passed to the Proxy can definitely mess with things. When PHP-FPM first came out before cPanel had it, the Proxy Match statements were problematic for Wordpress Multisites until newer methods came out.

 

[AoM_Scott]

Thanks for the clarification. We make use of presumably similar rewrites in our app but not all of these URLs were affected (they don't generally have the ".php" extension and it only rewrites to files that actually exist). I just wanted to check I fully understood the affected URLs in-case part of our site was affected without us knowing.

What concerns me more, is how easily (auto)-updates are pushed out which break websites. We use the STABLE build level to try and mitigate this risk but I believe EA4 packages do not have similar levels and are updated when upcp runs. As this generally happens overnight, any bad configuration changes could bring down a website for several hours. This has happened to me before when WHM 68 changed back to default virtualhost templates without any prior warning and broke all our PHP scripts.

Would it not be better to hold updates that need configuration changes so that the admin can check that no websites or customisations are affected? A "Your Apache config needs to be rebuilt" message could be shown or emailed to the admin or "The latest update requires your Apache config to be rebuilt, which may affect your user's websites, please manually update the server." would allow more pro-active monitoring of breaking changes?

 

[kdean]

I received the following in my support ticket.

As a temporary workaround, you can do one of the following:
1. Disable php-fpm for the affected sites
2. Revert ea-apache24-2.4.37 back to ea-apache24-2.4.35
3. Manually modify /var/cpanel/templates/apache2_4/ssl_vhost.default and /var/cpanel/templates/apache2_4/vhost.default to remove the code causing the issue.

old code:
=====

[%- IF vhost.php_fpm %]
   <IfModule proxy_fcgi_module>
       <FilesMatch \.(phtml|php[0-9]*)$>
           SetHandler proxy:unix:[% vhost.php_fpm_socket %]|fcgi://[% wildcard_safe(vhost.servername) %]
          </FilesMatch>
           RewriteCond %{REQUEST_FILENAME} \.php$
        RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} !-f
        RewriteRule (.*) - [H=text/html]
   </IfModule>
[%- END %]

=====

fixed code:
=====

[%- IF vhost.php_fpm %]
   <IfModule proxy_fcgi_module>
       <FilesMatch \.(phtml|php[0-9]*)$>
           SetHandler proxy:unix:[% vhost.php_fpm_socket %]|fcgi://[% wildcard_safe(vhost.servername) %]
          </FilesMatch>
   </IfModule>
[%- END %]

=====

 

[thewebexpert]

Any update? I see this workaround, but what about a proper fix from CPANEL?

 

[cPanelMichael]

Any update? I see this workaround, but what about a proper fix from CPANEL?

Hello @thewebexpert,

The solution is going through testing at this time. I'll update this thread again once that's finished and the update is published.

Thank you.

 

[czjreilly]

Hello @thewebexpert,

The solution is going through testing at this time. I'll update this thread again once that's finished and the update is published.

Thank you.

Is there an estimate for when the solution will be released? This is affecting 2 of our sites and we do not wish to disable FPM at this time as it may affect the other accounts on our server as well. We tried the fix above posted by kdean but it did not resolve the issue for us.

Thanks in advance!

 

[VO Group]

Hi all,

I wanted to share a nasty issue we had today after an cPanel auto update last night. Our client said that they were having issues with being unable to log in to wp-admin. We diagnosed it as a redirection issue and then somehow to do with .htaccess.

After checking .htaccess against the official Wordpress one for multisite there was only a small discrepancy which was to do with our alternate wordpress location (/wp) (see screenshot)

Unfortunately we couldn't get the alternative path going and in the end moved the files to the web root which fixed the site for our customer and allowed them to get their orders out, however it was unsatisfying and we would have preferred to fix the .htaccess or whatever was causing the rewrite rules to fail.

Please if anyone's out there do you have a suggestion?

 

[Sahar B.]

Hi,
Any official update on this?

Only disabling php-fpm worked for me as a temp solution, but this of course is at the expense of performance.

Thanks

 

[michieloosterling]

We had the same problem on two VPSs I manage. Simply turning PHP-FPM off didn't quite fix it for us, as like user VO Group above, pretty much all the WP installs we run (150+) are based on Roots.io Bedrock, which requires a different document root. This is set via .htaccess rules, but those were being overridden by the mod_rewrite additions to the VHOST configurations.
And with 150+ websites broken (links beyond the front page would result in 404s) we rolled back to before the update was applied and turned off all automatic updating (from stable!). Sucks, as we lost data that way, but better than trying for hours to fix something we can't change.
As someone mentioned above, perhaps some better checking is required before applying EasyApache updates to stable?

Either way, look forward to hearing we can turn auto updates for stable on again, if only from a security perspective!

 

[cPanelLauren]

Hi Everyone,


This issue was fixed last night by reverting changes made to ea-apache24-config. You may need to clean the yum cache before running a cPanel update (or just yum update) to resolve this issue.

Thanks!

 

[Zoltan Szabo]

Thank you for the fix.

Now I have 2 questions:
Does PHP FPM conf. stays like this, or we can expect future changes?
Do we have a feature request for accepting EA changes not just auto update in background? (I like to upvote it)

All best
Zoli

 

[cPanelMichael]

Hello Everyone,

Regarding the automatic EA4 package updates, you can set Operating System Package Updates to Never Update or Manual Updates Only in WHM >> Update Preferences if you want to prevent automatic updates to the EA4 RPMs during the nightly cPanel update. This will prevent most automatic EA4 package updates; the exception being that EA4 updates are automatically applied when Daily Updates is set to Automatic and the cPanel update requires a specific package updated (EA4 or system). To handle all updates manually, set both Daily Updates and Operating System Package Updates to Never Update.

I've also linked this thread to internal case CPANEL-19615, which was opened to consider offering a separate option to configure how EA4 package updates are handled.

Thank you.

 

[Dean999]

It may be redundant at this point, but as no-one has actually explained in detail the cause of the problem, I thought I should do that.
As noted above, ea-apache24-2.4.37 added the following rewrite-rule to the httpd.conf configuration file:

RewriteCond %{REQUEST_FILENAME} \.php$
RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_URI} !-f
RewriteRule (.*) - [H=text/html]

The intent of this rule is to mark non-existent (file-not-found) PHP scripts as text/html, so that they generate precisely the same "Not Found" message as other missing files. (This appears to be the only purpose of the rule, AFAIK).

Unfortunately, for this rule to work as intended, it would have to be applied AFTER any other URL rewriting.
But as it is in the httpd.conf file, it gets applied FIRST, before any mod_rewrite rules that may be in a .htaccess file. This is the root of the problem.

As an example, suppose you have a URL scheme like this (for various choices of {clientname}):

www.mywebsite.com/{clientname}/homepage.php

and your .htaccess file internally rewrites it to:

www.mywebsite.com/homepage.php?client={clientname}

The new rule would check whether a PHP file actually exists at the file-system location:
WEBROOT/{clientname}/homepage.php
when actually it is of course here:
WEBROOT/homepage.php

So, not finding it in a {clientname} subfolder, it marks your homepage.php script to-be-handled as 'text/html'.
This marker persists through all further rewrite rules, and (apparently) causes PHP-FPM (but not other kinds of PHP) to return a "File not found" error.

To properly fix the rule so that it works as intended, it would have to be applied last, after all other mod_rewrite rules (wherever they are located) have finished. I'm not 100% sure, but I don't think it's possible to insert a mod_rewrite rule into httpd.conf that will behave like that.

 

[LucasRolff]

Will there be an official statement from cPanel, how this could bypass QA in the first place? It's a rather serious bug that gets released and kills websites around the world