SOLVED "File not found." instead of custom 404 file with php-fpm enabled

[PeteS]

@cPanelMichael Still would like some cPanel commentary on this.

(Could the >9 minute issue we related to the webmail being set to refresh every 10 minutes *I think*?)

 

[cPRex]

Hey there! Just to recap, our concern is now that the system only works for 9 minutes after an Apache restart? Is that what we're seeing?

Can you post the exact data you're using in the include file(s) so I can do some testing with this?

 

[PeteS]

Thank for stepping back into this...

Hey there! Just to recap, our concern is now that the system only works for 9 minutes after an Apache restart? Is that what we're seeing?

Can you post the exact data you're using in the include file(s) so I can do some testing with this?

Not quite. It's not related to an Apache restart at all. The 9 minutes is in reference to the last time a browser makes connection to the wbmail page (no login required, just access to the page).

I don't know what you are referring to when you asked about an include file.

This quote from my 4/29/20 post sums up what appears to be the case for any domain configured as described (see that post for details): "So what appears to be the case is that if PHP-FPM is enabled, and ProxyErrorOverride is enabled (to allow custom 404 pages via .htaccess), info is set and cached in the browser that becomes stale after 9 minutes and then the connection error happens. If the browser cache is cleared, or the credential expires naturally in the browser after a long time, then access works again. Additionally, once a working connection is established, it will work indefinitely as long as the page is refreshed every <9 minutes."

Steps to replicatet:
1- turn on PHP-FPM for a domain
2- set 404.php error page in .htaccess
3- go to example.com/non-existant.php (you will get "File not found")
4- enable ProxyErrorOverride under the "Pre Main Include" or "Post VirtualHost Include" in WHM Home » Service Configuration » Apache Configuration » Include Editor
5- repeat step 3 (404.php now displays as expected)
6- go to webmail.example.com (you may or may not get the error, depending on previous access to the page)
7- if you get the error, clear cache and it will load the page
8- wait more than 9 minutes (at least in my server's case, not sure if the 9 minutes is universal) and the error will appear/reappear
9- as long ar you continue to refresh each <9 minutes, connection is maintained, otherwise it is lost

Please let me know if you can confirm this on a test server.

 

[cPRex]

Thanks for the clear details - let me do some testing with this and I'll get back with you soon.

 

[cPRex]

So I did the testing and after waiting the 9-10 minutes I get a 401 error. What process are you taking to "reset" that connection to remove the error?

 

[PeteS]

So I did the testing and after waiting the 9-10 minutes I get a 401 error. What process are you taking to "reset" that connection to remove the error?

Thanks for looking into this!

That's the part about something being set in the browser's cache... You must either clear the browser cache, or wait (X number of hours, users reported "over night") for the credential to expire naturally in the browser, then access is restored again. Access will be maintained as long as it is refreshed every <9 minutes.

Clearly, setting ProxyErrorOverride to on interferes with the normal interaction between webmail and the browser. (I wonder what else it might mess up, potentially.) If that setting is being "abused" in some way as a hack for the PHP-FPM/404 page issue then it should be abandoned as a "solution." But if that is the case, then we need to find out what needs to change with PHP-FPM to allow proper execution of .htaccess code (the real problem here, as I see it). Similarly, I'm not in favor of some "hacky" solution in .htaccess either, because I'd like to not have a special PHP-FPM version of code in them for only some accounts. That would turn into a maintenance nightmare. There are several threads on here about this marked "solved," but that's not really the case AFAIK, as it turned out with this one last year.

TL;DR I'd like to get to the bottom of this, not just "make it work."

 

[cPRex]

I gotcha - and I've confirmed that as well.

I'll make a case with the developers and update this thread soon.

 

[cPRex]

I've been doing some more research on this and our documentation at Advanced Apache Configuration | cPanel & WHM Documentation says the following:

"We strongly recommend that you do not set the ProxyErrorOverride directive to On in a global include file. This may cause unexpected behavior."

I added the "ProxyErrorOverride On" to a specific domain using the userdata includes outlined here:

Modify Apache Virtual Hosts with Include Files | cPanel & WHM Documentation

Use the command line interface to add include files that modify the content of the virtual hosts in your Apache configuration.

docs.cpanel.net

and that actually is not working properly. So, instead of making a case for the original problem, which seems to be working as intended as it is expected to break the proxy subdomains system, I've filed a slightly different case with our developers, CPANEL-36086, so they can look into that behavior.

I hope that helps, but I'm sorry I don't have a good solution at this time.

 

[PeteS]

@cPRex Actually this helps a great deal! (I should have researched ProxyErrorOverride myself rather than just take the advice given, even though it was a cPanel staff member.) I agree that we should abandon that usage. I suggest noting this in the other treads that recommend this solution.

I see in that doc that " On servers that run PHP-FPM, you cannot use the ErrorDocument directive unless..." I can see how that is find for a use case with one, or very few accounts, but not otherwise. It's not really maintainable solution. In my case I have no desire to customize the vhost for individual domains, and keep that synced with PHP-FPM. So that's out...

It just seems wrong for PHP-FPM to break the normal usage of .htaccess for .php files. Would you agree? Is that the focus of CPANEL-36086? Other than monitoring for the case number in change logs (which is kind of obsure and also after the fact), is there a way to follow that? Can you at least flag it i your system to have you update this thread about it, so we will get notified?

My solution for now is to not use PHP-FPM if I care about custom php error pages (which I generally set up). Not a stellar solution, though.

 

[cPRex]

It just seems wrong for PHP-FPM to break the normal usage of .htaccess for .php files. Would you agree?

Nope - the way that PHP-FPM handles that is normal, which is why we need that custom include in the first place. The focus of the case is to why adding the customization to an individual vhost isn't working as expected.

I'm monitoring that on my end for sure and I'll post something as soon as the developers have had a chance to look.

 

[PeteS]

Nope - the way that PHP-FPM handles that is normal, which is why we need that custom include in the first place. The focus of the case is to why adding the customization to an individual vhost isn't working as expected.

Ok, I will accept your statement, but can you point me to docs that detail why it would be normal or desirable for FPM to prevent ErrorDocument in .htaccess from being honored for php files? I have read the FPM docs in the past (but not the one you cited above about the interaction of FPM, the ProxyErrorOverride directive, and vhost includes. (Sorry, I misunderstood what you meant about includes the firs time - I was thinking php include files. Ha!) I can see that I need to go back and re-read current FPM docs. There are also a number of changes in WHM as well since I last read up on it.

More specifically, what I'm saying seems "wrong" is that enabling FPM provides such a terse unhelpful "File not found." message. Wouldn't it be good to have the ability to customize that message (per account, and at least server-wide) in WHM? I haven't found a way to do that manually either, but need to look more.

Having to set that any time FPM is enabled for a php site that depends on ErrorDocument is cumbersome. For sites like a Wordpress site, a prime candidate for FPM, this is not an issue since WP handles 404 beautifully. My strategy will now be to not use FPM by default (most of my sites are built in php and expect to use ErrorDocument), but to enable FPM for Wordpress (and similar) sites. Then either add the vhost include for other sites when needed (once it works), or re-write how they handle 404s.

Speaking of WHM changes and managing FPM on a per site basis, MultiPHP Manager no longer allows a sort on the PHP-FPM column. Ugh... why? (I believe we used to be able sort that column.)

Docs questions:
Is there any more detail on what the differnce in behavior is referenced by? "If you set the virtual host’s ProxyErrorOverride directive to On, the server’s error behavior will differ from servers that do not run PHP-FPM and those that do not set the ProxyErrorOverride directive." found here?

Here it says "Include files with local overrides cause the system to permanently disable the Force HTTPS Redirects option in cPanel’s Domains interface (cPanel >> Home >> Domains >> Domains)." This is under "Apply to all virtual hosts on the system" but I want to confirm whether turning on this directive for individual vhosts would have the same effect on force HTTPS for that domain. Hopefully not.

Thanks for the help!

 

[cPRex]

I wish I knew! That one actually isn't up to us, but is a PHP-FPM thing in general. Here's some more generic, non-cPanel details on that where other users run into the same thing:

Apache 2.4 + PHP-FPM, catching error pages

Here is my vhost file: <VirtualHost *:80> ServerName awesome.dev ## Vhost docroot DocumentRoot "/var/www/awesome" ## Directories, there should at least be a declaration for /v...

stackoverflow.com

I'll check into the column sort thing on older versions and let you know what I find. We can still set up older versions for testing in our private environment for just such an occasion.

The difference mentioned is just that non-PHP-FPM boxes don't need that specified in the configuration in order for the ErrorDocument directive to work properly.

 

[cPRex]

Well sure enough - I created a cpanel version 90 machine and checked the page there and you could sort the PHP-FPM column, so I've created case CPANEL-36096 with our developers to look into that.

Thanks for your great feedback!

 

[PeteS]

Thanks for the link, I'll check it out. This whole thing has gone sideways from where I thought it should originally, but I'm glad I'm getting my head around it now and feel like it's a better direction.

CPANEL-3096 sounds good. Hope you can get them to put that back in (or fix what they broke ).