Brotli, Gzip correct order

[Zoltan Szabo]

Dear All,

In a nutshell:
How to set apache to favour Brotli over Gzip if both available!

Story + Troubleshooting:
Today I have enabled brotli via WHM => EasyApache => Apache modules => mod_brotli
Online test show status okay: Brotli Test | Brotli compression check

So far so good, but in chrome 72 dev tools I still see "content-encoding: gzip"
The Chrome default request header is: "accept-encoding: gzip, deflate, br"

I have changed the request header with a "ModHeader" named chrome extension.
If its set to "br" it serves well with brotli if its gzip it server with "gzip".
Here I tried to change the request header order to "accept-encoding: br, gzip"
And its still GZIP <= Here is the problem

I can clearly state that with the default apache Gzip (optimise website) and mod_brotli together prefers Gzip over Brotli which is a false logic, I want it on the other way!

Can anyone help me how to configure apache to favour Brotli over Gzip if both available.

 

[Zoltan Szabo]

Just to avoid any confusion here, my application doesn't pre compress any source. (None)
So its not the case that Brotli avoides double compression coused by gzip pre compression.

But I do beleive double compression avoidance could still be a problem. e.g. Gzil/deflate runs first then Brotli doesn't as it avoides double compression, so it just stays Gzip. (I might be wrong with this assumption - please correct me)

Any idea how to solve this ?

 

[cPanelMichael]

Hello @Zoltan Szabo,

Can you open a support ticket so we can take a closer look at this issue? You can post the ticket number here and we'll link this thread to it.

Thank you.

 

[Zoltan Szabo]

Hello @Zoltan Szabo,

Can you open a support ticket so we can take a closer look at this issue? You can post the ticket number here and we'll link this thread to it.

Thank you.

Dear @cPanelMichael ,

Before opening a support ticket I tried to troubleshoot it myself. I think I have a solution and also found a bug.

I think the problem is with the inclusion order of the WHM brotli config file and the Cpanel optimise config file.
See article here: Setting up Brotli Compression on your WebServer
"Note that you need to specify the Brotli config first, or Apache will use gzip in preference to Brotli."

Means Brotli needs to be first than Gzip/Deflate

Temporaly solution:
1. Keep mod_brotli installed (in WHM)
2. Disable Optimise website (In Cpanel)
3. Add Gzip / Deflate to each of the running websites .htaccess which is loaded long after Brotli init.

This way it works very well, but this is still a kind of bug in Cpanel, by time you need to include Gzip or Brotli or both option in optimize website! Please upvote it here: Configure Brotli per-account

I also discovered that webmanifest file zipping is somehow missing from default Gzip and also Brotli config file
(application/manifest+json) MIME also needs to be added otherwise webmanifest not zipped!

@cPanelMichael can you let me know where can I report this bug?

Bye,
Zoli

 

[cPanelMichael]

Hello @Zoltan Szabo,

Our defect report form is available on the link below:

cPanel Customer Portal

Let me know the ticket number assigned to the report once it's opened and I'll update this thread with the outcome.

Thank you.

Update: Here's a quote from the support ticket explaining the outcome:

I reviewed our internal documentation and determined that enabling brotli compression as a global default configuration was a decision that was intentionally made by our developers. Therefore in this case this behavior does not warrant a bug report, as it is an intended feature.

Thank you.

 

[Zoltan Szabo]

Hello @Zoltan Szabo,

Our defect report form is available on the link below:

cPanel Customer Portal

Let me know the ticket number assigned to the report once it's opened and I'll update this thread with the outcome.

Thank you.

Hello @cPanelMichael

The Request ID is: 11678007

Thank you,

 

[frederic legrand]

is there any update on that bug?

 

[cPanelLauren]

This was never deemed an issue, it was an intentional decision as noted in the response above. There is a feature request for per-account Brotli here: https://features.cpanel.net/topic/configure-brotli-per-account

 

[philsward]

This was never deemed an issue

So deflate will always serve first even if brotli is enabled?

 

[philsward]

This literally makes Brotli unnecessary...

It's like giving us two Ford Mustangs: one from 1992 and one from 2013 and saying "You can look at the 2013 model, but you'll only ever get to drive the 1992 model."

Unless I'm missing something, we should be driving the 2013 Mustang and falling back to the 1992 Mustang when the 2013 breaks.

 

[philsward]

Notes:
* Mileage may vary. I'm on CentOS7, cPanel 90.0.16
* This assumes you have full control over the server ie sudo / root
* I think I got everything needed. It may need additional tweaks from here.

How to fix:

1) Navigate to WHM Include Editor:

Home »Service Configuration »Apache Configuration »Include Editor

2) Click the dropdown for Pre Main Include and choose All Versions

3) Check for and Delete the following lines

<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
</IfModule>
<IfModule mod_setenvif.c>
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
</IfModule>

Discovery: The offending line causing all of my problems was:

<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
</IfModule>

For whatever reason, it enforces gzip regardless of where or how you set Brotli. It's possible this line was left-over from a previous migration, or my hosting company inserted it for me. I honestly don't know if this line is added by cPanel.

4) Update -> Restart Apache -> Yes

5) SSH into the server

6) Create and edit deflate.conf

sudo vi /etc/apache2/conf.d/deflate.conf

7) Paste the following into deflate.conf

<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/x-javascript application/javascript application/json application/x-font-ttf application/vnd.ms-fontobject image/x-icon

#Don't compress content which is already compressed
SetEnvIfNoCase Request_URI \
\.(gif|jpe?g|png|swf|woff|woff2|gz|br) no-gzip dont-vary

# Make sure proxies don't deliver the wrong content
Header append Vary Accept-Encoding env=!dont-vary
</IfModule>

<IfModule mod_setenvif.c>
# Netscape 4.x has some problems...
BrowserMatch ^Mozilla/4 gzip-only-text/html
# Netscape 4.06-4.08 have some more problems
BrowserMatch ^Mozilla/4\.0[678] no-gzip
# MSIE masquerades as Netscape, but it is fine
# BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
# NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
# the above regex won't work. You can use the following
# workaround to get the desired effect:
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI .(?:gif|jpe?g|png)$ no-gzip dont-vary
</IfModule>

8) Write Quit vi

:wq

9) Edit brotli.conf

sudo vi /etc/apache2/conf.d/brotli.conf

10) Add gz and br to the list of files to not compress

From:

SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png)$ no-brotli

To:

SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|gz|br)$ no-brotli

11) Restart Apache

systemctl restart httpd

12) Done!

Test it out and you should start seeing brotli as the default on files.

Note: If you're using a CMS that pre-compresses css and js files to create css.gz or js.gz, you won't be serving them with Brotli, therefore they will still show up as gzip. This is normal. This is called "double compression" when you try to compress an already compressed file so we're telling both Brotli and the fallback gzip, not to re-compress .gz files. Since they are already gzip (.gz), they will be served as gzip.

If the previous statement leads you down the rabbit hole of wondering if you can pre-compress files with Brotli, the answer is "yes", but it isn't easy unless you know how to code.

 

[cPRex]

Hey there! Up until step 4, I can confirm the pre_main_global.conf is empty by default on new cPanel systems as that area is only used for customizing Apache. As you mentioned, this could have been part of an earlier migration or customizations to the gzip/deflate module provided by your host.

I can't speak for the rest as that isn't something we test on our end, but that's great info!

 

[philsward]

Thanks @cPRex

Might bring this up as an issue (I'm not messing with it)

The rules possibly need some work, but overall, the current implementation of Brotli isn't very good. It should be the default if enabled, falling back to gzip for older browsers.

 

[cPRex]

With the current configuration as noted by our developers above, there isnt' an option to make it a global default besides performing the customizations you mentioned earlier. I did check the earlier case and there isn't currently a plan to adjust that behavior.

 

[philsward]

@cPRex Yeah, if the deflate lines aren't in the global by default, then it must be a hosting specific line, which means OOTB this should work for most folks.

I do recommend however, suggesting that "if" deflate is enabled, to create its own .conf file with generic settings to serve .gz files, just like brotli is adding it's own brotli.conf file with recommended settings. This would probably help out a lot of folks.

 

[philsward]

Update: If you're still seeing Content Encoding: gzip when serving PHP, check to see if zlib is enabled for each respective php version. If this is enabled, I've found it will force gzip before br. Try disabling it.

The "Optimize" setting for each account on the server seems to make no difference.