servtastic

Member
Sep 3, 2020
19
1
3
USA
cPanel Access Level
Root Administrator
Hey All,

I'm setting up a new instance on digital ocean and am running into odd behavior when I install/turn on nginx. [8gig / 4 core optimized centos 8]

Steps:
Spin up droplet, do setup tweaks, install cpanel from the command line
cpmove a test site from another server, repoint test site to new server via cloudflare (dns only)
running an internal apache bench against the test site (500 req, 50 concurrent) I get 4000 requests per second out of the box - changing to apache event it's nearly 5000
easy install ea-nginx, rerun tests and it's 400 req per second - plus ab chokes in the last 10% of the run, going from 11ms to 52001ms - because of nginx
uninstall nginx and ab is back to 5000 req per second.

Running ab on my machine against the test site I was getting ~140 req per sec - cpanel nginx installed, it's ~14

I tried enginetron and it bumped the req per sec to just over 200 so I know nginx will help, I just can't get cpanel's nginx to work right.

Any suggestions of what I might be missing (cpanel support can't duplicate the issue - which is extra weird)

Steve
(Side note: I disabled enginetron to test but the disable killed apache and I was never able to get http back, so destroyed the instance.)
 

cPDavidL

Linux Analyst II
Oct 15, 2012
79
18
133
cPanel Access Level
Root Administrator

servtastic

Member
Sep 3, 2020
19
1
3
USA
cPanel Access Level
Root Administrator
Hey,

Thanks for the follow up, but this isn't about high traffic. or tweaking config. This is about ea-nginx choking the server when installed and making connections 10x slower.
As mentioned I installed enginetron and it bumped req per sec from 140 to 200 (ish) so I can see that an nginx cache helps.
With a default Droplet, default cpanel install - adding ea-nginx (easyapache 4) kills the http response times
I've done this test multiple times on 2 different default installs and get the same result - adding ea-nginx = 10 slower.
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
5,463
698
313
cPanel Access Level
Root Administrator
That's definitely interesting, although I can't say I've seen other similar reports of that type of slowness. If you'd like us to do some more in-depth testing, it would be a good idea to submit a ticket to our team so we can see one of the systems you are working with and examine that in real-time.
 

servtastic

Member
Sep 3, 2020
19
1
3
USA
cPanel Access Level
Root Administrator
Fired up a new instance, documented all steps to install cp, ran tests with and without nginx and am getting the exact same issue.
Put in a ticket - let's see if they believe me this time.
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
5,463
698
313
cPanel Access Level
Root Administrator
Our team was able to trace the issue to a problem with the proxy_cache_lock setting causing issues with the testing. There were several additional things mentioned in the ticket as well, but at least we were able to determine the issue isn't slowness that is present by default :D
 

servtastic

Member
Sep 3, 2020
19
1
3
USA
cPanel Access Level
Root Administrator
I have decided to give up on nginx for now - which is a shame but it currently adds more problems than solutions.

No http2 out of the box. Really weird but there it is.

Clearing cache. I use WP Rocket (WPR) on every site and I *really* like rocket-nginx (a tweak where nginx delivers the WPR cache file directly.) However, without the direct delivery, clearing WPR cache doesn't clear nginx cleanly.

Not easily customizable. I have cpnginx on another box and can change main nginx configs from a UI. Having to manage nginx from the command line is a time suck as it is yet another structure / software to learn. eg. If I want to change the cache time from 60 mins to 5 secs, I should be able to do that from a UI - this is cpanel after all.

And with enough cpu (digital ocean, optimized, 8gig/4core), plus php8, Apache is fast enough. Using the event MPM, for me a typical WP site loads in 500ms and even a fully loaded woocommerce site with a dozen woo plugins and jetpack can be under a second (in the USA.) I'll live with that for now and look at nginx once cp96 comes out for centos7 (or not)
 

servtastic

Member
Sep 3, 2020
19
1
3
USA
cPanel Access Level
Root Administrator
... and a follow up issue with nginx as currently configured (if I understand the docs correctly.)

Nginx accepts the inbound connection and responds, but does not pass any logs back to apache.

If that is the case, then mod_security (inside of apache) does not get any information for that connection. Which means mod_security rules do not get applied to nginx cache connections.

Example:
Connection 1: legitimate IP requests a file / pages. File/page gets proxied by nginx.
Connection 2-10+: bad ips, non valid requests, should be blocked by apache. Do not get blocked by nginx which happily serves the file/page to requestors that should be blocked.

Again, if I am reading the docs right, isn't Nginx bypassing mod_security in some cases by not logging back to apache?
 

servtastic

Member
Sep 3, 2020
19
1
3
USA
cPanel Access Level
Root Administrator
Informal testing on the old instance I am about to destroy (digitalocean, 4core/8gig/basic.)
I have confirmed the behavior mentioned in my previous post.

1. apache only - logging works, no http2 module, no http2 (confirmed with tools.keycdn.com)
2. apache only - logging works, installed http2 via easyapache / http2 working (confirmed with tools.keycdn.com)
3. install ea-nginx via easyapache (see errors below)
4. nginx proxy, no logging in apache logs, no http2 (confirmed with tools.keycdn.com)

I would appear Nginx does not provide http2 (out of the box) and negates apache providing http2 when nginx is enabled. Plus does not log to apache logs, in theory negating some mod security / LFD rules.

Notes:
Nginx was provisioned (eventually) by EasyApache but was restarted by chkserv with the following warning
May 15 14:36:43 ns1.servtastic.net nginx[28344]: nginx: [warn] could not build optimal server_names_hash, you should increase either server_names_hash_max_size: 1024 or server_names_hash_bucket_size: 128; ignoring server_names_hash_bucket_size