Server is failing to asynchronously serve files? Extremely bad performance when embedded videos are present on page.

[ap-cnet]

I am in the process of slowly setting up a replacement cPanel/WHM server to move all of hosting to from a server that was compromised last month. Every time i get close to prime time for the full migration another test seems to fail. This time we deployed a brand new WordPress site to the new server rather than deploying it on our old server, and immediately we had problems. The first ~30% of the front page's source code loads, and then it hangs for 5 minutes before finishing.

In my testing I found that there was a locally hosted video file embedded on the front page and it took roughly 5 minutes to download. Sure enough, removing the embed causes the site to load nearly instantly.

In the interim we loaded this site on the old cPanel server as the customer was hot to have it up. It works flawlessly as expected on there, video embed and all.

So I started poking around with firefox dev tools and recording performance and network requests, but nothing seems out of the ordinary and I don't see any sort of error listed. After more testing, it seems like up until i request a page with the video embedded, the site works fine. I can load up a few tabs and it all works. As soon as that video file is requested, ALL requests to that domain are hung up and waiting on the video to finish downloading. All subsequent tabs, etc. If I go over to another machine while the first machine is still loading the site it seems to work ok until once again I request a page with the embedded video. So it seems that per client, everything works until the video file is requested and then suddenly all requests for that client are somehow queued up behind the video.

I have no idea what I may have done to cause this behavior, and it just seems highly bizarre, but I would really like to get it sorted out as we really need to get off that old server as soon as possible.

 

[cPRex]

Hey there,

I'm not sure I'm going to be of much assistance on this one, since the type of content you are serving won't be related to any cPanel settings.

One thing that may be worth checking is that you have a similar Apache and PHP configuration on the new machine as on the old one. You can actually move the entire EasyApache configuration over to the new system directly in the WHM >> Transfer Tool with just a few clicks, which is outlined in our documentation here:

Transfer Tool | cPanel & WHM Documentation

This interface copies multiple accounts from a remote server to your cPanel & WHM server.

docs.cpanel.net

Ensuring those configurations match would be a good first step in troubleshooting this issue. I'm sure other users will comment as well, but that's where I'd start.

 

[ap-cnet]

I actually did transfer all configs, but I made a few changes:

• PHP 70 and 71 removed, PHP 80 added
• mod_security2 added (but also tested with it removed)
• mod_mpm_worker removed and mod_mpm_prefork added
• mod_suexec and mod_suphp removed and mod_ruid2 added

So.. for the sake of completeness, after doublechecking the EA config I went ahead and made the new server use the same EA settings as the old server, switching back to mpm_worker and suexec/suphp. The issue seems to be resolved on that config. I then went and started re-enabling those settings one at a time and now it seems I am back to where I started config wise but suddenly everything is working as expected. Server is on prefork with ruid2 and mod_sec enabled and its no longer blocking the page load even on a hard refresh.

Something must have been broken with either ruid2 or mpm_prefork the first time it was configured but now its better? Tried it on multiple endpoints and multiple browsers and it appears all is well.

Curious to know what might have happened.

 

[cPRex]

Based on your description I'm not completely sure what could have happened with that. If you see something similar happen in the future it would be good to see if there are any logs written to the /etc/apache2/logs/error_log file on the system that may provide more details.

 

[ap-cnet]

I did check the error logs at the time but there was nothing that seemed to match up with what I was experiencing.

Parsing through it again now, the only entries I see from Friday that are not expected are quite a few "Request exceeded the limit of 10 internal redirects" but I am not sure which site that references on the server and the client IP for those log entries are not my IP, so i think it was some other site/request. In fact, there aren't any error log entries logged for a connection to my IP. This server has a bunch of old internal sites that were migrated as test subjects and it wouldn't surprise me if one of those is entirely broken but still getting hit by crawlers.

 

[cPRex]

The "exceeded" error you mention would usually happen due to a misconfiguration inside the .htaccess files on the machine, but you'd see that specific error in the browser if that was the case. No matter what, I am glad things are working well at this point.