Slow requests for random static files

[AGY]

I have an odd situation here. I'm migrating accounts to a new server and have found that most of the time loading sites from the new server will result in a long wait for one or two random static files, taking between 15 and 60 seconds, with some edge cases taking much longer. Tcpdump shows the server receiving the request but apache doesn't see it until after the delay, then serves it normally without any abnormal logging.

Current Server:

/etc/redhat-release:CentOS release 6.10 (Final)
/usr/local/cpanel/version:11.82.0.14
/var/cpanel/envtype:kvm
CPANEL=release

New Server:

/etc/redhat-release:CentOS Linux release 7.6.1810 (Core)
/usr/local/cpanel/version:11.82.0.14
/var/cpanel/envtype:xen hvm
CPANEL=release

Both servers have the ConfigServer package of cxs, csf, lfd, and mod_security.

I have 8 CPUs and 16 GB of RAM and plenty of fast disk and network capacity. Nothing I've found points to a performance issue. I can load the same site (and others) on another browser and computer at the same time and get rapid response (though with the hanging file most of the time), and the system is lightly loaded.

I am running the current EasyApache 4 packages with mpm_prefork, php 7.2, and php-fpm.

The problem exhibits itself on multiple sites, both Wordpress and Drupal, the only correlation being a decent number of files requested. These sites load fine on the old server. The files that hang are typically javascript files, though image files occasionally hang as well - note that the hangs are all on static files, and the exact file seems fairly random. I am shift clicking the reload button to ignore browser cache, and have seen this happen in Chrome, Firefox, and Safari, from multiple physical locations, and several computers.

I believe I have narrowed the area of interest to between the network interface and apache. I tunneled to the new server with ssh -D for a SOCKS proxy to eliminate networking gear, firewalls, and all the bits in-between as culprits. For good measure I restarted the server, shutdown csf and lfd, and whitelisted IPs as well to no effect.

I've tweaked every apache, php, and php-fpm parameter I could find to no avail, except, that turning KeepAlive off in Apache seemed to significantly reduce the occurrences - but did not totally eliminate the issue. Adjusting keep alive parameters had no effect. I don't think php or php-fpm are the issue as these are static files that apache should serve directly, though I have carefully compared configurations between the servers, tweaked every setting I could there, and set logging to debug.

Running apache with LogLevel set to debug showed nothing noticeably wrong. No log entries appear for the file(s) until it is finally served, and then the log entries are normal. I ran tcpdump on the server while running strace against all apache processes (and any children spawned) and found that the dump show the http request coming in, but the file did not appear in the apache trace at all. If strace is running the delay never ends and the file is not served until/unless I cancel the strace, then the file eventually is served.

The setup with Xen and Xentools doesn't seem to expose much functionality to ethtool:

# ethtool eth0
Settings for eth0:
Link detected: yes


Interface stats show no issues:


eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet <snip> netmask 255.255.255.248 broadcast <snip>
ether <snip> txqueuelen 1000 (Ethernet)
RX packets 35496058 bytes 3013947102 (2.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40361583 bytes 122823435445 (114.3 GiB)
TX errors 0 dropped 1 overruns 0 carrier 0 collisions 0


Nothing in dmesg, secure, messages, etc.

Any ideas?

 

[cPanelMichael]

Hello @AGY,

Could you open a support ticket so we can take a closer look at the affected server to check for any obvious causes? You can post the ticket number here once it's opened and we'll link this thread to it.

Thank you.

 

[AGY]

Ticket 13365125.

 

[AGY]

Disabling the Comodo mod_security rules 210080 or 210090 remedied the problem, but likely disabled part of the bigger ruleset as they were part of the initialization. This turned out to be an issue with the /var/cpanel/secdatadir/nobody-ip.pag file, and zeroing it out and restarting apache has fixed the problem.