Figuring out the cause of "hanging" on JS (CSF, mysqltuner, EA4)

[tone]

Since a recent update (of either WHM or CSF), I've noticed significant performance issues. I've done the following:

• Checked with the host (reputable host, sustained and burst speedtests from multiple locations are ~1000-3000 mbps)
• Run EA4 with a fresh profile
• Reset CSF configuration
• Changed cache and cdn options, where applicable
• Run mysqltuner

MySQLTuner
Here's the output of mysqltuner:

root@hostname [~]# perl mysqltuner.pl
 >>  MySQLTuner 1.7.0 - Major Hayden <[email protected]>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[!!] Currently running unsupported MySQL version 10.0.29-MariaDB
[OK] Operating on 64-bit architecture

-------- Log file Recommendations ------------------------------------------------------------------
[--] Log file: /var/lib/mysql/hostname.com.err(12M)
[OK] Log file /var/lib/mysql/hostname.com.err exists
[OK] Log file /var/lib/mysql/hostname.com.err is readable.
[OK] Log file /var/lib/mysql/hostname.com.err is not empty
[OK] Log file /var/lib/mysql/hostname.com.err is smaller than 32 Mb
[!!] /var/lib/mysql/hostname.com.err contains 510 warning(s).
[!!] /var/lib/mysql/hostname.com.err contains 68320 error(s).
[--] 55 start(s) detected in /var/lib/mysql/hostname.com.err
[--] 1) 170221  5:10:26 [Note] /usr/sbin/mysqld: ready for connections.
[--] 2) 170202  2:02:14 [Note] /usr/sbin/mysqld: ready for connections.
[--] 3) 170201  8:39:26 [Note] /usr/sbin/mysqld: ready for connections.
[--] 4) 170115  2:16:05 [Note] /usr/sbin/mysqld: ready for connections.
[--] 5) 161222  7:13:34 [Note] /usr/sbin/mysqld: ready for connections.
[--] 6) 161215 18:37:05 [Note] /usr/sbin/mysqld: ready for connections.
[--] 7) 161215 18:36:53 [Note] /usr/sbin/mysqld: ready for connections.
[--] 8) 2016-12-04 12:54:04 223734 [Note] /usr/sbin/mysqld: ready for connections.
[--] 9) 2016-12-04 03:56:03 2601 [Note] /usr/sbin/mysqld: ready for connections.
[--] 10) 2016-12-03 12:41:18 441908 [Note] /usr/sbin/mysqld: ready for connections.
[--] 47 shutdown(s) detected in /var/lib/mysql/hostname.com.err
[--] 1) 170202  2:02:13 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 2) 170201  8:39:24 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 3) 170115  2:16:03 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 4) 161222  7:13:32 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 5) 161215 18:37:04 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 6) 2016-12-15 18:36:11 223734 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 7) 2016-12-04 12:54:03 2601 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 8) 2016-12-04 03:54:22 441908 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 9) 2016-12-03 12:41:16 354003 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 10) 2016-11-09 02:25:44 763101 [Note] /usr/sbin/mysqld: Shutdown complete

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 997M (Tables: 855)
[--] Data in InnoDB tables: 1G (Tables: 2035)
[--] Data in MEMORY tables: 1M (Tables: 29)
[OK] Total fragmented tables: 0

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[--] There are 612 basic passwords in the list.

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 1d 14h 9m 56s (1M q [13.420 qps], 90K conn, TX: 554G, RX: 478M)
[--] Reads / Writes: 88% / 12%
[--] Binary logging is disabled
[--] Physical Memory     : 31.3G
[--] Max MySQL memory    : 6.9G
[--] Other process memory: 1.2G
[--] Total buffers: 2.8G global + 2.8M per thread (1500 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 2.9G (9.36% of installed RAM)
[OK] Maximum possible memory usage: 6.9G (22.07% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (2/1M)
[OK] Highest usage of available connections: 2% (33/1500)
[OK] Aborted connections: 0.15%  (136/90420)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[OK] Query cache efficiency: 41.5% (958K cached / 2M selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (83 temp sorts / 59K sorts)
[OK] No joins without indexes
[!!] Temporary tables created on disk: 62% (36K on disk / 58K total)
[OK] Thread cache hit rate: 99% (33 created / 90K connections)
[OK] Table cache hit rate: 92% (3K open / 3K opened)
[OK] Open file limit used: 11% (1K/15K)
[OK] Table locks acquired immediately: 99% (982K immediate / 983K locks)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.
[--] Memory used by P_S: 0B
[--] Sys schema isn't installed.

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 8 thread(s).
[--] Using default value is good enough for your version (10.0.29-MariaDB)

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 19.5% (418M used / 2B cache)
[OK] Key buffer size / total MyISAM indexes: 2.0G/290.5M
[OK] Read Key buffer hit rate: 95.3% (4M cached / 200K reads)
[!!] Write Key buffer hit rate: 36.1% (317K cached / 202K writes)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 318.0M/1.8G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (30.188679245283 %): 48.0M * 2/318.0M should be equal 25%
[!!] InnoDB buffer pool <= 1G and Innodb_buffer_pool_instances(!=1).
[--] InnoDB Buffer Pool Chunk Size not used or defined in your version
[OK] InnoDB Read buffer efficiency: 99.97% (373241654 hits/ 373348911 total)
[!!] InnoDB Write Log efficiency: 84.72% (163080 hits/ 192489 total)
[OK] InnoDB log waits: 0.00% (0 waits / 29409 writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/1B
[!!] Aria pagecache hit rate: 62.1% (20M cached / 7M reads)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.

-------- RocksDB Metrics ---------------------------------------------------------------------------
[--] RocksDB is disabled.

-------- Spider Metrics ----------------------------------------------------------------------------
[--] Spider is disabled.

-------- Connect Metrics ---------------------------------------------------------------------------
[--] Connect is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] This is a standalone server.

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/lib/mysql/hostname.com.err file
    Control error line(s) into /var/lib/mysql/hostname.com.err file
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries which have no LIMIT clause
    Performance should be activated for better diagnostics
    Consider installing Sys schema from GitHub - mysql/mysql-sys: The MySQL sys schema
Variables to adjust:
    query_cache_type (=0)
    tmp_table_size (> 256M)
    max_heap_table_size (> 16M)
    performance_schema = ON enable PFS
    innodb_buffer_pool_size (>= 1G) if possible.
    innodb_log_file_size * innodb_log_files_in_group should be equals to 1/4 of buffer pool size (=159M) if possible.
    innodb_buffer_pool_instances (=1)

Example of issue
Here's an example of the problem on a 'simple' Wordpress installation:

1. Enter URL
2. Login to admin
3. Click Posts or Pages, an area where multiple js is loaded (editor) and database lookups are needed, as well as images etc.
4. Watch network in Chrome and see it taking up to 20 seconds to load (or sometimes timeout) mostly js, css, and image files.

I then disable CSF and find absolutely no issue - everything loads in under one second (~50ms).

Loading image files of 20-50 MB each directly from the server (avoiding PHP and database) results in the full speed of my connection being utilised.

EA4 List

Current Profile
The currently installed packages on the server.

Apache 2.4
config

config-runtime

mod_asis

mod_authn_anon

mod_authn_socache

mod_bwlimited

mod_cache

mod_cache_disk

mod_cgid

mod_deflate

mod_env

mod_expires

mod_headers

mod_hostinglimits

mod_mpm_worker

mod_proxy

mod_proxy_fcgi

mod_proxy_http

mod_security2

mod_ssl

mod_suexec

mod_suphp

mod_unique_id

mod_version

tools

PHP 5.6
build

libc-client

pear

php-bcmath

php-bz2

php-calendar

php-cli

php-common

php-curl

php-dba

php-debuginfo

php-devel

php-enchant

php-exif

php-fileinfo

php-fpm

php-ftp

php-gd

php-gettext

php-gmp

php-iconv

php-imap

php-intl

php-ioncube

php-ldap

php-litespeed

php-mbstring

php-mcrypt

php-mysqlnd

php-odbc

php-pdo

php-pgsql

php-posix

php-process

php-pspell

php-snmp

php-soap

php-sockets

php-sourceguardian

php-tidy

php-xml

php-xmlrpc

php-zendguard

php-zip

runtime

PHP 7.0
build

libc-client

pear

php-bcmath

php-bz2

php-calendar

php-cli

php-common

php-curl

php-dba

php-devel

php-enchant

php-exif

php-fileinfo

php-fpm

php-ftp

php-gd

php-gettext

php-gmp

php-iconv

php-imap

php-intl

php-ldap

php-litespeed

php-mbstring

php-mcrypt

php-mysqlnd

php-odbc

php-opcache

php-pdo

php-pgsql

php-posix

php-process

php-pspell

php-snmp

php-soap

php-sockets

php-tidy

php-xml

php-xmlrpc

php-zip

runtime

Others
apr

apr-util

cpanel-tools

documentroot

libmcrypt

modsec-sdbm-util

php-cli

profiles-cpanel

Any suggestions would be greatly appreciated. The server load rarely goes above 1 (E3 1245v2) and RAM usage is ~20% (32GB).

 

[cPanelMichael]

Example of issue
Here's an example of the problem on a 'simple' Wordpress installation:

1. Enter URL
2. Login to admin
3. Click Posts or Pages, an area where multiple js is loaded (editor) and database lookups are needed, as well as images etc.
4. Watch network in Chrome and see it taking up to 20 seconds to load (or sometimes timeout) mostly js, css, and image files.

I then disable CSF and find absolutely no issue - everything loads in under one second (~50ms).

Hello @tone,

The fact that everything works as expected when CSF is disabled suggests an issue with the firewall. Here are a couple of threads on the CSF forums where other users report the same issue:

CSF 7.0.3 -- problem on all VPS servers - ConfigServer Community Forum
Slow loading websites due to Firewall IP blocks - ConfigServer Community Forum

You may want to review the CSF logs to see what activity is occurring to get a better idea of what's reducing website load times.

Thank you.

 

[tone]

Hello @tone,

The fact that everything works as expected when CSF is disabled suggests an issue with the firewall. Here are a couple of threads on the CSF forums where other users report the same issue:

CSF 7.0.3 -- problem on all VPS servers - ConfigServer Community Forum
Slow loading websites due to Firewall IP blocks - ConfigServer Community Forum

You may want to review the CSF logs to see what activity is occurring to get a better idea of what's reducing website load times.

Thank you.

Hello Michael,

Thank you for your helpful reply. It seems one cause was CSF blocking or severely limiting all Cloudflare IPs. Understabily, this caused 522 and similar errors. Whitelisting the Cloudflare IP range has resolved the issue, but it's by no means ideal (from what I understand).

Do you have any opinion my the mysqltuner output?

Regards,

Piers

EDIT: I also noticed that EasyApache4 is 'buggy' compared to EA3. EA3 worked and all parts or modules were compiled, this does not seem to be the case with EA4 (for example, not installing one option for php56 when it's selected). I've run the configuration again and it seems to have worked as intended.

 

[cPanelMichael]

Do you have any opinion my the mysqltuner output?

You could check under "Variables to adjust" output and make some of the recommended changes to see if you notice a performance increase.

EDIT: I also noticed that EasyApache4 is 'buggy' compared to EA3. EA3 worked and all parts or modules were compiled, this does not seem to be the case with EA4 (for example, not installing one option for php56 when it's selected). I've run the configuration again and it seems to have worked as intended.

Feel free to open a new thread with any particular issues you are facing and we can help troubleshoot the problem.

Thank you.