Extremely slow SSL after migration to new server

[jestep]

We're in the process of migrating a bunch of sites to a new linode cpanel server.

Server is setup running CentOS7, 16Gb of RAM.

We wanted MPM ITK with the new server and no longer had the option to use 2.2, so we're using 2.4 and Easyapache4. The old server was using 2.2 and Easyapache 3. This is basically the only difference between the setup of the 2 servers.

Basically the issue right now is when serving over SSL, it is extremely slow on the new server. Basically it takes 15 - 30 seconds just for the SSL handshake alone.

Have checked /proc/sys/kernel/random/entropy_avail and it's not a problem.

Output from connecting via curl from another server:

root@web2 [~]# time curl -Iv https://www.MYSITE.COM
* About to connect() to www.MYSITE.COM port 443 (#0)
*   Trying MY_IP_ADDRESS... connected
* Connected to www.MYSITE.COM (MY_IP_ADDRESS) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=www.MYSITE.COM,OU=PositiveSSL,OU=Domain Control Validated
*       start date: May 02 00:00:00 2016 GMT
*       expire date: May 02 23:59:59 2017 GMT
*       common name: www.MYSITE.COM
*       issuer: CN=COMODO RSA Domain Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: www.MYSITE.COM
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 01 Jun 2016 15:38:43 GMT
Date: Wed, 01 Jun 2016 15:38:43 GMT
< Server: Apache
Server: Apache
< X-Frame-Options: SAMEORIGIN
X-Frame-Options: SAMEORIGIN
< Set-Cookie: PHPSESSID=v705s1cud67u90um3d75vl8dn5; expires=Fri, 03-Jun-2016 15:38:43 GMT; Max-Age=172800; path=/
Set-Cookie: PHPSESSID=v705s1cud67u90um3d75vl8dn5; expires=Fri, 03-Jun-2016 15:38:43 GMT; Max-Age=172800; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
Pragma: no-cache
< Set-Cookie: MYSITE=YTo0OntzOjY6InNvdXJjZSI7aTo4O3M6ODoiY2FtcGFpZ24iO047czo3OiJjb250ZW50IjtOO3M6NDoidGVybSI7Tjt9; expires=Fri, 01-Jul-2016 15:38:43 GMT; Max-Age=2592000; path=/; secure
Set-Cookie: MYSITE=YTo0OntzOjY6InNvdXJjZSI7aTo4O3M6ODoiY2FtcGFpZ24iO047czo3OiJjb250ZW50IjtOO3M6NDoidGVybSI7Tjt9; expires=Fri, 01-Jul-2016 15:38:43 GMT; Max-Age=2592000; path=/; secure
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8

<
* Connection #0 to host www.MYSITE.COM left intact
* Closing connection #0

real    0m26.640s
user    0m0.034s
sys     0m0.022s

Apache configuration:
TraceEnable Off
ServerSignature Off
ServerTokens ProductOnly
FileETag None

<Directory "/">
AllowOverride All
Options ExecCGI FollowSymLinks IncludesNOEXEC SymLinksIfOwnerMatch
</Directory>

StartServers 5
<IfModule prefork.c>
MinSpareServers 5
MaxSpareServers 10
</IfModule>

ServerLimit 256
MaxRequestWorkers 150
MaxConnectionsPerChild 10000
KeepAlive On
KeepAliveTimeout 5
MaxKeepAliveRequests 100
Timeout 300

<IfModule ssl_module>
# cipher and protocol directives can be set in WHM under 'Apache Configuration' -> 'Global Configuration'
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
SSLProtocol All -SSLv2 -SSLv3
SSLPassPhraseDialog builtin

<IfModule socache_shmcb_module>
SSLUseStapling on
SSLStaplingCache shmcb:/etc/apache2/run/stapling_cache_shmcb(256000)

# Prevent browsers from failing if an OCSP server is temporarily broken.
SSLStaplingReturnResponderErrors off
SSLStaplingErrorCacheTimeout 60
SSLSessionCache shmcb:/etc/apache2/run/ssl_gcache_data_shmcb(1024000)
</IfModule>
<IfModule !socache_shmcb_module>
SSLSessionCache dbm:/etc/apache2/run/ssl_gcache_data_dbm
</IfModule>

SSLSessionCacheTimeout 300
Mutex file:/etc/apache2/run ssl-cache
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin

# The Listen port can be updated using 'Tweak Settings' -> 'System',
# However, if you have any Apache Reserved IPs, then this Tweak setting will
# be ignored. Instead, each IP on your system (excluding Apache Reserved IPs)
# will be listed here.
Listen 0.0.0.0:443
Listen [::]:443

AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
</IfModule>

I'm not even sure where to go from here. There's nothing I can find that would be causing a 30 second SSL handshake. We even re-created the certificates and private keys completely just to be sure it wasn't something there. Nothing relevant in any of the server's logs. We've tried disabling the firewall, and no difference there either.

If I test using pingdom.com or another took, it typically takes a full 60 seconds to load a page.

I haven't had a chance to revert apache from MPM-ITK to see if that might be the problem, will probably be my next attempt. But wanted to see if there's something I might be missing here. Something is obviously misconfigured on this server.

 

[jestep]

So, I restarted apache, and the time went back down to below 1 second. After a few minutes, it's bouncing between about 5 and 10, and now it's back up to 30 seconds.

 

[cPanelMichael]

Hello,

The following third-party URL offers some additional commands you can use to measure the amount of time it's taking for the SSL connections:

SSL handshake latency and HTTPS optimizations. :: semicomplete.com - Jordan Sissel

Have you tested from multiple locations to rule out any network issues from the test server you are using to make the connection? Were you able to test with MPM-ITK disabled to see if that makes a difference?

Thank you.