SOLVED How to debug slow response time when load average is already low?

[SpaceCowboy]

i notice certain delay/lag when navigating the site but load average of the server is low (around 1.5~2). So my question is where can i look for the cause of this delay that happens right after clicking any post on the site, it feels like it takes some good ms before actually start loading.

Server Info:

TOP:

top - 16:31:56 up 43 days, 15:10,  1 user,  load average: 1.74, 1.92, 1.95
Tasks: 257 total,   2 running, 255 sleeping,   0 stopped,   0 zombie
Cpu(s): 18.6%us,  1.0%sy,  0.0%ni, 80.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16290720k total, 15831376k used,   459344k free,   687116k buffers
Swap:  8159228k total,      180k used,  8159048k free,  4277740k cached

Linux srv 2.6.32-642.el6.x86_64 #1 SMP Tue May 10 17:27:01 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

CPU:

vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz
stepping        : 3
microcode       : 30
cpu MHz         : 3501.000
cache size      : 8192 KB

Memory:

MemTotal:       16290720 kB
MemFree:          581156 kB
Buffers:          686224 kB
Cached:          4194092 kB
SwapCached:            8 k

Mysql Conf:

[mysqld]
performance-schema=0
   max_connections=165
   wait_timeout=7000
   interactive_timeout=7000
   query_cache_type=0
   join_buffer_size=16M
   tmp_table_size=64M
   max_heap_table_size=64M
   thread_cache_size=8
   table_open_cache=3000
   max_allowed_packet=16M
   open_files_limit=10000
   innodb_buffer_pool_size = 8192M
   innodb_log_file_size = 256M
   innodb_buffer_pool_instances=8
   innodb_log_buffer_size = 8M
   key_buffer_size = 64M
   innodb_file_per_table

Mysql tunner suggestions:

perl mysqltuner.pl
 >>  MySQLTuner 1.6.12 - Major Hayden <[email protected]>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 10.0.29-MariaDB
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA
[--] Data in MyISAM tables: 21M (Tables: 19)
[--] Data in InnoDB tables: 390M (Tables: 160)
[!!] Total fragmented tables: 5

-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[--] There are 605 basic passwords in the list.

-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION

-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 29d 23h 31m 34s (1B q [449.884 qps], 15M conn, TX: 3520G, RX: 203G)
[--] Reads / Writes: 99% / 1%
[--] Binary logging is disabled
[--] Physical Memory     : 15.5G
[--] Max MySQL memory    : 11.3G
[--] Other process memory: 1.4G
[--] Total buffers: 8.3G global + 18.7M per thread (165 max threads)
[--] P_S Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 9.4G (60.71% of installed RAM)
[OK] Maximum possible memory usage: 11.3G (72.55% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (461/1B)
[OK] Highest usage of available connections: 38% (64/165)
[OK] Aborted connections: 0.01%  (981/15016320)
[OK] Query cache is disabled by default due to mutex contention on multiprocessor machines.
[OK] Sorts requiring temporary tables: 0% (3K temp sorts / 314M sorts)
[!!] Joins performed without indexes: 11811
[OK] Temporary tables created on disk: 13% (41M on disk / 314M total)
[OK] Table cache hit rate: 86% (532 open / 616 opened)
[OK] Open file limit used: 1% (132/10K)
[OK] Table locks acquired immediately: 99% (1B immediate / 1B locks)

-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is enabled.
[--] Thread Pool Size: 8 thread(s).
[--] Using default value is good enough for your version (10.0.29-MariaDB)

-------- Performance schema ------------------------------------------------------------------------
[--] Performance schema is disabled.

-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 21.3% (14M used / 67M cache)
[OK] Key buffer size / total MyISAM indexes: 64.0M/2.1M
[OK] Read Key buffer hit rate: 100.0% (423M cached / 2K reads)
[!!] Write Key buffer hit rate: 31.5% (302K cached / 207K writes)

-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/1B
[OK] Aria pagecache hit rate: 97.9% (1B cached / 34M reads)

-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB buffer pool / data size: 8.0G/390.9M
[OK] InnoDB buffer pool instances: 8
[!!] InnoDB Used buffer: 6.87% (36013 used/ 524280 total)
[OK] InnoDB Read buffer efficiency: 100.00% (541943673096 hits/ 541943697185 total)
[!!] InnoDB Write Log efficiency: 2.33% (181844 hits/ 7802250 total)
[OK] InnoDB log waits: 0.00% (0 waits / 7620406 writes)

-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.

-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.

-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] This is a standalone server.

-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
   Run OPTIMIZE TABLE to defragment tables for better performance
   Adjust your join queries to always utilize indexes
Variables to adjust:
   join_buffer_size (> 16.0M, or always use indexes with joins)

 

[cPanelMichael]

Hello,

Could you provide more information about the website? For instance, is it a static HTML page, or does it utilize a specific PHP application such as WordPress?

Thanks!

 

[SpaceCowboy]

Hello,

Could you provide more information about the website? For instance, is it a static HTML page, or does it utilize a specific PHP application such as WordPress?

Thanks!

Hi thanks for the reply, yes it is indeed wordpress. However it serves static HTML to all visitors except logged in users (the admin and about 5 writers). Registration is disabled. We are using W3 total cache to generate the html cache. So it's pretty much html for everyone.
You can see this here - Removed -

 

[Infopro]

That site is blazing fast for the amount of content, from here.

 

[SpaceCowboy]

That site is blazing fast for the amount of content, from here.

Ok i see but if you look at the loading icon on the browser tab? It first spins CCW and after that it starts spinning CW and that's where it's actually loading. My problem is with the CCW load time, it takes too long. Do you see what i mean?

 

[Infopro]

Why are we looking at the spinner? Your site is heavy with content and images. It loads great, it gets good scores here, I just checked it:
gtmetrix.com

There may be tweaking to do, but I think you're looking for issues that aren't there.

 

[SpaceCowboy]

i looked it up and it is called TTFB i think, i measured it with F12 menu on chrome. it's more than 1 second. And i think it's bad, really. I'm not crazy!

 

[Infopro]

I would never suggest you're crazy, I think you should be proud of the fast website. I think you'd agree there is a lot of content on your site. There are several other threads posted recently regarding TTFB you might check out:
Extremely slow TTFB
Another TTFB thread

In one of those threads Anthony Parsons posted a link to CloudFlare blog worth a peek as well:
Stop worrying about Time To First Byte (TTFB)

 

[SpaceCowboy]

Thanks i'll investigate. PS: I wasn't mad just joking.

 

[SpaceCowboy]

FIGURED IT OUT!
The TTFB depends on all the things that exist on the page, scripts, and the php code that generates it.
I realized why you were seeing it very fast, you see the HTML cache that has a low TTFB.
I am logged in and see the php+mysql version which is slow and heavy so the TTFB for me is a lot higher.
I can stop worrying about this now.

 

[Infopro]

Nice. Thanks for updating the thread with your findings.