CPU wait while everything else is optimal

[hicom]

Would appreciate some direction or input. CentOS 7.2 x64, 32GB RAM, E5-2609 Dual Processors 12-cores total, 2 x 500GB SSD Disks RAID-1, MariaDB 10.0, Apache 2.4 + PHP suPHP Prefork.

The load you'll see below with 650 active users who are mostly using a chat room with MySQL backend. What is concerning me is the CPU wait is 7%-10% which end up slowing down the application. Everything else is fine, even the CPU doesn't seem to be fully utilized.

How can I find out what is causing the higher Wait ?

top - 19:47:50 up 10:03,  1 user,  load average: 2.17, 2.40, 2.41
Tasks: 943 total,  7 running, 935 sleeping,  0 stopped,  1 zombie
%Cpu0  : 12.0 us, 10.7 sy,  0.0 ni, 67.2 id,  9.7 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu1  :  9.7 us,  8.7 sy,  0.0 ni, 72.6 id,  9.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  8.7 us,  7.7 sy,  0.0 ni, 74.8 id,  8.4 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu3  :  9.7 us,  8.7 sy,  0.0 ni, 72.6 id,  8.4 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu4  : 10.1 us, 12.5 sy,  0.0 ni, 69.0 id,  7.7 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu5  : 12.0 us, 12.3 sy,  0.0 ni, 67.0 id,  8.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu6  :  9.1 us,  6.0 sy,  0.0 ni, 80.9 id,  4.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  : 12.3 us,  8.6 sy,  0.0 ni, 74.4 id,  4.7 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu8  : 13.4 us,  5.0 sy,  0.0 ni, 78.5 id,  3.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu9  :  6.3 us,  9.0 sy,  0.0 ni, 80.0 id,  4.7 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu10 : 12.0 us,  8.0 sy,  0.0 ni, 76.6 id,  3.3 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu11 : 10.2 us,  6.9 sy,  0.0 ni, 78.2 id,  4.6 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32672512 total, 28286664 free,  2184524 used,  2201324 buff/cache
KiB Swap:  7812092 total,  7812092 free,  0 used. 30028120 avail Mem

  PID USER  PR  NI  VIRT  RES  SHR S  %CPU %MEM  TIME+ COMMAND
 3596 mysql  20  0 1181580 319288  9280 S  78.2  1.0 601:18.10 mysqld
21505 nobody  20  0  317164  16012  8220 S  2.0  0.0  0:04.96 httpd
21753 nobody  20  0  318200  16092  7572 S  2.0  0.0  0:03.36 httpd
22713 nobody  20  0  318156  15404  6864 S  2.0  0.0  0:02.92 httpd
22881 nobody  20  0  317172  14488  6636 S  2.0  0.0  0:02.44 httpd
16573 bestfan+  20  0  317944  16484  7864 D  1.7  0.1  0:12.52 httpd
20502 nobody  20  0  317272  15508  7896 S  1.7  0.0  0:06.63 httpd
21529 nobody  20  0  317932  15096  6488 S  1.7  0.0  0:04.35 httpd
21683 nobody  20  0  317208  13948  6340 S  1.7  0.0  0:05.12 httpd

iostat: looks very good, no load on disks

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  7.50  0.01  5.06  0.54  0.00  86.88

Device:  rrqm/s  wrqm/s  r/s  w/s  rkB/s  wkB/s avgrq-sz avgqu-sz  await r_await w_await  svctm  %util
sda  0.03  1.64  1.23  29.21  53.26  169.46  14.63  0.02  0.76  5.48  0.56  0.19  0.58
sdb  0.00  0.00  0.01  0.00  0.13  0.07  37.09  0.00  6.06  6.81  0.00  5.82  0.01

avg-cpu:  %user  %nice %system %iowait  %steal  %idle
  10.34  0.00  6.90  3.66  0.00  79.10

Device:  rrqm/s  wrqm/s  r/s  w/s  rkB/s  wkB/s avgrq-sz avgqu-sz  await r_await w_await  svctm  %util
sda  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00
sdb  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00

vmstat: no memory issues, no blocks

 vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b  swpd  free  buff  cache  si  so  bi  bo  in  cs us sy id wa st
 6  0  0 28145148  1052 2202556  0  0  4  14  95  19  8  5 87  1  0
 3  0  0 28147484  1052 2202548  0  0  0  0 30490 34875  9  9 75  8  0
 1  0  0 28147436  1052 2202544  0  0  0  0 24018 30270  8  6 82  3  0
 5  0  0 28143656  1052 2202436  0  0  0  1288 30721 37232 11  8 77  5  0
 4  1  0 28146204  1052 2202420  0  0  0  0 28151 33762 10  8 78  4  0

 

[cPanelMichael]

The load you'll see below with 650 active users who are mostly using a chat room with MySQL backend. What is concerning me is the CPU wait is 7%-10% which end up slowing down the application. Everything else is fine, even the CPU doesn't seem to be fully utilized.

Hello,

Have you ran a MySQL tuning application to determine if any changes to the MySQL configuration can help? There's a thread here that should work for MariaDB as well:

https://forums.cpanel.net/threads/m...stats-gathering-tool-for-cpanel-server.268011

Thank you.

 

[hicom]

Hello,

Have you ran a MySQL tuning application to determine if any changes to the MySQL configuration can help? There's a thread here that should work for MariaDB as well:

https://forums.cpanel.net/threads/m...stats-gathering-tool-for-cpanel-server.268011

Thank you.

Yes I did run MySQLTuner and made some adjustment before my post, still issue persist. I'm concerned that CPU WAIT is going up even though there is no Disk, Mem or CPU bottleneck. There is 80% idle CPU when WAIT is 4% and disks have no load.

 

[cPanelMichael]

You may also find this StackOverflow thread helpful:

How to find out which process is consuming "wait CPU" (i.e. I/O blocked)

Thank you.