Unusual load MySQL

bassrehab · Mar 20, 2014

I tried the mysqlmymonlite.sh to gather the info but apparently it was stuck at (see below).

Code:

 Running make install
F/FD/FDALY/Test-Tester-0.109.tar.gz is just needed temporarily during building or testing. Do you want to install it permanently? [yes]

Anyways, i tried to fetch most info that may help you identify the problem.

The problem and overview:

1. Mysql load was normal at 7%-8% historically.
2. Suddenly for the past one week, i see it consistently at 25%, server load too has increased hence
3. Since then it has kept increasing incrementally to now 32%.
4. No major codebase changes.
5. Standard Apache 2.4 installation with Php 5.4 + Node + Redis + Mongo.
6. 2 Operational Sites, with average combine traffic of 15K pageviews daily.
7. Mysql Cache was turned off as over the months i found that it was generating more overhead than providing any help.
8. Queries not-using-indexes has remained high forever (pardon, programmatic handicap)

tuning-primer.sh

Code:

	-- MYSQL PERFORMANCE TUNING PRIMER --
	     - By: Matthew Montgomery -

MySQL Version 5.5.36-cll x86_64

Uptime = 13 days 18 hrs 56 min 12 sec
Avg. qps = 196
Total Questions = 233818121
Threads Connected = 5

Server has been running for over 48hrs.
It should be safe to follow these recommendations

To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service

SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 1.000000 sec.
You have 101 out of 233818159 that take longer than 1.000000 sec. to complete
Your long_query_time seems to be fine

BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html

WORKER THREADS
Current thread_cache_size = 16
Current threads_cached = 12
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine

MAX CONNECTIONS
Current max_connections = 151
Current threads_connected = 5
Historic max_used_connections = 29
The number of used connections is 19% of the configured maximum.
Your max_connections variable seems to be fine.

INNODB STATUS
Current InnoDB index space = 13 M
Current InnoDB data space = 558 M
Current InnoDB buffer pool free = 4 %
Current innodb_buffer_pool_size = 560 M
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory

MEMORY USAGE
Max Memory Ever Allocated : 1021 M
Configured Max Per-thread Buffers : 2.10 G
Configured Max Global Buffers : 608 M
Configured Max Memory Limit : 2.69 G
Physical Memory : 23.59 G
Max memory limit seem to be within acceptable norms

KEY BUFFER
Current MyISAM index space = 104 M
Current key_buffer_size = 32 M
Key cache miss rate is 1 : 305208
Key buffer free ratio = 40 %
Your key_buffer_size seems to be too high.
Perhaps you can use these resources elsewhere

QUERY CACHE
Query cache is supported but not enabled
Perhaps you should set the query_cache_size

SORT OPERATIONS
Current sort_buffer_size = 2 M
Current read_rnd_buffer_size = 8 M
Sort buffer seems to be fine

JOINS
Current join_buffer_size = 2.00 M
You have had 53101252 queries where a join could not use an index properly
You have had 347 joins without keys that check for key usage after each row
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
If you are unable to optimize your queries you may want to increase your
join_buffer_size to accommodate larger joins in one pass.

Note! This script will still suggest raising the join_buffer_size when
ANY joins not using indexes are found.

OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine

TABLE CACHE
Current table_open_cache = 10240 tables
Current table_definition_cache = 10240 tables
You have a total of 1491 tables
You have 1819 open tables.
The table_cache value seems to be fine

TEMP TABLES
Current max_heap_table_size = 32 M
Current tmp_table_size = 32 M
Of 53364610 temp tables, 33% were created on disk
Perhaps you should increase your tmp_table_size and/or max_heap_table_size
to reduce the number of disk-based temporary tables
Note! BLOB and TEXT columns are not allow in memory tables.
If you are using these columns raising these values might not impact your 
ratio of on disk temp tables.

TABLE SCANS
Current read_buffer_size = 2 M
Current table scan ratio = 699 : 1
read_buffer_size seems to be fine

TABLE LOCKING
Current Lock Wait ratio = 1 : 3632
You may benefit from selective use of InnoDB.
If you have long running SELECT's against MyISAM tables and perform
frequent updates consider setting 'low_priority_updates=1'
If you have a high concurrency of inserts on Dynamic row-length tables
consider setting 'concurrent_insert=ALWAYS'.

mysqltuner.pl

Code:

root@some-network [~/scripts]# perl mysqltuner*

 >>  MySQLTuner 1.2.0 - Major Hayden <[email protected]>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.36-cll
[OK] Operating on 64-bit architecture

-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster 
[--] Data in MyISAM tables: 311M (Tables: 1113)
[--] Data in InnoDB tables: 558M (Tables: 324)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in MEMORY tables: 14M (Tables: 13)
[!!] Total fragmented tables: 74

-------- Security Recommendations  -------------------------------------------
[OK] All database users have passwords assigned

-------- Performance Metrics -------------------------------------------------
[--] Up for: 13d 18h 57m 24s (233M q [196.269 qps], 11M conn, TX: 4220B, RX: 117B)
[--] Reads / Writes: 96% / 4%
[--] Total buffers: 640.0M global + 14.2M per thread (151 max threads)
[OK] Maximum possible memory usage: 2.7G (11% of installed RAM)
[OK] Slow queries: 0% (101/233M)
[OK] Highest usage of available connections: 19% (29/151)
[OK] Key buffer size / total MyISAM indexes: 32.0M/104.6M
[OK] Key buffer hit rate: 100.0% (6B cached / 21K reads)
[!!] Query cache is disabled
[OK] Sorts requiring temporary tables: 0% (6K temp sorts / 66M sorts)
[!!] Joins performed without indexes: 53107745
[!!] Temporary tables created on disk: 33% (27M on disk / 80M total)
[OK] Thread cache hit rate: 99% (339 created / 11M connections)
[!!] Table cache hit rate: 8% (1K open / 21K opened)
[OK] Open file limit used: 7% (2K/32K)
[OK] Table locks acquired immediately: 99% (424M immediate / 424M locks)
[OK] InnoDB data size / buffer pool: 558.4M/560.0M

-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Enable the slow query log to troubleshoot bad queries
    Adjust your join queries to always utilize indexes
    When making adjustments, make tmp_table_size/max_heap_table_size equal
    Reduce your SELECT DISTINCT queries without LIMIT clauses
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
    query_cache_size (>= 8M)
    join_buffer_size (> 2.0M, or always use indexes with joins)
    tmp_table_size (> 32M)
    max_heap_table_size (> 32M)
    table_cache (> 10240)

mysql config

Code:

[mysqld]
bind-address=127.0.0.1
innodb_file_per_table=1
innodb_buffer_pool_size=560M
key_buffer_size=32M
join_buffer_size=2M
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=8M
myisam_sort_buffer_size=32M
table_open_cache=10240
table_definition_cache=10240
thread_cache_size=16
tmp_table_size=32M
max_heap_table_size=32M
long_query_time=1
open_files_limit=32768

Load Averages sar -p

Code:

  root@some-network [~/scripts]# sar -p
Linux 3.10.9-xxxx-grs-ipv6-64 (some.hostname.net) 	Thursday 20 February 2014 	_x86_64_	(8 CPU)

12:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01  IST     all      2.20      0.00      0.35      0.42      0.00     97.03
12:20:01  IST     all      2.34      0.00      0.37      0.33      0.00     96.95
12:30:01  IST     all      2.15      0.00      0.34      0.36      0.00     97.15
12:40:01  IST     all      1.93      0.05      0.39      0.39      0.00     97.24
12:50:01  IST     all      2.26      0.00      0.47      0.38      0.00     96.89
01:00:01  IST     all      1.89      0.00      0.34      0.38      0.00     97.40
01:10:01  IST     all      1.71      0.00      0.31      0.33      0.00     97.65
01:20:01  IST     all      2.14      0.00      0.36      0.41      0.00     97.09
01:30:01  IST     all      1.71      0.01      0.27      0.53      0.00     97.49
01:40:01  IST     all      1.82      0.00      0.28      0.38      0.00     97.51
01:50:01  IST     all      1.88      0.00      0.31      0.36      0.00     97.44
02:00:01  IST     all      1.66      0.00      0.25      0.35      0.00     97.75
02:10:01  IST     all      1.46      0.00      0.22      0.35      0.00     97.97
02:20:01  IST     all      1.94      0.01      0.38      0.37      0.00     97.31
02:30:01  IST     all      1.55      0.00      0.22      0.33      0.00     97.91
02:40:01  IST     all      1.67      0.00      0.25      0.31      0.00     97.77
02:50:01  IST     all      1.78      0.04      0.31      0.37      0.00     97.50
03:00:01  IST     all      4.30      0.00      1.19      0.42      0.00     94.09
03:10:01  IST     all      1.79      0.01      0.28      0.32      0.00     97.61
03:20:01  IST     all      1.54      0.00      0.20      0.33      0.00     97.92
03:30:01  IST     all      1.87      0.00      0.29      0.31      0.00     97.53
03:40:01  IST     all      1.92      0.01      0.30      0.36      0.00     97.41
03:50:01  IST     all      2.45      0.01      0.44      0.44      0.00     96.67
04:00:01  IST     all      1.67      0.00      0.31      0.46      0.00     97.56
04:10:01  IST     all      1.79      0.00      0.30      0.39      0.00     97.52
04:20:01  IST     all      1.97      0.01      0.29      0.37      0.00     97.36
04:30:01  IST     all      1.54      0.00      0.24      0.35      0.00     97.87
04:40:01  IST     all      1.74      0.04      0.29      0.33      0.00     97.60
04:50:01  IST     all      1.63      0.04      0.24      0.33      0.00     97.75
05:00:01  IST     all      1.59      0.00      0.22      0.31      0.00     97.87
05:10:01  IST     all      1.87      0.01      0.27      0.34      0.00     97.51
05:20:01  IST     all      1.56      0.00      0.25      0.31      0.00     97.87
05:30:01  IST     all      1.46      0.00      0.22      0.31      0.00     98.02
05:40:01  IST     all      1.56      0.00      0.25      0.30      0.00     97.89
05:50:01  IST     all      1.47      0.00      0.22      0.27      0.00     98.04
06:00:01  IST     all      1.70      0.01      0.25      0.30      0.00     97.74
06:10:01  IST     all      1.65      0.00      0.27      0.31      0.00     97.77
06:20:01  IST     all      1.44      0.00      0.22      0.30      0.00     98.04
06:30:01  IST     all      1.82      0.00      0.30      0.28      0.00     97.60
06:40:01  IST     all      1.61      0.00      0.25      0.26      0.00     97.88
06:50:01  IST     all      1.83      0.04      0.37      0.27      0.00     97.49
07:00:01  IST     all      1.59      0.00      0.25      0.32      0.00     97.83

07:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:10:01  IST     all      1.64      0.00      0.26      0.33      0.00     97.77
07:20:01  IST     all      1.67      0.00      0.31      0.30      0.00     97.72
07:30:01  IST     all      1.63      0.00      0.34      0.30      0.00     97.73
07:40:01  IST     all      1.36      0.01      0.24      0.29      0.00     98.11
07:50:01  IST     all      1.99      0.00      0.35      0.35      0.00     97.31
08:00:01  IST     all      2.58      0.00      0.61      0.36      0.00     96.44
08:10:01  IST     all      2.49      0.00      0.53      0.35      0.00     96.63
08:20:01  IST     all      2.90      0.00      0.73      0.35      0.00     96.02
08:30:01  IST     all      2.33      0.01      0.42      0.34      0.00     96.90
08:40:01  IST     all      3.06      0.04      0.75      0.36      0.00     95.79
08:50:01  IST     all      2.43      0.00      0.40      0.36      0.00     96.80
09:00:01  IST     all      3.08      0.00      0.65      0.36      0.00     95.90
09:10:01  IST     all      2.96      0.00      0.58      0.41      0.00     96.04
09:20:01  IST     all      3.28      0.01      0.62      0.39      0.00     95.70
09:30:01  IST     all      3.11      0.00      0.73      0.38      0.00     95.79
09:40:01  IST     all      2.40      0.00      0.42      0.45      0.00     96.73
09:50:01  IST     all      2.51      0.00      0.45      0.35      0.00     96.69
10:00:01  IST     all      2.26      0.00      0.39      0.39      0.00     96.96
10:10:01  IST     all      2.10      0.01      0.34      0.38      0.00     97.17
10:20:01  IST     all      2.08      0.00      0.39      0.43      0.00     97.10
10:30:01  IST     all      2.70      0.00      0.51      0.35      0.00     96.44
10:40:01  IST     all      3.09      0.04      0.57      0.38      0.00     95.91
10:50:01  IST     all      3.05      0.00      0.58      0.37      0.00     96.00
11:00:01  IST     all      2.91      0.01      0.50      0.37      0.00     96.21
11:10:01  IST     all      3.22      0.00      0.70      0.44      0.00     95.64
11:20:01  IST     all      3.27      0.00      0.59      0.42      0.00     95.70
11:30:01  IST     all      2.94      0.00      0.50      0.39      0.00     96.17
11:40:01  IST     all      3.10      0.00      0.55      0.38      0.00     95.96
11:50:01  IST     all      3.52      0.01      0.63      0.41      0.00     95.43
12:00:01  IST     all      3.06      0.00      0.49      0.37      0.00     96.08
12:10:01  IST     all      3.81      0.00      0.74      0.45      0.00     95.00
12:20:01  IST     all      3.76      0.00      0.71      0.37      0.00     95.16
12:30:02  IST     all      3.82      0.01      0.61      0.37      0.00     95.20
12:40:01  IST     all      3.15      0.05      0.50      0.38      0.00     95.92
12:50:01  IST     all      2.91      0.00      0.41      0.36      0.00     96.32
01:00:01  IST     all      3.08      0.00      0.46      0.35      0.00     96.12
01:10:01  IST     all      3.25      0.00      0.51      0.41      0.00     95.82
01:20:01  IST     all      3.38      0.00      0.63      0.40      0.00     95.59
01:30:01  IST     all      3.79      0.01      0.71      0.37      0.00     95.12
01:40:01  IST     all      3.10      0.00      0.45      0.34      0.00     96.11
01:50:01  IST     all      2.87      0.00      0.40      0.34      0.00     96.38
02:00:01  IST     all      3.37      0.00      0.57      0.41      0.00     95.64

02:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
02:10:01  IST     all      4.25      0.00      0.87      0.42      0.00     94.46
02:20:01  IST     all      3.54      0.01      0.61      0.38      0.00     95.46
02:30:01  IST     all      3.63      0.00      0.73      0.38      0.00     95.25
02:40:01  IST     all      3.56      0.00      0.53      0.43      0.00     95.47
02:50:01  IST     all      3.92      0.05      0.68      0.40      0.00     94.95
03:00:01  IST     all      3.51      0.00      0.56      0.35      0.00     95.59
03:10:01  IST     all      4.27      0.01      0.73      0.41      0.00     94.58
03:20:01  IST     all      4.03      0.00      0.76      0.41      0.00     94.80
03:30:01  IST     all      3.75      0.01      0.61      0.36      0.00     95.27
03:40:01  IST     all      3.86      0.00      0.72      0.43      0.00     94.99
03:50:01  IST     all      4.30      0.00      0.76      0.40      0.00     94.54
04:00:01  IST     all      3.53      0.00      0.59      0.38      0.00     95.50
04:10:01  IST     all      6.22      0.01      0.65      0.37      0.00     92.75
04:20:01  IST     all      2.61      0.00      0.43      0.34      0.00     96.61
04:30:01  IST     all      3.40      0.00      0.63      0.34      0.00     95.63
04:40:01  IST     all      3.69      0.00      0.56      0.39      0.00     95.35
04:50:01  IST     all      3.30      0.05      0.54      0.35      0.00     95.76
05:00:01  IST     all      3.48      0.01      0.52      0.35      0.00     95.64
05:10:01  IST     all      3.72      0.00      0.60      0.39      0.00     95.29
05:20:01  IST     all      3.86      0.00      0.65      0.36      0.00     95.13
05:30:01  IST     all      2.82      0.00      0.49      0.38      0.00     96.31
05:40:01  IST     all      2.73      0.00      0.47      0.37      0.00     96.43
05:50:01  IST     all      2.70      0.34      0.43      0.37      0.00     96.16
06:00:01  IST     all      2.61      0.00      0.39      0.32      0.00     96.68
06:10:01  IST     all      2.67      0.00      0.41      0.37      0.00     96.54
06:20:01  IST     all      2.65      0.00      0.41      0.36      0.00     96.58
06:30:01  IST     all      2.78      0.00      0.44      0.30      0.00     96.47
06:40:01  IST     all      2.81      0.05      0.42      0.33      0.00     96.39
06:50:01  IST     all      2.92      0.00      0.50      0.32      0.00     96.26
07:00:01  IST     all      2.72      0.00      0.49      0.34      0.00     96.44
07:10:01  IST     all      2.81      0.00      0.75      0.34      0.00     96.10
07:20:01  IST     all      2.29      0.00      0.42      0.29      0.00     96.99
07:30:01  IST     all      2.34      0.01      0.38      0.30      0.00     96.97
07:40:01  IST     all      2.61      0.00      0.42      0.33      0.00     96.64
07:50:01  IST     all      2.67      0.00      0.51      0.32      0.00     96.50
08:00:01  IST     all      2.25      0.00      0.37      0.31      0.00     97.07
08:10:01  IST     all      2.32      0.00      0.36      0.33      0.00     96.98
08:20:01  IST     all      2.07      0.01      0.36      0.32      0.00     97.25
08:30:01  IST     all      3.06      0.00      0.63      0.35      0.00     95.96
08:40:01  IST     all      2.70      0.04      0.57      0.29      0.00     96.40
08:50:01  IST     all      2.47      0.00      0.40      0.34      0.00     96.78
09:00:01  IST     all      2.10      0.00      0.36      0.34      0.00     97.20

09:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
09:10:01  IST     all      2.33      0.01      0.37      0.35      0.00     96.95
09:20:01  IST     all      2.97      0.00      0.51      0.32      0.00     96.18
09:30:01  IST     all      2.35      0.00      0.37      0.36      0.00     96.92
09:40:01  IST     all      2.22      0.00      0.33      0.31      0.00     97.13
09:50:01  IST     all      2.44      0.00      0.42      0.35      0.00     96.79
10:00:01  IST     all      2.69      0.01      0.44      0.36      0.00     96.50
10:10:01  IST     all      2.78      0.00      0.56      0.36      0.00     96.29
10:20:01  IST     all      2.89      0.00      0.59      0.35      0.00     96.17
10:30:01  IST     all      2.42      0.00      0.44      0.34      0.00     96.79
10:40:01  IST     all      2.52      0.04      0.40      0.33      0.00     96.71
10:50:01  IST     all      2.30      0.01      0.36      0.33      0.00     97.00
11:00:01  IST     all      2.12      0.00      0.32      0.32      0.00     97.23
11:10:01  IST     all      2.13      0.00      0.36      0.35      0.00     97.15
11:20:01  IST     all      2.45      0.00      0.39      0.34      0.00     96.82
11:30:01  IST     all      2.61      0.00      0.49      0.36      0.00     96.54
11:40:01  IST     all      2.09      0.01      0.34      0.34      0.00     97.22
11:50:01  IST     all      2.27      0.00      0.38      0.34      0.00     97.00
12:00:01  IST     all     13.58      0.02      1.90      0.49      0.00     84.01
12:10:01  IST     all     15.27      0.00      2.27      0.58      0.00     81.88
12:20:01  IST     all     15.82      0.00      2.33      0.50      0.00     81.34
12:30:01  IST     all     15.66      0.00      2.25      0.47      0.00     81.61
12:40:01  IST     all     15.36      0.00      2.26      0.52      0.00     81.86
12:50:01  IST     all     15.75      0.11      2.27      0.51      0.00     81.38
01:00:01  IST     all     16.36      0.00      2.44      0.55      0.00     80.65
01:10:01  IST     all     15.83      0.00      2.34      0.64      0.00     81.19
01:20:01  IST     all     15.98      0.00      2.29      0.71      0.00     81.01
01:30:01  IST     all     16.05      0.00      2.39      0.67      0.00     80.89
01:40:01  IST     all     15.98      0.01      2.36      0.61      0.00     81.04
01:50:01  IST     all     16.54      0.00      2.63      0.61      0.00     80.22
02:00:01  IST     all     16.67      0.00      2.38      0.61      0.00     80.33
02:10:01  IST     all     15.86      0.00      2.33      0.54      0.00     81.27
02:20:01  IST     all     15.89      0.00      2.28      0.48      0.00     81.35
02:30:01  IST     all     15.74      0.01      2.35      0.57      0.00     81.32
02:40:01  IST     all     15.68      0.10      2.30      0.61      0.00     81.32
02:50:01  IST     all     15.10      0.00      2.28      0.49      0.00     82.13
03:00:01  IST     all     15.41      0.00      2.23      0.51      0.00     81.85
03:10:01  IST     all     16.04      0.00      2.33      0.63      0.00     81.00
03:20:01  IST     all     16.51      0.01      2.40      0.58      0.00     80.50
03:30:01  IST     all     15.20      0.00      2.20      0.43      0.00     82.16
03:40:01  IST     all     15.96      0.02      2.33      0.48      0.00     81.21
03:50:01  IST     all     16.38      0.00      2.35      0.47      0.00     80.79
04:00:01  IST     all     15.37      0.04      2.23      0.45      0.00     81.91

04:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
04:10:01  IST     all     15.55      0.01      2.26      0.54      0.00     81.64
04:20:01  IST     all     15.62      0.00      2.24      0.62      0.00     81.51
04:30:01  IST     all     15.77      0.00      2.26      0.57      0.00     81.39
04:40:01  IST     all     15.74      0.09      2.32      0.56      0.00     81.29
04:50:01  IST     all     15.45      0.00      2.26      0.43      0.00     81.85
05:00:01  IST     all     16.45      0.01      2.44      0.54      0.00     80.56
05:10:01  IST     all     15.25      0.00      2.26      0.51      0.00     81.98
05:20:01  IST     all     16.03      0.00      2.33      0.49      0.00     81.15
05:30:01  IST     all     16.10      0.00      2.31      0.49      0.00     81.09
05:40:01  IST     all     15.48      0.00      2.25      0.54      0.00     81.73
05:50:01  IST     all     14.72      0.01      2.27      0.44      0.00     82.57
06:00:01  IST     all     15.40      0.00      2.20      0.47      0.00     81.92
06:10:01  IST     all     15.94      0.00      2.31      0.51      0.00     81.23
06:20:01  IST     all     14.39      0.00      2.05      0.49      0.00     83.06
06:30:01  IST     all     13.76      0.00      1.95      0.52      0.00     83.77
06:40:01  IST     all     15.10      0.10      2.22      0.61      0.00     81.96
06:50:01  IST     all     14.88      0.00      2.22      0.51      0.00     82.38
07:00:01  IST     all     14.71      0.00      2.14      0.52      0.00     82.62
07:10:01  IST     all     14.42      0.00      2.11      0.46      0.00     83.01
07:20:01  IST     all     13.62      0.00      1.97      0.56      0.00     83.86
07:30:01  IST     all     13.82      0.01      1.99      0.52      0.00     83.66
07:40:01  IST     all     13.88      0.00      1.97      0.46      0.00     83.69
07:50:01  IST     all     13.34      0.00      1.92      0.54      0.00     84.20
08:00:01  IST     all     14.61      0.00      2.21      0.42      0.00     82.76
08:10:01  IST     all     14.48      0.00      2.11      0.51      0.00     82.89
08:20:01  IST     all     15.40      0.01      2.24      0.50      0.00     81.85
08:30:01  IST     all     15.30      0.00      2.19      0.37      0.00     82.13
08:40:01  IST     all     16.15      0.00      2.40      0.60      0.00     80.84
08:50:01  IST     all     16.31      0.11      2.39      0.54      0.00     80.65
09:00:01  IST     all     14.46      0.00      2.10      0.57      0.00     82.86
09:10:01  IST     all     15.64      0.01      2.30      0.57      0.00     81.49
09:20:01  IST     all     16.32      0.00      2.43      0.61      0.00     80.64
09:30:01  IST     all     15.15      0.00      2.18      0.54      0.00     82.13
09:40:01  IST     all     15.94      0.00      2.31      0.53      0.00     81.21
09:50:01  IST     all     16.27      0.00      2.43      0.57      0.00     80.73
10:00:01  IST     all     15.19      0.01      2.24      0.54      0.00     82.02
10:10:01  IST     all     15.37      0.00      2.22      0.54      0.00     81.87
10:20:01  IST     all     15.77      0.00      2.27      0.62      0.00     81.34
10:30:01  IST     all     15.80      0.00      2.26      0.63      0.00     81.31
10:40:01  IST     all     16.02      0.00      2.33      0.60      0.00     81.04
10:50:01  IST     all     16.08      0.10      2.34      0.66      0.00     80.82
11:00:01  IST     all     14.85      0.00      2.15      0.57      0.00     82.43

11:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
11:10:01  IST     all     15.40      0.00      2.31      0.52      0.00     81.77
11:20:01  IST     all     15.86      0.00      2.29      0.56      0.00     81.29
11:30:01  IST     all     15.54      0.00      2.29      0.59      0.00     81.58
11:40:01  IST     all     15.70      0.01      2.27      0.54      0.00     81.48
11:50:01  IST     all     15.81      0.00      2.29      0.77      0.00     81.12
12:00:01  IST     all     16.26      0.00      2.37      0.75      0.00     80.62
12:10:01  IST     all     14.53      0.00      2.04      0.75      0.00     82.68
12:20:01  IST     all     14.98      0.00      2.13      0.51      0.00     82.37
12:30:01  IST     all     16.39      0.01      2.39      0.51      0.00     80.70
12:40:01  IST     all     16.57      0.10      2.41      0.54      0.00     80.39
12:50:01  IST     all     14.42      0.00      2.10      0.65      0.00     82.83
01:00:01  IST     all     16.87      0.00      2.44      0.86      0.00     79.82
01:10:01  IST     all     15.01      0.00      2.22      0.67      0.00     82.09
01:20:01  IST     all     15.13      0.01      2.13      0.49      0.00     82.24
01:30:01  IST     all     16.26      0.00      2.49      0.58      0.00     80.65
01:40:01  IST     all     15.42      0.00      2.22      0.55      0.00     81.80
01:50:01  IST     all     15.60      0.00      2.25      0.86      0.00     81.28
02:00:01  IST     all     14.82      0.00      2.10      0.79      0.00     82.28
02:10:01  IST     all     19.04      0.01      2.52      0.72      0.00     77.72
02:20:01  IST     all     15.42      0.00      2.20      0.75      0.00     81.62
02:30:01  IST     all     15.51      0.00      2.22      0.77      0.00     81.50
02:40:01  IST     all     15.28      0.09      2.26      0.54      0.00     81.82
02:50:01  IST     all     15.96      0.00      2.33      0.68      0.00     81.03
03:00:01  IST     all     16.47      0.00      2.42      0.83      0.00     80.27
03:10:01  IST     all     16.25      0.01      2.39      0.79      0.00     80.57
03:20:01  IST     all     16.27      0.00      2.39      0.88      0.00     80.46
03:30:02  IST     all     17.21      0.00      2.58      0.68      0.00     79.53
03:40:01  IST     all     15.56      0.00      2.22      0.87      0.00     81.35
03:50:01  IST     all     16.05      0.00      2.33      0.90      0.00     80.71
04:00:01  IST     all     16.19      0.01      2.34      0.85      0.00     80.60
04:10:01  IST     all     16.06      0.00      2.34      0.83      0.00     80.76
04:20:02  IST     all     16.80      0.00      2.44      0.75      0.00     80.01
04:30:01  IST     all      8.06      0.00      1.09      0.57      0.00     90.28
04:40:01  IST     all     15.70      0.10      2.26      0.52      0.00     81.43
04:50:01  IST     all     16.19      0.01      2.33      0.53      0.00     80.95
05:00:01  IST     all     15.34      0.00      2.22      0.53      0.00     81.91
05:10:01  IST     all     15.89      0.00      2.28      0.54      0.00     81.30
05:20:01  IST     all     15.75      0.00      2.28      0.54      0.00     81.43
05:30:01  IST     all     15.95      0.00      2.35      0.68      0.00     81.02
05:40:01  IST     all     16.17      1.63      2.40      0.55      0.00     79.25
05:50:01  IST     all     17.11      0.01      2.54      0.59      0.00     79.76
06:00:01  IST     all     17.19      0.00      2.55      0.55      0.00     79.71

06:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
06:10:01  IST     all     16.83      0.00      2.51      0.65      0.00     80.02
06:20:01  IST     all     16.78      0.00      2.49      0.63      0.00     80.10
06:30:01  IST     all     15.85      0.00      2.35      0.59      0.00     81.20
06:40:01  IST     all     16.74      0.10      2.45      0.55      0.00     80.16
06:50:01  IST     all     15.78      0.00      2.33      0.59      0.00     81.29
07:00:01  IST     all     16.33      0.00      2.45      0.76      0.00     80.45
07:10:01  IST     all     17.40      0.00      2.56      0.83      0.00     79.20
07:20:01  IST     all     17.14      0.00      2.50      0.66      0.00     79.69
07:30:01  IST     all     18.79      0.01      2.76      0.61      0.00     77.83
07:40:01  IST     all     17.23      0.00      2.49      0.62      0.00     79.65
07:50:01  IST     all     18.18      0.00      2.66      0.63      0.00     78.53
08:00:01  IST     all     17.46      0.00      2.50      0.58      0.00     79.45
08:10:01  IST     all     17.51      0.00      2.59      0.75      0.00     79.14
08:20:01  IST     all     16.55      0.01      2.42      0.78      0.00     80.25
08:30:01  IST     all     16.15      0.00      2.32      0.56      0.00     80.97
08:40:01  IST     all     17.59      0.00      2.50      0.72      0.00     79.18
Average:        all     12.60      0.02      1.78      0.49      0.00     85.11  



--------------------------------------------------------------------------------------------

hardware info

Code:

root@some-network [~/scripts]# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2801.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5600.32
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2801.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5600.32
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2801.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5600.32
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2801.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5600.32
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2801.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5600.32
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2801.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 1
cpu cores	: 4
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5600.32
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2801.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 2
cpu cores	: 4
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5600.32
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
stepping	: 5
microcode	: 0x11
cpu MHz		: 2801.000
cache size	: 8192 KB
physical id	: 0
siblings	: 8
core id		: 3
cpu cores	: 4
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5600.32
clflush size	: 64
cache_alignment	: 64
address sizes	: 36 bits physical, 48 bits virtual
power management:

memory info

Code:

root@some-network [~/scripts]# cat /proc/meminfo
MemTotal:       24737800 kB
MemFree:        16371108 kB
Buffers:          788680 kB
Cached:          4937140 kB
SwapCached:            0 kB
Active:          5279624 kB
Inactive:        1600700 kB
Active(anon):    1157900 kB
Inactive(anon):   116256 kB
Active(file):    4121724 kB
Inactive(file):  1484444 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        525308 kB
SwapFree:         525308 kB
Dirty:              1004 kB
Writeback:             0 kB
AnonPages:       1153496 kB
Mapped:           166080 kB
Shmem:            120240 kB
Slab:            1254664 kB
SReclaimable:     995104 kB
SUnreclaim:       259560 kB
KernelStack:        3104 kB
PageTables:        11188 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    12894208 kB
Committed_AS:    3086672 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      326396 kB
VmallocChunk:   34359409652 kB
HardwareCorrupted:     0 kB
DirectMap4k:        5632 kB
DirectMap2M:    25151488 kB

I just want to know is there any particular reason for sudden increase in mysql load?
Thank you guys.

thinkbot · Mar 20, 2014

key_buffer_size set over 110M
also increase innodb_buffer_pool_size=560M
InnoDB can use more
and decreate table cache to 2

In your case I would turn on query cache

Maybe it's time to optimize that code and add some caching
[!!] Joins performed without indexes: 53107745
You should review database structure if it needs some indexes

I would suggest here simple solution,

add
log-queries-not-using-indexes

track those queries not using indexes, and add those indexes, or if necessary optimize queries to utilize indexes

In most cases when basic mysql optimizations (few key settings) are done, the biggest part that can impact performance is the code
People tend to search some magic solutions in mysql, some people have tens of variables in their my.cnf, but they dont look into their code

bassrehab · Mar 20, 2014

Thank you for your reply.

thinkbot said:
key_buffer_size set over 110M
also increase innodb_buffer_pool_size=560M
InnoDB can use more
and decreate table cache to 2

key_buffer_size is now set to 128M (~myisam database size)
Innodb ~ i haven't changed.

decreate table cache to 2? you mean 2048?

thinkbot said:
In your case I would turn on query cache

Query cache doesnt help my case at all. There is frequent defragmentation and a constant overhead. I have checked with various values from 128M to 32M, and cache_min_res.

thinkbot said:
Maybe it's time to optimize that code and add some caching
[!!] Joins performed without indexes: 53107745
You should review database structure if it needs some indexes

I would suggest here simple solution,

add
log-queries-not-using-indexes

track those queries not using indexes, and add those indexes, or if necessary optimize queries to utilize indexes

Yes, i am aware of non-index using joins. Its is not a new thing for our setting, i have always had non-index using joins (i knows its bad for performance). Anyways, My problem is investigating the "sudden" rise in mysql load over last one week.

thinkbot · Mar 20, 2014

yes, I mean table cache 2048

but did y ou check what type of queries are those "non-index"
can you set long query time to 0
and then run

cd /root
wget http://percona.com/get/pt-query-digest
chmod +x pt-query-digest
pt-query-digest /var/lib/mysql/mysql-slow.log > slow.txt;

to generate review of slow queries

and there you can find those longest and most often executing ones
than just run explain extended in mysql and optimize database structure/query

cPanelMichael · Mar 21, 2014

You may also want to try using a command such as "mysqladmin processlist" to get a better idea of what's happening when you notice higher than normal MySQL usage (since you stated this has only started recently).

Thank you.

bassrehab · Mar 22, 2014

A quick update:

1. key_buffer_size now set at 128M
2. File open limit (auto set by cpanel ~8250)
3. Table Cache ~ 2048
4. Current Mysql load ~ 55%
5. http traffic checked for any attacks (nothing unusual)
6. Had done hard restarts, checked hardwares for r/w faults none.

I am not sure why the load keeps increasing. Its as if some kinda overhead is building on. And it seemingly is incremental. My next plan of action is just try disabling one db at a time and probably isolate the menace.

@thinkbot - i wonder if fractional slow_query times can be set.

@cPanelMichael - have done that already. I was hoping to find sleeping queries or anything unusual but none there either.

Thread starter	Similar threads	Forum	Replies	Date
D	High Load Average	Workarounds and Optimization	1	Jul 8, 2020
M	CPU Load + 200 on mysql	Workarounds and Optimization	1	Apr 8, 2020
T	Finding Cause of High CPU Load	Workarounds and Optimization	4	Dec 18, 2019
S	MySQL CPU load	Workarounds and Optimization	4	Nov 22, 2019
W	Problem with MYSQL Load CPU	Workarounds and Optimization	1	Nov 5, 2019

Search

Search

Unusual load MySQL

bassrehab

Member

thinkbot

Well-Known Member

bassrehab

Member

thinkbot

Well-Known Member

cPanelMichael

Administrator

bassrehab

Member