The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Unusual load MySQL

Discussion in 'Workarounds and Optimization' started by bassrehab, Mar 20, 2014.

  1. bassrehab

    bassrehab Member

    Joined:
    Dec 13, 2013
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    I tried the mysqlmymonlite.sh to gather the info but apparently it was stuck at (see below).
    Code:
     Running make install
    F/FD/FDALY/Test-Tester-0.109.tar.gz is just needed temporarily during building or testing. Do you want to install it permanently? [yes] 
    Anyways, i tried to fetch most info that may help you identify the problem.

    The problem and overview:

    1. Mysql load was normal at 7%-8% historically.
    2. Suddenly for the past one week, i see it consistently at 25%, server load too has increased hence
    3. Since then it has kept increasing incrementally to now 32%.
    4. No major codebase changes.
    5. Standard Apache 2.4 installation with Php 5.4 + Node + Redis + Mongo.
    6. 2 Operational Sites, with average combine traffic of 15K pageviews daily.
    7. Mysql Cache was turned off as over the months i found that it was generating more overhead than providing any help.
    8. Queries not-using-indexes has remained high forever (pardon, programmatic handicap)


    tuning-primer.sh


    Code:
    	-- MYSQL PERFORMANCE TUNING PRIMER --
    	     - By: Matthew Montgomery -
    
    MySQL Version 5.5.36-cll x86_64
    
    Uptime = 13 days 18 hrs 56 min 12 sec
    Avg. qps = 196
    Total Questions = 233818121
    Threads Connected = 5
    
    Server has been running for over 48hrs.
    It should be safe to follow these recommendations
    
    To find out more information on how each of these
    runtime variables effects performance visit:
    http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html
    Visit http://www.mysql.com/products/enterprise/advisors.html
    for info about MySQL's Enterprise Monitoring and Advisory Service
    
    SLOW QUERIES
    The slow query log is NOT enabled.
    Current long_query_time = 1.000000 sec.
    You have 101 out of 233818159 that take longer than 1.000000 sec. to complete
    Your long_query_time seems to be fine
    
    BINARY UPDATE LOG
    The binary update log is NOT enabled.
    You will not be able to do point in time recovery
    See http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html
    
    WORKER THREADS
    Current thread_cache_size = 16
    Current threads_cached = 12
    Current threads_per_sec = 0
    Historic threads_per_sec = 0
    Your thread_cache_size is fine
    
    MAX CONNECTIONS
    Current max_connections = 151
    Current threads_connected = 5
    Historic max_used_connections = 29
    The number of used connections is 19% of the configured maximum.
    Your max_connections variable seems to be fine.
    
    INNODB STATUS
    Current InnoDB index space = 13 M
    Current InnoDB data space = 558 M
    Current InnoDB buffer pool free = 4 %
    Current innodb_buffer_pool_size = 560 M
    Depending on how much space your innodb indexes take up it may be safe
    to increase this value to up to 2 / 3 of total system memory
    
    MEMORY USAGE
    Max Memory Ever Allocated : 1021 M
    Configured Max Per-thread Buffers : 2.10 G
    Configured Max Global Buffers : 608 M
    Configured Max Memory Limit : 2.69 G
    Physical Memory : 23.59 G
    Max memory limit seem to be within acceptable norms
    
    KEY BUFFER
    Current MyISAM index space = 104 M
    Current key_buffer_size = 32 M
    Key cache miss rate is 1 : 305208
    Key buffer free ratio = 40 %
    Your key_buffer_size seems to be too high.
    Perhaps you can use these resources elsewhere
    
    QUERY CACHE
    Query cache is supported but not enabled
    Perhaps you should set the query_cache_size
    
    SORT OPERATIONS
    Current sort_buffer_size = 2 M
    Current read_rnd_buffer_size = 8 M
    Sort buffer seems to be fine
    
    JOINS
    Current join_buffer_size = 2.00 M
    You have had 53101252 queries where a join could not use an index properly
    You have had 347 joins without keys that check for key usage after each row
    You should enable "log-queries-not-using-indexes"
    Then look for non indexed joins in the slow query log.
    If you are unable to optimize your queries you may want to increase your
    join_buffer_size to accommodate larger joins in one pass.
    
    Note! This script will still suggest raising the join_buffer_size when
    ANY joins not using indexes are found.
    
    OPEN FILES LIMIT
    Current open_files_limit = 32768 files
    The open_files_limit should typically be set to at least 2x-3x
    that of table_cache if you have heavy MyISAM usage.
    Your open_files_limit value seems to be fine
    
    TABLE CACHE
    Current table_open_cache = 10240 tables
    Current table_definition_cache = 10240 tables
    You have a total of 1491 tables
    You have 1819 open tables.
    The table_cache value seems to be fine
    
    TEMP TABLES
    Current max_heap_table_size = 32 M
    Current tmp_table_size = 32 M
    Of 53364610 temp tables, 33% were created on disk
    Perhaps you should increase your tmp_table_size and/or max_heap_table_size
    to reduce the number of disk-based temporary tables
    Note! BLOB and TEXT columns are not allow in memory tables.
    If you are using these columns raising these values might not impact your 
    ratio of on disk temp tables.
    
    TABLE SCANS
    Current read_buffer_size = 2 M
    Current table scan ratio = 699 : 1
    read_buffer_size seems to be fine
    
    TABLE LOCKING
    Current Lock Wait ratio = 1 : 3632
    You may benefit from selective use of InnoDB.
    If you have long running SELECT's against MyISAM tables and perform
    frequent updates consider setting 'low_priority_updates=1'
    If you have a high concurrency of inserts on Dynamic row-length tables
    consider setting 'concurrent_insert=ALWAYS'.
    

    mysqltuner.pl

    Code:
    root@some-network [~/scripts]# perl mysqltuner*
    
     >>  MySQLTuner 1.2.0 - Major Hayden <major@mhtx.net>
     >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
     >>  Run with '--help' for additional options and output filtering
    
    -------- General Statistics --------------------------------------------------
    [--] Skipped version check for MySQLTuner script
    [OK] Currently running supported MySQL version 5.5.36-cll
    [OK] Operating on 64-bit architecture
    
    -------- Storage Engine Statistics -------------------------------------------
    [--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster 
    [--] Data in MyISAM tables: 311M (Tables: 1113)
    [--] Data in InnoDB tables: 558M (Tables: 324)
    [--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
    [--] Data in MEMORY tables: 14M (Tables: 13)
    [!!] Total fragmented tables: 74
    
    -------- Security Recommendations  -------------------------------------------
    [OK] All database users have passwords assigned
    
    -------- Performance Metrics -------------------------------------------------
    [--] Up for: 13d 18h 57m 24s (233M q [196.269 qps], 11M conn, TX: 4220B, RX: 117B)
    [--] Reads / Writes: 96% / 4%
    [--] Total buffers: 640.0M global + 14.2M per thread (151 max threads)
    [OK] Maximum possible memory usage: 2.7G (11% of installed RAM)
    [OK] Slow queries: 0% (101/233M)
    [OK] Highest usage of available connections: 19% (29/151)
    [OK] Key buffer size / total MyISAM indexes: 32.0M/104.6M
    [OK] Key buffer hit rate: 100.0% (6B cached / 21K reads)
    [!!] Query cache is disabled
    [OK] Sorts requiring temporary tables: 0% (6K temp sorts / 66M sorts)
    [!!] Joins performed without indexes: 53107745
    [!!] Temporary tables created on disk: 33% (27M on disk / 80M total)
    [OK] Thread cache hit rate: 99% (339 created / 11M connections)
    [!!] Table cache hit rate: 8% (1K open / 21K opened)
    [OK] Open file limit used: 7% (2K/32K)
    [OK] Table locks acquired immediately: 99% (424M immediate / 424M locks)
    [OK] InnoDB data size / buffer pool: 558.4M/560.0M
    
    -------- Recommendations -----------------------------------------------------
    General recommendations:
        Run OPTIMIZE TABLE to defragment tables for better performance
        Enable the slow query log to troubleshoot bad queries
        Adjust your join queries to always utilize indexes
        When making adjustments, make tmp_table_size/max_heap_table_size equal
        Reduce your SELECT DISTINCT queries without LIMIT clauses
        Increase table_cache gradually to avoid file descriptor limits
    Variables to adjust:
        query_cache_size (>= 8M)
        join_buffer_size (> 2.0M, or always use indexes with joins)
        tmp_table_size (> 32M)
        max_heap_table_size (> 32M)
        table_cache (> 10240)
        
        
        
        

    mysql config

    Code:
    [mysqld]
    bind-address=127.0.0.1
    innodb_file_per_table=1
    innodb_buffer_pool_size=560M
    key_buffer_size=32M
    join_buffer_size=2M
    sort_buffer_size=2M
    read_buffer_size=2M
    read_rnd_buffer_size=8M
    myisam_sort_buffer_size=32M
    table_open_cache=10240
    table_definition_cache=10240
    thread_cache_size=16
    tmp_table_size=32M
    max_heap_table_size=32M
    long_query_time=1
    open_files_limit=32768
    

    Load Averages sar -p


    Code:
      root@some-network [~/scripts]# sar -p
    Linux 3.10.9-xxxx-grs-ipv6-64 (some.hostname.net) 	Thursday 20 February 2014 	_x86_64_	(8 CPU)
    
    12:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
    12:10:01  IST     all      2.20      0.00      0.35      0.42      0.00     97.03
    12:20:01  IST     all      2.34      0.00      0.37      0.33      0.00     96.95
    12:30:01  IST     all      2.15      0.00      0.34      0.36      0.00     97.15
    12:40:01  IST     all      1.93      0.05      0.39      0.39      0.00     97.24
    12:50:01  IST     all      2.26      0.00      0.47      0.38      0.00     96.89
    01:00:01  IST     all      1.89      0.00      0.34      0.38      0.00     97.40
    01:10:01  IST     all      1.71      0.00      0.31      0.33      0.00     97.65
    01:20:01  IST     all      2.14      0.00      0.36      0.41      0.00     97.09
    01:30:01  IST     all      1.71      0.01      0.27      0.53      0.00     97.49
    01:40:01  IST     all      1.82      0.00      0.28      0.38      0.00     97.51
    01:50:01  IST     all      1.88      0.00      0.31      0.36      0.00     97.44
    02:00:01  IST     all      1.66      0.00      0.25      0.35      0.00     97.75
    02:10:01  IST     all      1.46      0.00      0.22      0.35      0.00     97.97
    02:20:01  IST     all      1.94      0.01      0.38      0.37      0.00     97.31
    02:30:01  IST     all      1.55      0.00      0.22      0.33      0.00     97.91
    02:40:01  IST     all      1.67      0.00      0.25      0.31      0.00     97.77
    02:50:01  IST     all      1.78      0.04      0.31      0.37      0.00     97.50
    03:00:01  IST     all      4.30      0.00      1.19      0.42      0.00     94.09
    03:10:01  IST     all      1.79      0.01      0.28      0.32      0.00     97.61
    03:20:01  IST     all      1.54      0.00      0.20      0.33      0.00     97.92
    03:30:01  IST     all      1.87      0.00      0.29      0.31      0.00     97.53
    03:40:01  IST     all      1.92      0.01      0.30      0.36      0.00     97.41
    03:50:01  IST     all      2.45      0.01      0.44      0.44      0.00     96.67
    04:00:01  IST     all      1.67      0.00      0.31      0.46      0.00     97.56
    04:10:01  IST     all      1.79      0.00      0.30      0.39      0.00     97.52
    04:20:01  IST     all      1.97      0.01      0.29      0.37      0.00     97.36
    04:30:01  IST     all      1.54      0.00      0.24      0.35      0.00     97.87
    04:40:01  IST     all      1.74      0.04      0.29      0.33      0.00     97.60
    04:50:01  IST     all      1.63      0.04      0.24      0.33      0.00     97.75
    05:00:01  IST     all      1.59      0.00      0.22      0.31      0.00     97.87
    05:10:01  IST     all      1.87      0.01      0.27      0.34      0.00     97.51
    05:20:01  IST     all      1.56      0.00      0.25      0.31      0.00     97.87
    05:30:01  IST     all      1.46      0.00      0.22      0.31      0.00     98.02
    05:40:01  IST     all      1.56      0.00      0.25      0.30      0.00     97.89
    05:50:01  IST     all      1.47      0.00      0.22      0.27      0.00     98.04
    06:00:01  IST     all      1.70      0.01      0.25      0.30      0.00     97.74
    06:10:01  IST     all      1.65      0.00      0.27      0.31      0.00     97.77
    06:20:01  IST     all      1.44      0.00      0.22      0.30      0.00     98.04
    06:30:01  IST     all      1.82      0.00      0.30      0.28      0.00     97.60
    06:40:01  IST     all      1.61      0.00      0.25      0.26      0.00     97.88
    06:50:01  IST     all      1.83      0.04      0.37      0.27      0.00     97.49
    07:00:01  IST     all      1.59      0.00      0.25      0.32      0.00     97.83
    
    07:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
    07:10:01  IST     all      1.64      0.00      0.26      0.33      0.00     97.77
    07:20:01  IST     all      1.67      0.00      0.31      0.30      0.00     97.72
    07:30:01  IST     all      1.63      0.00      0.34      0.30      0.00     97.73
    07:40:01  IST     all      1.36      0.01      0.24      0.29      0.00     98.11
    07:50:01  IST     all      1.99      0.00      0.35      0.35      0.00     97.31
    08:00:01  IST     all      2.58      0.00      0.61      0.36      0.00     96.44
    08:10:01  IST     all      2.49      0.00      0.53      0.35      0.00     96.63
    08:20:01  IST     all      2.90      0.00      0.73      0.35      0.00     96.02
    08:30:01  IST     all      2.33      0.01      0.42      0.34      0.00     96.90
    08:40:01  IST     all      3.06      0.04      0.75      0.36      0.00     95.79
    08:50:01  IST     all      2.43      0.00      0.40      0.36      0.00     96.80
    09:00:01  IST     all      3.08      0.00      0.65      0.36      0.00     95.90
    09:10:01  IST     all      2.96      0.00      0.58      0.41      0.00     96.04
    09:20:01  IST     all      3.28      0.01      0.62      0.39      0.00     95.70
    09:30:01  IST     all      3.11      0.00      0.73      0.38      0.00     95.79
    09:40:01  IST     all      2.40      0.00      0.42      0.45      0.00     96.73
    09:50:01  IST     all      2.51      0.00      0.45      0.35      0.00     96.69
    10:00:01  IST     all      2.26      0.00      0.39      0.39      0.00     96.96
    10:10:01  IST     all      2.10      0.01      0.34      0.38      0.00     97.17
    10:20:01  IST     all      2.08      0.00      0.39      0.43      0.00     97.10
    10:30:01  IST     all      2.70      0.00      0.51      0.35      0.00     96.44
    10:40:01  IST     all      3.09      0.04      0.57      0.38      0.00     95.91
    10:50:01  IST     all      3.05      0.00      0.58      0.37      0.00     96.00
    11:00:01  IST     all      2.91      0.01      0.50      0.37      0.00     96.21
    11:10:01  IST     all      3.22      0.00      0.70      0.44      0.00     95.64
    11:20:01  IST     all      3.27      0.00      0.59      0.42      0.00     95.70
    11:30:01  IST     all      2.94      0.00      0.50      0.39      0.00     96.17
    11:40:01  IST     all      3.10      0.00      0.55      0.38      0.00     95.96
    11:50:01  IST     all      3.52      0.01      0.63      0.41      0.00     95.43
    12:00:01  IST     all      3.06      0.00      0.49      0.37      0.00     96.08
    12:10:01  IST     all      3.81      0.00      0.74      0.45      0.00     95.00
    12:20:01  IST     all      3.76      0.00      0.71      0.37      0.00     95.16
    12:30:02  IST     all      3.82      0.01      0.61      0.37      0.00     95.20
    12:40:01  IST     all      3.15      0.05      0.50      0.38      0.00     95.92
    12:50:01  IST     all      2.91      0.00      0.41      0.36      0.00     96.32
    01:00:01  IST     all      3.08      0.00      0.46      0.35      0.00     96.12
    01:10:01  IST     all      3.25      0.00      0.51      0.41      0.00     95.82
    01:20:01  IST     all      3.38      0.00      0.63      0.40      0.00     95.59
    01:30:01  IST     all      3.79      0.01      0.71      0.37      0.00     95.12
    01:40:01  IST     all      3.10      0.00      0.45      0.34      0.00     96.11
    01:50:01  IST     all      2.87      0.00      0.40      0.34      0.00     96.38
    02:00:01  IST     all      3.37      0.00      0.57      0.41      0.00     95.64
    
    02:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
    02:10:01  IST     all      4.25      0.00      0.87      0.42      0.00     94.46
    02:20:01  IST     all      3.54      0.01      0.61      0.38      0.00     95.46
    02:30:01  IST     all      3.63      0.00      0.73      0.38      0.00     95.25
    02:40:01  IST     all      3.56      0.00      0.53      0.43      0.00     95.47
    02:50:01  IST     all      3.92      0.05      0.68      0.40      0.00     94.95
    03:00:01  IST     all      3.51      0.00      0.56      0.35      0.00     95.59
    03:10:01  IST     all      4.27      0.01      0.73      0.41      0.00     94.58
    03:20:01  IST     all      4.03      0.00      0.76      0.41      0.00     94.80
    03:30:01  IST     all      3.75      0.01      0.61      0.36      0.00     95.27
    03:40:01  IST     all      3.86      0.00      0.72      0.43      0.00     94.99
    03:50:01  IST     all      4.30      0.00      0.76      0.40      0.00     94.54
    04:00:01  IST     all      3.53      0.00      0.59      0.38      0.00     95.50
    04:10:01  IST     all      6.22      0.01      0.65      0.37      0.00     92.75
    04:20:01  IST     all      2.61      0.00      0.43      0.34      0.00     96.61
    04:30:01  IST     all      3.40      0.00      0.63      0.34      0.00     95.63
    04:40:01  IST     all      3.69      0.00      0.56      0.39      0.00     95.35
    04:50:01  IST     all      3.30      0.05      0.54      0.35      0.00     95.76
    05:00:01  IST     all      3.48      0.01      0.52      0.35      0.00     95.64
    05:10:01  IST     all      3.72      0.00      0.60      0.39      0.00     95.29
    05:20:01  IST     all      3.86      0.00      0.65      0.36      0.00     95.13
    05:30:01  IST     all      2.82      0.00      0.49      0.38      0.00     96.31
    05:40:01  IST     all      2.73      0.00      0.47      0.37      0.00     96.43
    05:50:01  IST     all      2.70      0.34      0.43      0.37      0.00     96.16
    06:00:01  IST     all      2.61      0.00      0.39      0.32      0.00     96.68
    06:10:01  IST     all      2.67      0.00      0.41      0.37      0.00     96.54
    06:20:01  IST     all      2.65      0.00      0.41      0.36      0.00     96.58
    06:30:01  IST     all      2.78      0.00      0.44      0.30      0.00     96.47
    06:40:01  IST     all      2.81      0.05      0.42      0.33      0.00     96.39
    06:50:01  IST     all      2.92      0.00      0.50      0.32      0.00     96.26
    07:00:01  IST     all      2.72      0.00      0.49      0.34      0.00     96.44
    07:10:01  IST     all      2.81      0.00      0.75      0.34      0.00     96.10
    07:20:01  IST     all      2.29      0.00      0.42      0.29      0.00     96.99
    07:30:01  IST     all      2.34      0.01      0.38      0.30      0.00     96.97
    07:40:01  IST     all      2.61      0.00      0.42      0.33      0.00     96.64
    07:50:01  IST     all      2.67      0.00      0.51      0.32      0.00     96.50
    08:00:01  IST     all      2.25      0.00      0.37      0.31      0.00     97.07
    08:10:01  IST     all      2.32      0.00      0.36      0.33      0.00     96.98
    08:20:01  IST     all      2.07      0.01      0.36      0.32      0.00     97.25
    08:30:01  IST     all      3.06      0.00      0.63      0.35      0.00     95.96
    08:40:01  IST     all      2.70      0.04      0.57      0.29      0.00     96.40
    08:50:01  IST     all      2.47      0.00      0.40      0.34      0.00     96.78
    09:00:01  IST     all      2.10      0.00      0.36      0.34      0.00     97.20
    
    09:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
    09:10:01  IST     all      2.33      0.01      0.37      0.35      0.00     96.95
    09:20:01  IST     all      2.97      0.00      0.51      0.32      0.00     96.18
    09:30:01  IST     all      2.35      0.00      0.37      0.36      0.00     96.92
    09:40:01  IST     all      2.22      0.00      0.33      0.31      0.00     97.13
    09:50:01  IST     all      2.44      0.00      0.42      0.35      0.00     96.79
    10:00:01  IST     all      2.69      0.01      0.44      0.36      0.00     96.50
    10:10:01  IST     all      2.78      0.00      0.56      0.36      0.00     96.29
    10:20:01  IST     all      2.89      0.00      0.59      0.35      0.00     96.17
    10:30:01  IST     all      2.42      0.00      0.44      0.34      0.00     96.79
    10:40:01  IST     all      2.52      0.04      0.40      0.33      0.00     96.71
    10:50:01  IST     all      2.30      0.01      0.36      0.33      0.00     97.00
    11:00:01  IST     all      2.12      0.00      0.32      0.32      0.00     97.23
    11:10:01  IST     all      2.13      0.00      0.36      0.35      0.00     97.15
    11:20:01  IST     all      2.45      0.00      0.39      0.34      0.00     96.82
    11:30:01  IST     all      2.61      0.00      0.49      0.36      0.00     96.54
    11:40:01  IST     all      2.09      0.01      0.34      0.34      0.00     97.22
    11:50:01  IST     all      2.27      0.00      0.38      0.34      0.00     97.00
    12:00:01  IST     all     13.58      0.02      1.90      0.49      0.00     84.01
    12:10:01  IST     all     15.27      0.00      2.27      0.58      0.00     81.88
    12:20:01  IST     all     15.82      0.00      2.33      0.50      0.00     81.34
    12:30:01  IST     all     15.66      0.00      2.25      0.47      0.00     81.61
    12:40:01  IST     all     15.36      0.00      2.26      0.52      0.00     81.86
    12:50:01  IST     all     15.75      0.11      2.27      0.51      0.00     81.38
    01:00:01  IST     all     16.36      0.00      2.44      0.55      0.00     80.65
    01:10:01  IST     all     15.83      0.00      2.34      0.64      0.00     81.19
    01:20:01  IST     all     15.98      0.00      2.29      0.71      0.00     81.01
    01:30:01  IST     all     16.05      0.00      2.39      0.67      0.00     80.89
    01:40:01  IST     all     15.98      0.01      2.36      0.61      0.00     81.04
    01:50:01  IST     all     16.54      0.00      2.63      0.61      0.00     80.22
    02:00:01  IST     all     16.67      0.00      2.38      0.61      0.00     80.33
    02:10:01  IST     all     15.86      0.00      2.33      0.54      0.00     81.27
    02:20:01  IST     all     15.89      0.00      2.28      0.48      0.00     81.35
    02:30:01  IST     all     15.74      0.01      2.35      0.57      0.00     81.32
    02:40:01  IST     all     15.68      0.10      2.30      0.61      0.00     81.32
    02:50:01  IST     all     15.10      0.00      2.28      0.49      0.00     82.13
    03:00:01  IST     all     15.41      0.00      2.23      0.51      0.00     81.85
    03:10:01  IST     all     16.04      0.00      2.33      0.63      0.00     81.00
    03:20:01  IST     all     16.51      0.01      2.40      0.58      0.00     80.50
    03:30:01  IST     all     15.20      0.00      2.20      0.43      0.00     82.16
    03:40:01  IST     all     15.96      0.02      2.33      0.48      0.00     81.21
    03:50:01  IST     all     16.38      0.00      2.35      0.47      0.00     80.79
    04:00:01  IST     all     15.37      0.04      2.23      0.45      0.00     81.91
    
    04:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
    04:10:01  IST     all     15.55      0.01      2.26      0.54      0.00     81.64
    04:20:01  IST     all     15.62      0.00      2.24      0.62      0.00     81.51
    04:30:01  IST     all     15.77      0.00      2.26      0.57      0.00     81.39
    04:40:01  IST     all     15.74      0.09      2.32      0.56      0.00     81.29
    04:50:01  IST     all     15.45      0.00      2.26      0.43      0.00     81.85
    05:00:01  IST     all     16.45      0.01      2.44      0.54      0.00     80.56
    05:10:01  IST     all     15.25      0.00      2.26      0.51      0.00     81.98
    05:20:01  IST     all     16.03      0.00      2.33      0.49      0.00     81.15
    05:30:01  IST     all     16.10      0.00      2.31      0.49      0.00     81.09
    05:40:01  IST     all     15.48      0.00      2.25      0.54      0.00     81.73
    05:50:01  IST     all     14.72      0.01      2.27      0.44      0.00     82.57
    06:00:01  IST     all     15.40      0.00      2.20      0.47      0.00     81.92
    06:10:01  IST     all     15.94      0.00      2.31      0.51      0.00     81.23
    06:20:01  IST     all     14.39      0.00      2.05      0.49      0.00     83.06
    06:30:01  IST     all     13.76      0.00      1.95      0.52      0.00     83.77
    06:40:01  IST     all     15.10      0.10      2.22      0.61      0.00     81.96
    06:50:01  IST     all     14.88      0.00      2.22      0.51      0.00     82.38
    07:00:01  IST     all     14.71      0.00      2.14      0.52      0.00     82.62
    07:10:01  IST     all     14.42      0.00      2.11      0.46      0.00     83.01
    07:20:01  IST     all     13.62      0.00      1.97      0.56      0.00     83.86
    07:30:01  IST     all     13.82      0.01      1.99      0.52      0.00     83.66
    07:40:01  IST     all     13.88      0.00      1.97      0.46      0.00     83.69
    07:50:01  IST     all     13.34      0.00      1.92      0.54      0.00     84.20
    08:00:01  IST     all     14.61      0.00      2.21      0.42      0.00     82.76
    08:10:01  IST     all     14.48      0.00      2.11      0.51      0.00     82.89
    08:20:01  IST     all     15.40      0.01      2.24      0.50      0.00     81.85
    08:30:01  IST     all     15.30      0.00      2.19      0.37      0.00     82.13
    08:40:01  IST     all     16.15      0.00      2.40      0.60      0.00     80.84
    08:50:01  IST     all     16.31      0.11      2.39      0.54      0.00     80.65
    09:00:01  IST     all     14.46      0.00      2.10      0.57      0.00     82.86
    09:10:01  IST     all     15.64      0.01      2.30      0.57      0.00     81.49
    09:20:01  IST     all     16.32      0.00      2.43      0.61      0.00     80.64
    09:30:01  IST     all     15.15      0.00      2.18      0.54      0.00     82.13
    09:40:01  IST     all     15.94      0.00      2.31      0.53      0.00     81.21
    09:50:01  IST     all     16.27      0.00      2.43      0.57      0.00     80.73
    10:00:01  IST     all     15.19      0.01      2.24      0.54      0.00     82.02
    10:10:01  IST     all     15.37      0.00      2.22      0.54      0.00     81.87
    10:20:01  IST     all     15.77      0.00      2.27      0.62      0.00     81.34
    10:30:01  IST     all     15.80      0.00      2.26      0.63      0.00     81.31
    10:40:01  IST     all     16.02      0.00      2.33      0.60      0.00     81.04
    10:50:01  IST     all     16.08      0.10      2.34      0.66      0.00     80.82
    11:00:01  IST     all     14.85      0.00      2.15      0.57      0.00     82.43
    
    11:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
    11:10:01  IST     all     15.40      0.00      2.31      0.52      0.00     81.77
    11:20:01  IST     all     15.86      0.00      2.29      0.56      0.00     81.29
    11:30:01  IST     all     15.54      0.00      2.29      0.59      0.00     81.58
    11:40:01  IST     all     15.70      0.01      2.27      0.54      0.00     81.48
    11:50:01  IST     all     15.81      0.00      2.29      0.77      0.00     81.12
    12:00:01  IST     all     16.26      0.00      2.37      0.75      0.00     80.62
    12:10:01  IST     all     14.53      0.00      2.04      0.75      0.00     82.68
    12:20:01  IST     all     14.98      0.00      2.13      0.51      0.00     82.37
    12:30:01  IST     all     16.39      0.01      2.39      0.51      0.00     80.70
    12:40:01  IST     all     16.57      0.10      2.41      0.54      0.00     80.39
    12:50:01  IST     all     14.42      0.00      2.10      0.65      0.00     82.83
    01:00:01  IST     all     16.87      0.00      2.44      0.86      0.00     79.82
    01:10:01  IST     all     15.01      0.00      2.22      0.67      0.00     82.09
    01:20:01  IST     all     15.13      0.01      2.13      0.49      0.00     82.24
    01:30:01  IST     all     16.26      0.00      2.49      0.58      0.00     80.65
    01:40:01  IST     all     15.42      0.00      2.22      0.55      0.00     81.80
    01:50:01  IST     all     15.60      0.00      2.25      0.86      0.00     81.28
    02:00:01  IST     all     14.82      0.00      2.10      0.79      0.00     82.28
    02:10:01  IST     all     19.04      0.01      2.52      0.72      0.00     77.72
    02:20:01  IST     all     15.42      0.00      2.20      0.75      0.00     81.62
    02:30:01  IST     all     15.51      0.00      2.22      0.77      0.00     81.50
    02:40:01  IST     all     15.28      0.09      2.26      0.54      0.00     81.82
    02:50:01  IST     all     15.96      0.00      2.33      0.68      0.00     81.03
    03:00:01  IST     all     16.47      0.00      2.42      0.83      0.00     80.27
    03:10:01  IST     all     16.25      0.01      2.39      0.79      0.00     80.57
    03:20:01  IST     all     16.27      0.00      2.39      0.88      0.00     80.46
    03:30:02  IST     all     17.21      0.00      2.58      0.68      0.00     79.53
    03:40:01  IST     all     15.56      0.00      2.22      0.87      0.00     81.35
    03:50:01  IST     all     16.05      0.00      2.33      0.90      0.00     80.71
    04:00:01  IST     all     16.19      0.01      2.34      0.85      0.00     80.60
    04:10:01  IST     all     16.06      0.00      2.34      0.83      0.00     80.76
    04:20:02  IST     all     16.80      0.00      2.44      0.75      0.00     80.01
    04:30:01  IST     all      8.06      0.00      1.09      0.57      0.00     90.28
    04:40:01  IST     all     15.70      0.10      2.26      0.52      0.00     81.43
    04:50:01  IST     all     16.19      0.01      2.33      0.53      0.00     80.95
    05:00:01  IST     all     15.34      0.00      2.22      0.53      0.00     81.91
    05:10:01  IST     all     15.89      0.00      2.28      0.54      0.00     81.30
    05:20:01  IST     all     15.75      0.00      2.28      0.54      0.00     81.43
    05:30:01  IST     all     15.95      0.00      2.35      0.68      0.00     81.02
    05:40:01  IST     all     16.17      1.63      2.40      0.55      0.00     79.25
    05:50:01  IST     all     17.11      0.01      2.54      0.59      0.00     79.76
    06:00:01  IST     all     17.19      0.00      2.55      0.55      0.00     79.71
    
    06:00:01  IST     CPU     %user     %nice   %system   %iowait    %steal     %idle
    06:10:01  IST     all     16.83      0.00      2.51      0.65      0.00     80.02
    06:20:01  IST     all     16.78      0.00      2.49      0.63      0.00     80.10
    06:30:01  IST     all     15.85      0.00      2.35      0.59      0.00     81.20
    06:40:01  IST     all     16.74      0.10      2.45      0.55      0.00     80.16
    06:50:01  IST     all     15.78      0.00      2.33      0.59      0.00     81.29
    07:00:01  IST     all     16.33      0.00      2.45      0.76      0.00     80.45
    07:10:01  IST     all     17.40      0.00      2.56      0.83      0.00     79.20
    07:20:01  IST     all     17.14      0.00      2.50      0.66      0.00     79.69
    07:30:01  IST     all     18.79      0.01      2.76      0.61      0.00     77.83
    07:40:01  IST     all     17.23      0.00      2.49      0.62      0.00     79.65
    07:50:01  IST     all     18.18      0.00      2.66      0.63      0.00     78.53
    08:00:01  IST     all     17.46      0.00      2.50      0.58      0.00     79.45
    08:10:01  IST     all     17.51      0.00      2.59      0.75      0.00     79.14
    08:20:01  IST     all     16.55      0.01      2.42      0.78      0.00     80.25
    08:30:01  IST     all     16.15      0.00      2.32      0.56      0.00     80.97
    08:40:01  IST     all     17.59      0.00      2.50      0.72      0.00     79.18
    Average:        all     12.60      0.02      1.78      0.49      0.00     85.11  
    
    
    
    --------------------------------------------------------------------------------------------
    

    hardware info

    Code:
    root@some-network [~/scripts]# cat /proc/cpuinfo
    processor	: 0
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 26
    model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
    stepping	: 5
    microcode	: 0x11
    cpu MHz		: 2801.000
    cache size	: 8192 KB
    physical id	: 0
    siblings	: 8
    core id		: 0
    cpu cores	: 4
    apicid		: 0
    initial apicid	: 0
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 11
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips	: 5600.32
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 36 bits physical, 48 bits virtual
    power management:
    
    processor	: 1
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 26
    model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
    stepping	: 5
    microcode	: 0x11
    cpu MHz		: 2801.000
    cache size	: 8192 KB
    physical id	: 0
    siblings	: 8
    core id		: 1
    cpu cores	: 4
    apicid		: 2
    initial apicid	: 2
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 11
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips	: 5600.32
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 36 bits physical, 48 bits virtual
    power management:
    
    processor	: 2
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 26
    model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
    stepping	: 5
    microcode	: 0x11
    cpu MHz		: 2801.000
    cache size	: 8192 KB
    physical id	: 0
    siblings	: 8
    core id		: 2
    cpu cores	: 4
    apicid		: 4
    initial apicid	: 4
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 11
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips	: 5600.32
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 36 bits physical, 48 bits virtual
    power management:
    
    processor	: 3
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 26
    model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
    stepping	: 5
    microcode	: 0x11
    cpu MHz		: 2801.000
    cache size	: 8192 KB
    physical id	: 0
    siblings	: 8
    core id		: 3
    cpu cores	: 4
    apicid		: 6
    initial apicid	: 6
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 11
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips	: 5600.32
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 36 bits physical, 48 bits virtual
    power management:
    
    processor	: 4
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 26
    model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
    stepping	: 5
    microcode	: 0x11
    cpu MHz		: 2801.000
    cache size	: 8192 KB
    physical id	: 0
    siblings	: 8
    core id		: 0
    cpu cores	: 4
    apicid		: 1
    initial apicid	: 1
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 11
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips	: 5600.32
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 36 bits physical, 48 bits virtual
    power management:
    
    processor	: 5
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 26
    model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
    stepping	: 5
    microcode	: 0x11
    cpu MHz		: 2801.000
    cache size	: 8192 KB
    physical id	: 0
    siblings	: 8
    core id		: 1
    cpu cores	: 4
    apicid		: 3
    initial apicid	: 3
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 11
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips	: 5600.32
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 36 bits physical, 48 bits virtual
    power management:
    
    processor	: 6
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 26
    model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
    stepping	: 5
    microcode	: 0x11
    cpu MHz		: 2801.000
    cache size	: 8192 KB
    physical id	: 0
    siblings	: 8
    core id		: 2
    cpu cores	: 4
    apicid		: 5
    initial apicid	: 5
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 11
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips	: 5600.32
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 36 bits physical, 48 bits virtual
    power management:
    
    processor	: 7
    vendor_id	: GenuineIntel
    cpu family	: 6
    model		: 26
    model name	: Intel(R) Xeon(R) CPU           W3530  @ 2.80GHz
    stepping	: 5
    microcode	: 0x11
    cpu MHz		: 2801.000
    cache size	: 8192 KB
    physical id	: 0
    siblings	: 8
    core id		: 3
    cpu cores	: 4
    apicid		: 7
    initial apicid	: 7
    fpu		: yes
    fpu_exception	: yes
    cpuid level	: 11
    wp		: yes
    flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
    bogomips	: 5600.32
    clflush size	: 64
    cache_alignment	: 64
    address sizes	: 36 bits physical, 48 bits virtual
    power management:
    

    memory info


    Code:
    root@some-network [~/scripts]# cat /proc/meminfo
    MemTotal:       24737800 kB
    MemFree:        16371108 kB
    Buffers:          788680 kB
    Cached:          4937140 kB
    SwapCached:            0 kB
    Active:          5279624 kB
    Inactive:        1600700 kB
    Active(anon):    1157900 kB
    Inactive(anon):   116256 kB
    Active(file):    4121724 kB
    Inactive(file):  1484444 kB
    Unevictable:           0 kB
    Mlocked:               0 kB
    SwapTotal:        525308 kB
    SwapFree:         525308 kB
    Dirty:              1004 kB
    Writeback:             0 kB
    AnonPages:       1153496 kB
    Mapped:           166080 kB
    Shmem:            120240 kB
    Slab:            1254664 kB
    SReclaimable:     995104 kB
    SUnreclaim:       259560 kB
    KernelStack:        3104 kB
    PageTables:        11188 kB
    NFS_Unstable:          0 kB
    Bounce:                0 kB
    WritebackTmp:          0 kB
    CommitLimit:    12894208 kB
    Committed_AS:    3086672 kB
    VmallocTotal:   34359738367 kB
    VmallocUsed:      326396 kB
    VmallocChunk:   34359409652 kB
    HardwareCorrupted:     0 kB
    DirectMap4k:        5632 kB
    DirectMap2M:    25151488 kB
    I just want to know is there any particular reason for sudden increase in mysql load?
    Thank you guys.
     
  2. thinkbot

    thinkbot Well-Known Member

    Joined:
    Oct 30, 2012
    Messages:
    326
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    key_buffer_size set over 110M
    also increase innodb_buffer_pool_size=560M
    InnoDB can use more
    and decreate table cache to 2

    In your case I would turn on query cache

    Maybe it's time to optimize that code and add some caching
    [!!] Joins performed without indexes: 53107745
    You should review database structure if it needs some indexes

    I would suggest here simple solution,

    add
    log-queries-not-using-indexes

    track those queries not using indexes, and add those indexes, or if necessary optimize queries to utilize indexes


    In most cases when basic mysql optimizations (few key settings) are done, the biggest part that can impact performance is the code
    People tend to search some magic solutions in mysql, some people have tens of variables in their my.cnf, but they dont look into their code
     
  3. bassrehab

    bassrehab Member

    Joined:
    Dec 13, 2013
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Thank you for your reply.

    key_buffer_size is now set to 128M (~myisam database size)
    Innodb ~ i haven't changed.

    decreate table cache to 2? you mean 2048?

    Query cache doesnt help my case at all. There is frequent defragmentation and a constant overhead. I have checked with various values from 128M to 32M, and cache_min_res.

    Yes, i am aware of non-index using joins. Its is not a new thing for our setting, i have always had non-index using joins (i knows its bad for performance). Anyways, My problem is investigating the "sudden" rise in mysql load over last one week.
     
  4. thinkbot

    thinkbot Well-Known Member

    Joined:
    Oct 30, 2012
    Messages:
    326
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    yes, I mean table cache 2048

    but did y ou check what type of queries are those "non-index"
    can you set long query time to 0
    and then run

    cd /root
    wget http://percona.com/get/pt-query-digest
    chmod +x pt-query-digest
    pt-query-digest /var/lib/mysql/mysql-slow.log > slow.txt;

    to generate review of slow queries

    and there you can find those longest and most often executing ones
    than just run explain extended in mysql and optimize database structure/query
     
  5. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,811
    Likes Received:
    671
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    You may also want to try using a command such as "mysqladmin processlist" to get a better idea of what's happening when you notice higher than normal MySQL usage (since you stated this has only started recently).

    Thank you.
     
  6. bassrehab

    bassrehab Member

    Joined:
    Dec 13, 2013
    Messages:
    5
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    A quick update:

    1. key_buffer_size now set at 128M
    2. File open limit (auto set by cpanel ~8250)
    3. Table Cache ~ 2048
    4. Current Mysql load ~ 55%
    5. http traffic checked for any attacks (nothing unusual)
    6. Had done hard restarts, checked hardwares for r/w faults none.


    I am not sure why the load keeps increasing. Its as if some kinda overhead is building on. And it seemingly is incremental. My next plan of action is just try disabling one db at a time and probably isolate the menace.

    @thinkbot - i wonder if fractional slow_query times can be set.

    @cPanelMichael - have done that already. I was hoping to find sleeping queries or anything unusual but none there either.
     
Loading...

Share This Page