The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

very high mysql CPU usage

Discussion in 'Workarounds and Optimization' started by david@pcnetwork, May 22, 2012.

  1. david@pcnetwork

    Joined:
    May 22, 2012
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    Good day,

    Im experiancing very high mysql usage, we have installed mysql govenor (running cloudlinux), and it was high before that as well.
    Please advise.
    cpu runs between 60% and 100%.

    we also have 10 gigs of ram in here which is not being utilised:


    Code:
    david@cpanel3 [~]# free -m
                 total       used       free     shared    buffers     cached
    Mem:         10240      10008        231          0        523       7303
    -/+ buffers/cache:       2181       8058
    Swap:         1055          0       1055
    david@cpanel3 [~]# 

    Code:
    david@cpanel3 [~]# cat /etc/my.cnf
    [mysqld]
    set-variable = max_connections=300
    log-slow-queries
    safe-show-database
    thread_cache_size = 4
    key_buffer_size = 512M
    query_cache_size = 80M
    query_cache_limit = 90M
    innodb_buffer_pool_size = 350M
    join_buffer_size = 40M
    tmp_table_size = 290MB
    max_heap_table_size = 290MB
    table_cache = 2500
    log-slow-queries=/var/lib/mysql/slow.log
    long_query_time=1
    
    Code:
    root@cpanel3 [/home/david]# ./mysqltuner.pl 
    
     >>  MySQLTuner 1.2.0 - Major Hayden <major@mhtx.net>
     >>  Bug reports, feature requests, and downloads at [url]http://mysqltuner.com/[/url]
     >>  Run with '--help' for additional options and output filtering
    
    -------- General Statistics --------------------------------------------------
    [--] Skipped version check for MySQLTuner script
    [OK] Currently running supported MySQL version 5.1.62-cll
    [OK] Operating on 64-bit architecture
    
    -------- Storage Engine Statistics -------------------------------------------
    [--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster 
    [--] Data in MyISAM tables: 2G (Tables: 1783)
    [--] Data in InnoDB tables: 308M (Tables: 206)
    [--] Data in MEMORY tables: 0B (Tables: 2)
    [!!] Total fragmented tables: 267
    
    -------- Security Recommendations  -------------------------------------------
    [OK] All database users have passwords assigned
    
    -------- Performance Metrics -------------------------------------------------
    [--] Up for: 19h 20m 32s (54M q [783.533 qps], 86K conn, TX: 38B, RX: 6B)
    [--] Reads / Writes: 99% / 1%
    [--] Total buffers: 1.2G global + 42.6M per thread (300 max threads)
    [!!] Maximum possible memory usage: 13.7G (136% of installed RAM)
    [OK] Slow queries: 0% (74/54M)
    [OK] Highest usage of available connections: 7% (22/300)
    [OK] Key buffer size / total MyISAM indexes: 512.0M/601.4M
    [OK] Key buffer hit rate: 99.5% (10M cached / 52K reads)
    [!!] Query cache efficiency: 11.3% (6M cached / 54M selects)
    [!!] Query cache prunes per day: 6484816
    [OK] Sorts requiring temporary tables: 0% (246 temp sorts / 100K sorts)
    [!!] Joins performed without indexes: 5630
    [OK] Temporary tables created on disk: 11% (21K on disk / 189K total)
    [OK] Thread cache hit rate: 96% (3K created / 86K connections)
    [OK] Table cache hit rate: 99% (2K open / 2K opened)
    [OK] Open file limit used: 69% (3K/5K)
    [OK] Table locks acquired immediately: 99% (47M immediate / 47M locks)
    [OK] InnoDB data size / buffer pool: 308.1M/350.0M
    
    -------- Recommendations -----------------------------------------------------
    General recommendations:
        Run OPTIMIZE TABLE to defragment tables for better performance
        MySQL started within last 24 hours - recommendations may be inaccurate
        Reduce your overall MySQL memory footprint for system stability
        Adjust your join queries to always utilize indexes
    Variables to adjust:
      *** MySQL's maximum memory usage is dangerously high ***
      *** Add RAM before increasing MySQL buffer variables ***
        query_cache_limit (> 90M, or use smaller result sets)
        query_cache_size (> 80M)
        join_buffer_size (> 40.0M, or always use indexes with joins)
    
    root@cpanel3 [/home/david]#
     
  2. david@pcnetwork

    Joined:
    May 22, 2012
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    top - 15:48:15 up 7 days, 9:30, 2 users, load average: 3.04, 3.14, 2.89
    Tasks: 300 total, 4 running, 293 sleeping, 0 stopped, 3 zombie
    Cpu(s): 13.2%us, 5.9%sy, 0.0%ni, 79.8%id, 0.0%wa, 0.0%hi, 0.1%si, 1.1%st
    Mem: 10485760k total, 10070812k used, 414948k free, 529400k buffers
    Swap: 1081336k total, 104k used, 1081232k free, 7402416k cached
     
  3. -GR-

    -GR- Active Member

    Joined:
    May 2, 2012
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    You can try these settings, I didn't change it by much but you only want your query_cache_limit to be a couple megs, I think the default is 2M. You had it larger then your query_cache. I also dropped your max connections to 150 as you are only using 22% of the total.

    Code:
    [mysqld]
    set-variable = max_connections=150
    log-slow-queries
    safe-show-database
    thread_cache_size = 4
    key_buffer_size = 610M
    query_cache_size = 128M
    query_cache_limit = 4M
    innodb_buffer_pool_size = 350M
    join_buffer_size = 40M
    tmp_table_size = 290MB
    max_heap_table_size = 290MB
    table_cache = 2500
    log-slow-queries=/var/lib/mysql/slow.log
    long_query_time=4
    
     
  4. -GR-

    -GR- Active Member

    Joined:
    May 2, 2012
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Also are you sure it is mysql that is causing the load? Going by the top you posted you have a lot of sleeping processes and 3 zombie processes.
     
  5. david@pcnetwork

    Joined:
    May 22, 2012
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    thanks for your advise.

    ive dropped the max connections to 100 as max used was 22. thats decreased ram usage. Ive also emptied the tables in eximstats which was 4 gigs and ive decreased logging period to 7 days. I also changed the exim stats engin to INNODB as it does a lot of rights.

    i then changed my my.cnf as per your suggestions.
    ill post info and another mysqltuner output in 24 hrs.

    thank you.
     
  6. -GR-

    -GR- Active Member

    Joined:
    May 2, 2012
    Messages:
    42
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    I also like using tuning-primer along with mysqltuner. Eva also posted his script for tuning as well here on the forums.
     
  7. david@pcnetwork

    Joined:
    May 22, 2012
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    Hi There,

    This has really helped in terms of the CPU usage. I can see a big difference today. Thank you

    When I run mysqltuner however, its asking me to increase query_cache_limit and size. Any comments here if this should be done ?


    -------- Performance Metrics -------------------------------------------------
    [--] Up for: 10h 24m 37s (11M q [310.564 qps], 35K conn, TX: 11B, RX: 1B)
    [--] Reads / Writes: 99% / 1%
    [--] Total buffers: 1.3G global + 62.6M per thread (100 max threads)
    [OK] Maximum possible memory usage: 7.4G (73% of installed RAM)
    [OK] Slow queries: 0% (11/11M)
    [OK] Highest usage of available connections: 16% (16/100)
    [OK] Key buffer size / total MyISAM indexes: 512.0M/29.9M
    [OK] Key buffer hit rate: 99.5% (491K cached / 2K reads)
    [!!] Query cache efficiency: 17.0% (1M cached / 11M selects)
    [!!] Query cache prunes per day: 92882
    [OK] Sorts requiring temporary tables: 0% (74 temp sorts / 17K sorts)
    [!!] Joins performed without indexes: 342
    [OK] Temporary tables created on disk: 10% (7K on disk / 71K total)
    [OK] Thread cache hit rate: 98% (656 created / 35K connections)
    [OK] Table cache hit rate: 98% (2K open / 2K opened)
    [OK] Open file limit used: 71% (3K/5K)
    [OK] Table locks acquired immediately: 99% (9M immediate / 9M locks)
    [OK] InnoDB data size / buffer pool: 312.8M/350.0M

    -------- Recommendations -----------------------------------------------------
    General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Adjust your join queries to always utilize indexes
    Variables to adjust:
    query_cache_limit (> 4M, or use smaller result sets)
    query_cache_size (> 128M)
    join_buffer_size (> 60.0M, or always use indexes with joins)
     
  8. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    The results are less than 24 hours long. You'll need to wait the full 24+ hours before re-running the script.
     
  9. david@pcnetwork

    Joined:
    May 22, 2012
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    Hi Tristan and thanks for your help this far. mysql is still using high cpu load. herewith my details since your suggestions:

    root@cpanel3 [~]# mysql_config --version
    5.1.62

    root@cpanel3 [~]# cat /etc/my.cnf
    [mysqld]
    set-variable = max_connections=100
    log-slow-queries
    safe-show-database
    thread_cache_size = 4
    key_buffer_size = 512M
    query_cache_size = 128M

    #dropped as per advise from forum
    query_cache_limit = 8M

    innodb_buffer_pool_size = 350M
    join_buffer_size = 60M
    tmp_table_size = 290MB
    max_heap_table_size = 290MB
    table_cache = 2500
    log-slow-queries=/var/lib/mysql/slow.log
    long_query_time=1

    #buffers
    key_buffer_size = 2000M
    read_buffer_size = 128K
    join_buffer_size = 128K


    root@cpanel3 [~]# /home/david/mysqltuner.pl

    >> MySQLTuner 1.2.0 - Major Hayden <major@mhtx.net>
    >> Bug reports, feature requests, and downloads at http://mysqltuner.com/
    >> Run with '--help' for additional options and output filtering

    -------- General Statistics --------------------------------------------------
    [--] Skipped version check for MySQLTuner script
    [OK] Currently running supported MySQL version 5.1.62-cll
    [OK] Operating on 64-bit architecture

    -------- Storage Engine Statistics -------------------------------------------
    [--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
    [--] Data in MyISAM tables: 306M (Tables: 1806)
    [--] Data in InnoDB tables: 315M (Tables: 206)
    [--] Data in MEMORY tables: 0B (Tables: 2)
    [!!] Total fragmented tables: 282

    -------- Security Recommendations -------------------------------------------
    [OK] All database users have passwords assigned

    -------- Performance Metrics -------------------------------------------------
    [--] Up for: 9d 15h 30m 44s (372M q [447.295 qps], 599K conn, TX: 193B, RX: 47B)
    [--] Reads / Writes: 99% / 1%
    [--] Total buffers: 2.7G global + 2.7M per thread (100 max threads)
    [OK] Maximum possible memory usage: 3.0G (29% of installed RAM)
    [OK] Slow queries: 0% (364/372M)
    [OK] Highest usage of available connections: 20% (20/100)
    [OK] Key buffer size / total MyISAM indexes: 2.0G/51.6M
    [OK] Key buffer hit rate: 99.9% (14M cached / 9K reads)
    [!!] Query cache efficiency: 9.5% (35M cached / 369M selects)
    [!!] Query cache prunes per day: 65954
    [OK] Sorts requiring temporary tables: 0% (2K temp sorts / 377K sorts)
    [!!] Joins performed without indexes: 5813
    [OK] Temporary tables created on disk: 13% (209K on disk / 1M total)
    [OK] Thread cache hit rate: 98% (8K created / 599K connections)
    [OK] Table cache hit rate: 26% (2K open / 9K opened)
    [OK] Open file limit used: 79% (4K/5K)
    [OK] Table locks acquired immediately: 99% (333M immediate / 333M locks)
    [OK] InnoDB data size / buffer pool: 315.2M/350.0M

    -------- Recommendations -----------------------------------------------------
    General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Adjust your join queries to always utilize indexes
    Variables to adjust:
    query_cache_limit (> 8M, or use smaller result sets)
    query_cache_size (> 128M)
    join_buffer_size (> 128.0K, or always use indexes with joins)

    root@cpanel3 [~]#
     
  10. chrismfz

    chrismfz Well-Known Member

    Joined:
    Jul 4, 2007
    Messages:
    109
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Greece
    cPanel Access Level:
    DataCenter Provider
    Try a myisamchk and a mysqlcheck after that. Maybe a table is marked as crashed or corrupted and that causes those issues. (Try a -f (force) -r (repair))

    Also Key buffer size is way to high without reason. Try 32 or 64mb for a start, check 24hours later.
    By the way, it's also a duplicate. Once defined 512mb and once again 2G.

    You can also try either comment slow queries log or increase the number to 4 or 5
    log-slow-queries=/var/lib/mysql/slow.log
    long_query_time=1

    to save IO but I don't believe that this causes trouble.

    You also have join_buffer_size 2 times in conf. Shouldn't be larger than 1-2MB by the way.
     
  11. rmackay

    rmackay Well-Known Member

    Joined:
    Nov 26, 2002
    Messages:
    75
    Likes Received:
    0
    Trophy Points:
    6
    check the size of your eximstats tables. a recent update set the retention level to 90 days, which caused my eximstats tables to hit 6GB+ in no time. This lead to some really high server loads.
     
  12. david@pcnetwork

    Joined:
    May 22, 2012
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    DataCenter Provider
    Hi Guys,

    Thanks for all the comments - this is very helpful.

    I did decrease eximstats to 7 days a while back - I think that did help slightly with disk IO.
    PMchrismfz: I also made the changes as per your suggestion

    Im still however getting very high mysql CPU loads on this cpanel machine.

    Herewith latest mysqltuner ouput as well as version and conf file, any other suggestions in terms of the mysqltuner output will be helpful.


    root@cpanel3 [/home/david/checkSoftCpanelPasswords]# cat /etc/my.cnf
    [mysqld]
    set-variable = max_connections=100
    log-slow-queries
    safe-show-database
    thread_cache_size = 4
    query_cache_size = 128M

    #dropped as per advise from forum
    query_cache_limit = 8M

    innodb_buffer_pool_size = 350M
    tmp_table_size = 290MB
    max_heap_table_size = 290MB
    table_cache = 2500
    log-slow-queries=/var/lib/mysql/slow.log
    long_query_time=4

    #buffers
    key_buffer_size = 64M
    read_buffer_size = 128K
    join_buffer_size = 2M
    #for mysql govenor
    userstat_running=1





    root@cpanel3 [/home/david/checkSoftCpanelPasswords]# /home/david/mysqltuner.pl

    >> MySQLTuner 1.2.0 - Major Hayden <major@mhtx.net>
    >> Bug reports, feature requests, and downloads at http://mysqltuner.com/
    >> Run with '--help' for additional options and output filtering

    -------- General Statistics --------------------------------------------------
    [--] Skipped version check for MySQLTuner script
    [OK] Currently running supported MySQL version 5.1.63-cll
    [OK] Operating on 64-bit architecture

    -------- Storage Engine Statistics -------------------------------------------
    [--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
    [--] Data in MyISAM tables: 409M (Tables: 1805)
    [--] Data in InnoDB tables: 315M (Tables: 206)
    [--] Data in MEMORY tables: 0B (Tables: 2)
    [!!] Total fragmented tables: 257

    -------- Security Recommendations -------------------------------------------
    [OK] All database users have passwords assigned

    -------- Performance Metrics -------------------------------------------------
    [--] Up for: 1d 14h 7m 32s (94M q [687.865 qps], 46K conn, TX: 21B, RX: 11B)
    [--] Reads / Writes: 99% / 1%
    [--] Total buffers: 834.0M global + 4.6M per thread (100 max threads)
    [OK] Maximum possible memory usage: 1.3G (12% of installed RAM)
    [OK] Slow queries: 0% (1/94M)
    [OK] Highest usage of available connections: 14% (14/100)
    [OK] Key buffer size / total MyISAM indexes: 64.0M/67.3M
    [OK] Key buffer hit rate: 99.7% (5M cached / 15K reads)
    [!!] Query cache efficiency: 5.2% (4M cached / 93M selects)
    [!!] Query cache prunes per day: 38446
    [OK] Sorts requiring temporary tables: 0% (603 temp sorts / 96K sorts)
    [!!] Joins performed without indexes: 926
    [OK] Temporary tables created on disk: 13% (30K on disk / 227K total)
    [OK] Thread cache hit rate: 99% (212 created / 46K connections)
    [OK] Table cache hit rate: 98% (2K open / 2K opened)
    [OK] Open file limit used: 72% (3K/5K)
    [OK] Table locks acquired immediately: 99% (89M immediate / 89M locks)
    [OK] InnoDB data size / buffer pool: 315.2M/350.0M

    -------- Recommendations -----------------------------------------------------
    General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Adjust your join queries to always utilize indexes
    Variables to adjust:
    query_cache_limit (> 8M, or use smaller result sets)
    query_cache_size (> 128M)
    join_buffer_size (> 2.0M, or always use indexes with joins)

    root@cpanel3 [/home/david/checkSoftCpanelPasswords]#



    Server version: 5.1.63-cll MySQL Community Server (GPL)



    Thanks
    David
    www.pcnetwork.co.za
     
Loading...

Share This Page